Multi-task reinforcement learning method based on emerging hybrid expert model

Abstract

The invention discloses a multi-task reinforcement learning method based on an emergence hybrid expert model, and the method comprises the following steps: constructing a multi-task reinforcement learning network which comprises a plurality of feedforward neural networks, and connecting a routing network in front of each feedforward neural network; performing clustering analysis on each feedforward neural network, and equally dividing the feedforward neural network into a plurality of expert modules; training each routing network based on a strategy gradient descent method, and updating parameters of corresponding expert modules; selecting experiment tasks in the multi-task learning environment; and for each experiment task, activating a corresponding expert module based on the trained routing network to perform decoupling processing, and completing multi-task reinforcement learning. According to the method, an efficient and stable optimization strategy is provided for multi-task learning, and the problem of mutual interference between parameter sharing and tasks in multi-task learning is effectively relieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of reinforcement learning, and in particular relates to a multi-task reinforcement learning method based on an emergent hybrid expert model. Background Art

[0002] Multi-task learning in the process of intelligent robot operation is a very challenging problem in reinforcement learning. Although joint training of multiple tasks allows policies to share parameters between different tasks, this makes the optimization problem more complicated because it is still unclear which parameters in the network should be reused between tasks and how gradients from different tasks interfere with each other. Therefore, in order to reduce the mutual interference between tasks during training, parameters cannot be simply shared across tasks, and this optimization problem needs to be alleviated in terms of policy representation.

[0003] In view of the problems in the above-mentioned multi-task reinforcement learning methods, it is urgent to propose a multi-task reinforcement learning method based on the emergent hybrid expert model. Summary of the invention

[0004] To solve the above technical problems, the present invention proposes a multi-task reinforcement learning method based on an emergent hybrid expert model, which can decouple the learning of different tasks in different expert modules, thereby reducing interference between tasks, improving the performance of each task, and solving the optimization problem between parameter sharing and task interference in multi-task learning, so as to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a multi-task reinforcement learning method based on an emergent hybrid expert model, comprising the following steps:

[0006] Constructing a multi-task reinforcement learning network, wherein the multi-task reinforcement learning network includes a plurality of feedforward neural networks, each of which is connected to a routing network;

[0007] Cluster analysis is performed on each feedforward neural network, which is divided into several expert modules;

[0008] Train each routing network based on the policy gradient descent method and update the parameters of the corresponding expert module;

[0009] Select experimental tasks in a multi-task learning environment;

[0010] For each experimental task, the corresponding expert module is activated based on the trained routing network to perform decoupling processing and complete multi-task reinforcement learning.

[0011] Optionally, the multi-task reinforcement learning network also includes a layer of fully connected neural network, and the fully connected neural network is connected to a plurality of feedforward neural networks; wherein each feedforward neural network includes three layers of networks, the first layer of the network is used as a key, the second layer of the network is used as an activation layer, and the third layer of the network is used as a value.

[0012] Optionally, the process of performing cluster analysis on each feedforward neural network and dividing it into a number of expert modules includes:

[0013] For each feedforward neural network, the number of sample sets and the number of category clusters to be divided are preset, and the number of samples in each category cluster is the same;

[0014] Randomly select a number of data center points from the sample set as the center of each category cluster;

[0015] Perform the first round of iterations for each sample, obtain the distance of each sample to each data center point and sort them;

[0016] Divide the samples into the closest category clusters;

[0017] Update the centroid of each category cluster and perform a second round of iteration for each sample;

[0018] When the number of iterations is reached, several clusters of categories are obtained after clustering;

[0019] The samples in each category cluster after clustering are combined into an expert module.

[0020] Optionally, the process of training each routing network based on the policy gradient descent method includes:

[0021] The policy gradient descent method adopts the SAC algorithm, which includes a policy network, two value networks, a target policy network and two target value networks;

[0022] Based on the SAC algorithm, the optimization target and loss of the value network are obtained by extracting batch data from the experience replay pool to update the value network parameters; based on the gradient ascent method, the optimization target and loss of the policy network are obtained to update the policy network parameters; based on the updated value network parameters and policy network parameters, the parameters of the target policy network and target value network are soft-updated.

[0023] Optionally, the process of training each routing network based on the policy gradient descent method further includes:

[0024] A target weight is set for each task based on the task difficulty, and the optimization targets of the value network and the policy network are adjusted based on the target weight.

[0025] Optionally, experimental tasks selected in the multi-task learning environment include: a switch task, a rotation task, a pushing task, a grasping task, an inserting task, a pulling out task, a dumping task, a stacking task, a pick-up and place task, and a knocking task of robot operation.

[0026] Optionally, the process of activating the corresponding expert module based on the trained routing network includes:

[0027] The input data is processed based on a fully connected neural network to obtain feature representation, and then the key vector of each expert module is obtained; the dot product operation is performed on the feature representation of the input data and the key vector of the expert module to obtain the activation score; the two expert modules with the highest activation scores are selected, and the activation weights are calculated respectively, and the activation weights are fused in a weighted manner as the final output.

[0028] Optionally, an evaluation of the decoupling performance of the expert module is also included:

[0029] The average success rate is selected as the evaluation indicator, and then MT10 environment task evaluation, cross-task evaluation and MT50 environment similar task evaluation are carried out.

[0030] The present invention also provides an electronic device, comprising: a memory and a processor; the memory is used to store a program; the processor is used to execute the program to implement each step of the multi-task reinforcement learning method based on the emergent hybrid expert model.

[0031] The present invention also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the various steps of the multi-task reinforcement learning method based on the emergent hybrid expert model are implemented.

[0032] Compared with the prior art, the present invention has the following advantages and technical effects:

[0033] The present invention comprises the following steps: constructing a multi-task reinforcement learning network, including a plurality of feedforward neural networks, each of which is connected to a routing network; performing cluster analysis on each feedforward neural network and dividing it into a plurality of expert modules; training each routing network based on a policy gradient descent method, and updating the parameters of the corresponding expert module; selecting experimental tasks in a multi-task learning environment; for each experimental task, activating the corresponding expert module based on the trained routing network to perform decoupling processing, and completing the multi-task reinforcement learning.

[0034] The present invention realizes task decoupling through an emergent expert module, reduces interference in the multi-task learning process, and improves learning efficiency; the method can effectively share parameters, learn each task in a personalized way, and improve performance and generalization ability; its structure is flexible and adaptable, and it can alleviate gradient conflicts and ensure smooth training, providing an efficient and stable optimization strategy for multi-task learning, and effectively alleviating the problem of parameter sharing and mutual interference between tasks in multi-task learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0037] Figure 2 This is a diagram showing the architecture of a multi-task reinforcement learning method according to an embodiment of the present invention;

[0038] Figure 3 A feedforward neural network architecture diagram of an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the K-means algorithm process of an embodiment of the present invention;

[0040] Figure 5 A detailed flow chart of the K-means algorithm according to an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of a feedforward neural network clustering expert generation according to an embodiment of the present invention;

[0042] Figure 7 Schematic diagram of the forward propagation calculation flow of a feedforward neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Embodiment 1

[0046] This embodiment provides a multi-task reinforcement learning method based on an emergent hybrid expert model, comprising the following steps:

[0047] Constructing a multi-task reinforcement learning network, wherein the multi-task reinforcement learning network includes a plurality of feedforward neural networks, each of which is connected to a routing network;

[0048] Cluster analysis is performed on each feedforward neural network, which is divided into several expert modules;

[0049] Train each routing network based on the policy gradient descent method and update the parameters of the corresponding expert module;

[0050] Select experimental tasks in a multi-task learning environment;

[0051] For each experimental task, the corresponding expert module is activated based on the trained routing network to perform decoupling processing and complete multi-task reinforcement learning.

[0052] Glossary:

[0053] Multi-task Reinforcement Learning Network: This is a machine learning architecture designed to learn multiple different tasks at the same time, usually used in the field of reinforcement learning. This network can improve learning efficiency and reduce overfitting by sharing certain common features or knowledge.

[0054] Feedforward Neural Network: This is the simplest type of artificial neural network in which information flows in only one direction, from the input layer to the hidden layer and then to the output layer. There are no reverse connections or self-feedback loops.

[0055] Routing Network: In multi-task learning, the routing network is responsible for dynamically selecting and activating the corresponding expert modules based on the input data. It is usually located in front of the feedforward neural network to determine which expert modules will participate in the processing of the current task.

[0056] Clustering Analysis: This is an unsupervised learning method used to divide samples in a data set into several clusters or groups so that the samples within the same cluster have high similarity, while the samples between different clusters have low similarity.

[0057] Expert Module: In multi-task learning, an expert module refers to a neural network module that specializes in processing a specific task or a subset of tasks. Each expert module may perform better on a specific task, and the task of the routing network is to select the appropriate expert module to process a specific input.

[0058] Policy Gradient Descent Method: This is a reinforcement learning algorithm that learns the optimal policy by directly performing gradient descent optimization on the parameters of the policy (i.e., decision rule). This method allows the algorithm to search in the policy space to find the policy that maximizes the expected return.

[0059] Decoupling Processing: In multi-task learning, decoupling processing refers to separating the processing processes of different tasks so that each task can be processed independently, reducing interference between tasks and improving learning efficiency.

[0060] As a specific embodiment, Figure 1 The main steps of the multi-task reinforcement learning method based on the emergent hybrid expert model in this embodiment are given as an example. Figure 1 As shown, the multi-task reinforcement learning method based on the emergent hybrid expert model in this embodiment includes the following steps:

[0061] Step 100, constructing an emergent hybrid expert model architecture;

[0062] The step 100 further comprises the following steps:

[0063] Step 110, first design a main structure of a multi-task reinforcement learning network, such as Figure 2 As shown in the figure, the structure includes an input processing layer network and multiple layers of the same network. The input processing layer is a fully connected neural network, followed by 4 identical FNNs, i.e. feedforward neural networks, and each FNN is preceded by a routing network. The specific network architecture of FNN is shown in the figure below. Figure 3 As shown, the entire network consists of three parts, the first layer of the network is used as the key, the second layer is the activation layer, and the third layer of the network is used as the value. There are many options for the activation layer. This embodiment selects the ReLU function, which is simple to calculate and does not cause gradient disappearance in the positive value area.

[0064] Step 120, based on the FNN in step 110, split each FNN into multiple expert modules, each expert module is responsible for learning a specific function and rule to support the completion of the overall task, and the routing network is responsible for dynamically selecting the appropriate expert module according to the requirements of the current task.

[0065] Step 130, determine the tasks to be experimented. Meta-World (MT10 and MT50) is a multi-task learning environment designed for robot operation. It aims to promote the algorithm trained in the simulation environment to be better transferred to the real-world robot. The environment contains a series of different robot operation tasks, which differ in the operation objects, target states, operation actions, etc. For example, tasks may include opening doors, turning door handles, pushing objects, grasping and moving objects, etc., reflecting the diversity of tasks; the MT10 environment contains 10 unique tasks, while the MT50 environment contains 50 unique tasks. These tasks are designed to be trained together to promote the generalization ability of the algorithm across multiple tasks. Specifically, the MT10 task in the Meta-World environment is selected for the experiment, and the tasks included are as follows:

[0066] Switching tasks: include opening and closing various types of switches.

[0067] Rotational tasks: For example, turning a doorknob or knob to a specific position.

[0068] Pushing task: Push the object to the specified target location.

[0069] Grasping tasks: Grasp and move objects using a robot gripper.

[0070] Insertion task: insert an object into a specific slot of another object.

[0071] Pull task: Pull an object from a slot.

[0072] Tipping Task: Tip the container to remove the contents.

[0073] Stacking task: stack one object on top of another.

[0074] Pick up and place tasks: Pick up an object and place it in a specific location or container.

[0075] Tapping task: Tapping an object with the robot end effector.

[0076] These tasks are designed to be both challenging and able to evaluate the robot's flexibility and adaptability in performing complex operations. Each task has its own specific goals and success conditions, which are critical to evaluating the performance of the algorithm.

[0077] Step 200, initializing model parameters and building an expert module;

[0078] The step 200 further comprises the following steps:

[0079] Step 210, initializing the parameters of the entire network, including the FNN and the routing network.

[0080] For each FNN, the weights and bias parameters are initialized randomly. The weights can be drawn from a small normal distribution (e.g., with mean 0 and standard deviation 0.01), while the biases are initialized to 0. Similar to FNN, the weights and bias parameters of the routing network should also be initialized randomly, with the weights drawn from a normal distribution and the biases initialized to zero. The initialization of the routing network should take into account that it needs to learn how to select the appropriate expert module based on the input data.

[0081] Step 220, based on the FNN and routing network after the parameters are initialized in step 210, the key of each FNN is clustered using the K-means method. The K-means method is a clustering method, and its algorithm process is as follows: Figure 4 Each FNN is set to 256 dimensions. In this embodiment, each FNN is evenly divided into 4 expert modules of the same size, that is, each expert module is composed of 64-dimensional neurons. Therefore, the keys of the FNN are clustered into 4 classes of size 64. The specific clustering process is as follows:

[0082] First, determine the sample set and the number of clusters k to be divided. In this case, k is 4, that is, the FNN keys are clustered into 4 classes, and the number of samples contained in each class is equal;

[0083] Randomly initialize 4 data center points from the sample set as the centers of the 4 classes;

[0084] Iterate each sample and calculate the distance between each sample and the four data centers;

[0085] Mark the sample as the category cluster closest to the sample;

[0086] After the first round of data traversal is completed, all samples have found their own clusters, but this is not the final category, and the second round of iteration is required;

[0087] Before the second iteration, the centroid within each cluster needs to be updated;

[0088] Calculate the centroid within each cluster;

[0089] The steps of the first and second iterations are repeated until the predetermined number of iterations is reached or the data within the cluster does not change in two consecutive iterations.

[0090] Figure 5 This is a detailed flowchart of the k-means algorithm, which is similar to the above Figure 4 The algorithm process is consistent.

[0091] Step 230, based on the clustering result in step 220, neurons in the same class are combined into an expert module, such as Figure 6 As shown, neurons of the same color are of the same class and are grouped into one expert. This is a simplified schematic diagram, where the dimension of the FNN is 9, and it is clustered into three expert modules, namely expert 1, expert 2, and expert 3. In this embodiment, the FNN is actually 256-dimensional and will be divided into four different expert modules, each of which has the same dimension of 64.

[0092] Based on the expert modules constructed above, it is ensured that the model can show certain modular characteristics during the pre-training process, laying the foundation for subsequent task decoupling.

[0093] Step 300, multi-task learning and expert module selection;

[0094] The step 300 further includes the following steps:

[0095] Step 310: for each task, dynamically activate the corresponding expert module through the routing network according to the input of the task. Figure 7 As shown, the routing function is realized through a gating mechanism. The following is the detailed calculation process:

[0096] Calculate the feature representation of the input data, such as Figure 2 , the input of the task will pass through an input processing layer, which is a fully connected neural network, to obtain the processed input x.

[0097] For each expert n, its key vector is calculated. By averaging the neurons in the expert, a key vector Kn that can represent the expert is obtained, and the key vectors of all experts are obtained in turn to match the input data x.

[0098] To calculate the activation score, we use the feature representation x of the input data and the key vector Kn of each expert to perform a dot product operation to obtain the activation score of each expert. For the input data x and the key vector Kn of expert n, the activation score gn(x) can be calculated as: gn(x) = x·Kn.

[0099] Based on the calculated activation scores, the Top-2 experts, i.e., the two expert modules with the highest activation scores, are selected for activation.

[0100] Based on the activation scores S1 and S2 of the top-2 experts E1 and E2, the activation weights of the two experts are calculated using the softmax function. The specific formula is:

[0101]

[0102] The weight of expert E1 is:

[0103] The weight of expert E2 is:

[0104] After softmax processing, the sum of the two weights w1 and w2 is 1.

[0105] If the final outputs of experts E1 and E2 are o1 and o2 respectively, then the final weighted action output of FNN is:

[0106] y=w1·o1+w2·o2;

[0107] Step 320, based on the routing network in step 310, training is started, using the Soft Actor-Critic (SAC) algorithm in reinforcement learning, a policy gradient descent method, to learn the optimal policy by alternately updating the policy network (actor) and the value network (critic).

[0108] The SAC algorithm consists of the following main parts:

[0109] Two value networks (Q networks): used to estimate the state-action value function.

[0110] A policy network: used to generate actions.

[0111] Two target Q networks: used to calculate target values ​​and reduce fluctuations during training.

[0112] A target policy network: used for policy evaluation in the target Q network.

[0113] The SAC algorithm process begins with initialization:

[0114] Initialize the parameters θ1 and θ2 of the two Q networks (Q_θ1 and Q_θ2).

[0115] Initialize the policy network π φ The parameter φ.

[0116] Initialize the parameters θ1' and θ2' of the target Q network (Q_θ1' and Q_θ2') to be the same as the initial parameters of the Q network.

[0117] Initialize the target policy network π φ 'Parameter φ', making it the same as the initial parameter of the policy network.

[0118] Initialize experience replay pool D.

[0119] The specific steps of the SAC algorithm are as follows:

[0120] Sampling: Sample a state s from the environment and use the policy network π φ Take action a.

[0121] Execute action: Perform action a in the environment and get reward and new state s'.

[0122] Storing experience: Store the transfer tuple (s, a, r, s', done) in the experience replay pool D.

[0123] Randomly draw a batch of experience from D: B = {(s i ,a i ,r i ,s' i ,done i )};

[0124] Update Q network:

[0125] For each sample (i) in batch data B, calculate the target value:

[0126]

[0127] The Q network parameters θ1 and θ2 are updated by minimizing the following loss:

[0128]

[0129] Update policy network:

[0130] Update the policy network parameters φ using gradient ascent to maximize the following objective:

[0131]

[0132] Update the target network:

[0133] Soft update parameters of target Q network and target policy network:

[0134] θ' i ←τθ i +(1-τ)θ' i ,

[0135] φ'←τφ+(1-τ)φ';

[0136] Here τ is a small constant, which is set to 0.005.

[0137] During the learning process, only the parameters of the activated expert modules are updated, and the parameters of other expert modules are kept unchanged to achieve decoupling between tasks.

[0138] By training the routing network, it is possible to more effectively dynamically select and activate expert modules based on the input of the task, and to achieve balanced optimization between tasks in a multi-task learning environment. This training method helps to improve the flexibility and efficiency of the entire network architecture when handling multiple tasks.

[0139] Step 330, balanced optimization of tasks. Since simpler tasks usually converge faster than difficult tasks, different weights are set for different tasks, and these weights are automatically learned. Specifically, the loss weight is directly related to the temperature parameter α in the SAC algorithm, that is, the parameter α in step 5 of the specific steps of the SAC algorithm in step 320. When log(π φ When the value of (a|s)) becomes larger, it means that the entropy becomes smaller, and α will increase to encourage exploration. On the contrary, if log(π φ A smaller value of (a|s) means that the entropy changes more, and α decreases, reducing the degree of exploration. For M different tasks, set different temperature parameters for each task. The target weight w of task i i With negative α i is proportional to the exponent of , as shown in the following formula:

[0140]

[0141] Then, according to this weight, the optimization target of the Q network in step 5 and the target of the strategy optimization in step 6 in the specific steps of the SAC algorithm in step 320 are adjusted.

[0142] Step 400, model evaluation:

[0143] Evaluate the performance of the model on various tasks to ensure the effectiveness of decoupled learning.

[0144] 1. First, define the evaluation indicators. Due to the characteristics of the experimental task, the average success rate is selected as the indicator for evaluating model performance.

[0145] 2. Secondly, we evaluate in the MT10 environment. For each task in the MT10 environment, we run the model independently and collect performance data. We record the performance of the model on each task and compare it with the pre-set performance threshold.

[0146] 3. Conduct cross-task evaluation to evaluate the model’s ability to transfer to similar tasks in the MT10 environment, ensure that disentangled learning can promote cross-task knowledge sharing, and analyze the performance differences of the model between different tasks to confirm whether the disentanglement is effective.

[0147] 4. Perform similar task evaluation in the MT50 environment. Select tasks similar to MT10 in the MT50 environment for evaluation, and collect performance data of the model on similar tasks in MT50.

[0148] Through the above four steps, this embodiment implements a multi-task reinforcement learning method based on an emergent hybrid expert model, which can effectively reduce interference between tasks and improve the learning efficiency and performance of each task.

[0149] Embodiment 2

[0150] This embodiment also provides an electronic device, including: a memory and a processor; the memory is used to store a program; the processor is used to execute the program to implement each step of the multi-task reinforcement learning method based on the emergent hybrid expert model.

[0151] Embodiment 3

[0152] This embodiment also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the multi-task reinforcement learning method based on the emergent hybrid expert model is implemented.

[0153] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-task reinforcement learning method based on an emergent hybrid expert model, characterized in that: The following steps are involved: Constructing a multi-task reinforcement learning network, wherein the multi-task reinforcement learning network includes a plurality of feedforward neural networks, each of which is connected to a routing network; Cluster analysis is performed on each feedforward neural network, which is divided into several expert modules; Train each routing network based on the policy gradient descent method and update the parameters of the corresponding expert module; Select experimental tasks in a multi-task learning environment; For each experimental task, the corresponding expert module is activated based on the trained routing network to perform decoupling processing and complete multi-task reinforcement learning.

2. The multi-task reinforcement learning method based on the emergent hybrid expert model according to claim 1, characterized in that: The multi-task reinforcement learning network also includes a layer of fully connected neural network, and the fully connected neural network is connected to a plurality of feedforward neural networks; wherein each feedforward neural network includes three layers of networks, the first layer of the network is used as a key, the second layer of the network is used as an activation layer, and the third layer of the network is used as a value.

3. The multi-task reinforcement learning method based on emergent hybrid expert model according to claim 1, characterized in that: The process of clustering each feedforward neural network and dividing it into several expert modules includes: For each feedforward neural network, the number of sample sets and the number of category clusters to be divided are preset, and the number of samples in each category cluster is the same; Randomly select a number of data center points from the sample set as the center of each category cluster; Perform the first round of iterations for each sample, obtain the distance of each sample to each data center point and sort them; Divide the samples into the closest category clusters; Update the centroid of each category cluster and perform a second round of iteration for each sample; When the number of iterations is reached, several clusters of categories are obtained after clustering; The samples in each category cluster after clustering are combined into an expert module.

4. The multi-task reinforcement learning method based on emergent hybrid expert model according to claim 1, characterized in that: The process of training each routing network based on the policy gradient descent method includes: The policy gradient descent method adopts the SAC algorithm, which includes a policy network, two value networks, a target policy network and two target value networks; Based on the SAC algorithm, the optimization target and loss of the value network are obtained by extracting batch data from the experience replay pool to update the value network parameters; based on the gradient ascent method, the optimization target and loss of the policy network are obtained to update the policy network parameters; based on the updated value network parameters and policy network parameters, the parameters of the target policy network and target value network are soft-updated.

5. The multi-task reinforcement learning method based on emergent hybrid expert model according to claim 4, characterized in that: The process of training each routing network based on the policy gradient descent method also includes: A target weight is set for each task based on the task difficulty, and the optimization targets of the value network and the policy network are adjusted based on the target weight.

6. The multi-task reinforcement learning method based on emergent hybrid expert model according to claim 1, characterized in that: The experimental tasks selected in the multi-task learning environment include: robot operation switch task, rotation task, pushing task, grasping task, insertion task, extraction task, dumping task, stacking task, pick-up and place task and knocking task.

7. The multi-task reinforcement learning method based on emergent hybrid expert model according to claim 1, characterized in that: The process of activating the corresponding expert module based on the trained routing network includes: The input data is processed based on a fully connected neural network to obtain feature representation, and then the key vector of each expert module is obtained; the dot product operation is performed on the feature representation of the input data and the key vector of the expert module to obtain the activation score; the two expert modules with the highest activation scores are selected, and the activation weights are calculated respectively, and the activation weights are fused in a weighted manner as the final output.

8. The multi-task reinforcement learning method based on emergent hybrid expert model according to claim 1, characterized in that: It also includes an evaluation of the decoupling performance of the expert modules: The average success rate is selected as the evaluation indicator, and then MT10 environment task evaluation, cross-task evaluation and MT50 environment similar task evaluation are carried out.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the multi-task reinforcement learning method based on the emergent hybrid expert model as described in any one of claims 1-8.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the multi-task reinforcement learning method based on an emergent hybrid expert model as described in any one of claims 1 to 8 is implemented.

Images