Motion control method for multi-task and multi-robot based on deep reinforcement learning
By using independent Q networks and joint Q networks in multi-task multi-robot system, combining context information and observation data to predict the value of decision-making actions, the problem of insufficient generalization ability in multi-task multi-robot motion control is solved, and task processing is improved efficiency.
Patent Information
- Application Number
- CN202411217554.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The motion control method of multi-task multi-robot based on deep reinforcement learning is insufficient in generalization capabilities when facing different tasks, resulting in low efficiency in handling multi-tasks.
By determining context information and current observation data, using independent Q networks and joint Q networks to predict the value of decision actions, each robot can independently make decisions and execute individual decision actions. The independent Q network includes multi-layer perceptrons, pre-trained language models, and crossovers Attention mechanism module and self-attention mechanism module, the joint Q network includes a transformer module and a combined network, and iterative training is performed based on the target loss function.
The efficiency of the multi-task processing process is improved, allowing multiple robots to accurately execute decision-making actions under different tasks, and improving the system's generalization ability and task completion speed.
Smart Images

Figure CN118752492B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot motion control technology, and in particular to a multi-task multi-robot motion control method based on deep reinforcement learning. Background Art
[0002] Multi-robot motion control refers to the design of reasonable control strategies and algorithms to enable multiple robots to maintain coordination and achieve efficient and accurate motion when performing common tasks. Deep Reinforcement Learning (DRL) is a technology that combines deep learning and reinforcement learning. In the motion control of multi-task multi-robots, DRL can be applied to path planning, action decision-making, task allocation and other aspects.
[0003] In related technologies, multi-task and multi-robot motion control methods based on deep reinforcement learning are often limited to specific single tasks. When faced with different tasks, the neural network model usually needs to be retrained, and its generalization ability is insufficient, resulting in low efficiency in processing multiple tasks. Summary of the invention
[0004] The present invention provides a multi-task multi-robot motion control method based on deep reinforcement learning, which is used to solve the problem that the multi-task multi-robot motion control method based on deep reinforcement learning in the related art has insufficient generalization ability when facing different tasks, resulting in low efficiency in processing multiple tasks.
[0005] The present invention provides a multi-task multi-robot motion control method based on deep reinforcement learning, comprising the following steps: determining context information, wherein the context information is used to indicate tasks to be performed by multiple robots; sending the context information to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision-making action; wherein the decision action corresponding to an individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks, and obstacles; inputting the current observation data and the context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the values of multiple decision actions based on the input information; and determining the individual decision action from the multiple decision actions based on the individual value estimation vector.
[0006] According to a multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, the independent Q network includes a first multi-layer perceptron, a second multi-layer perceptron, a pre-trained language model, a first cross-attention mechanism module, a second cross-attention mechanism module and a plurality of self-attention mechanism modules connected end to end; the current observation data and the context information are input into the independent Q network to obtain the individual value estimation vector output by the independent Q network, including: inputting the current observation data into the first multi-layer perceptron to obtain the current feature embedding output by the first multi-layer perceptron; inputting the context information into the pre-trained language model to obtain the task encoding output by the pre-trained language model; using the current feature embedding as The query variable and the task code are input into the first cross-attention mechanism module as the key and value to obtain the specific task representation output by the first cross-attention mechanism module; the specific task representation is input into the multiple self-attention mechanism modules connected end to end to obtain the entity features output by the multiple self-attention mechanism modules connected end to end, wherein the self-attention mechanism module uses the same input as the query variable, key and value; the entity features are input into the second cross-attention mechanism module as the key and value and the task code as the query variable to obtain the hidden state output by the second cross-attention mechanism module; the hidden state is input into the second multi-layer perceptron to obtain the individual value estimation vector output by the second multi-layer perceptron.
[0007] According to a multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, the individual value estimation vector includes a group of individual values, each individual value in the group of individual values corresponds to a decision action among the multiple decision actions; determining the individual decision action from the multiple decision actions according to the individual value estimation vector includes: determining the decision action corresponding to the largest individual value in the group of individual values as the individual decision action.
[0008] According to a multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, before determining the context information, the method also includes: obtaining historical data, wherein the historical data includes the context information, observation data and decision actions performed by the multiple robots in the historical time period; obtaining task regret, wherein the task regret is used to represent the gap between the actual reward and the optimal reward of each task in the multiple tasks, and the multiple tasks are the tasks performed by the multiple robots in the historical time period; determining multiple task learning weight parameters according to the task regret, wherein the multiple task learning weight parameters correspond one-to-one to the multiple tasks; sampling the historical data based on the multiple task learning weight parameters to obtain training data; initializing the parameters of the robot network, wherein the robot network includes multiple independent Q networks with shared parameters, and the multiple independent Q networks correspond one-to-one to the multiple robots; using the target loss function to iteratively train the robot network based on the training data until the training termination condition is reached to obtain a trained robot network.
[0009] According to a multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, the historical data also includes the global state and reward signal at all moments in the historical time period, the robot network also includes a joint Q network, wherein the global state is used to represent the feature information of all robots; the use of the target loss function to iteratively train the robot network based on the training data includes: acquiring first training data and second training data based on the training data, wherein the first training data includes target context information, a first global state at a first moment, observation data and decision actions of each of the multiple robots at the first moment, the second training data includes the target context information, a second global state at a second moment and a target reward signal, and observation data of each of the multiple robots at the second moment, the second moment being the next moment of the first moment; inputting the first training data into the multiple independent Q network, obtain multiple first value estimation vectors output by the multiple independent Q networks; determine multiple first values from the multiple first value estimation vectors based on the decision action of each robot in the multiple robots at the first moment; input the multiple first values, the target context information and the first global state into the joint Q network to obtain the first joint value output by the joint Q network; input the second training data into the multiple independent Q networks to obtain the multiple second value estimation vectors output by the multiple independent Q networks; determine multiple optimal second values based on the multiple second value estimation vectors; input the multiple optimal second values, the target context information and the second global state into the joint Q network to obtain the second joint value output by the joint Q network; determine the training target according to the second joint value and the target reward signal; use the target loss function to iteratively train the robot network based on the training target and the first joint value.
[0010] According to a multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, the joint Q network includes a converter module and a combination network; the multiple first values, the target context information and the first global state are input into the joint Q network to obtain the first joint value output by the joint Q network, including: inputting the target loop state, the target context information, the first global state and multiple first hidden states into the converter module to obtain the first weight and the first deviation output by the converter module, wherein the target loop state is the loop state output by the converter module at the previous moment, and the multiple first hidden states are the intermediate values obtained by the multiple independent Q networks in the process of obtaining the multiple first value estimation vectors according to the first training data; the multiple first values are input into the combination network to obtain the first joint value output by the combination network based on the first weight and the first deviation.
[0011] The present invention also provides a multi-task multi-robot motion control system based on deep reinforcement learning, comprising the following modules: a determination module, used to determine context information, wherein the context information is used to indicate tasks to be performed by multiple robots; a sending module, used to send the context information to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision-making action; wherein the decision action corresponding to the individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks and obstacles; inputting the current observation data and the context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the value of multiple decision actions based on the input information; determining the individual decision action from the multiple decision actions based on the individual value estimation vector.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multi-task multi-robot motion control method based on deep reinforcement learning as described above is implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multi-task multi-robot motion control method based on deep reinforcement learning as described in any one of the above.
[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the multi-task multi-robot motion control methods based on deep reinforcement learning as described above.
[0015] The multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, after obtaining context information, regards each robot among the multiple robots as an individual robot, inputs the context information and the current observation data corresponding to the individual robot into an independent Q network, obtains the value estimation vector corresponding to the individual robot, and based on the value estimation vector, the individual robot can accurately determine the individual decision-making action corresponding to the task to be performed. At the same time, each robot among the multiple robots performs the corresponding steps, and the type and number of the multiple robots will not affect the determination of the individual decision-making action. For different tasks, the method of the present invention can control the individual robots to perform the corresponding individual decision-making actions to complete the tasks to be performed, thereby solving the problem that the multi-task multi-robot motion control method based on deep reinforcement learning in the related art is insufficient in generalization ability when facing different tasks, resulting in low efficiency in processing multiple tasks, thereby improving the efficiency of the multi-task processing process. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 This is one of the flow charts of the multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention.
[0018] Figure 2 It is a multi-task scene diagram of the multi-particle simulation environment provided by the present invention.
[0019] Figure 3 It is a structural schematic diagram of the independent Q network provided by the present invention.
[0020] Figure 4 This is the second flow chart of the multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention.
[0021] Figure 5 It is an example training curve diagram provided by the present invention.
[0022] Figure 6 It is a schematic diagram of the multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention.
[0023] Figure 7 It is a structural diagram of the joint Q network provided by the present invention.
[0024] Figure 8 It is a structural schematic diagram of the multi-task multi-robot motion control system based on deep reinforcement learning provided by the present invention.
[0025] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0028] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0029] Multi-robot motion control refers to the design of reasonable control strategies and algorithms to enable multiple robots to maintain coordination and achieve efficient and accurate motion when performing common tasks. Deep Reinforcement Learning (DRL) is a technology that combines deep learning and reinforcement learning. In the motion control of multi-task multi-robots, DRL can be applied to path planning, action decision-making, task allocation and other aspects.
[0030] In the related technology, for multiple robots of different types and numbers, the same task objectives are usually regarded as different tasks. The multi-task multi-robot motion control method based on deep reinforcement learning is often limited to a specific single task. When faced with different tasks, the neural network model usually needs to be retrained, and its generalization ability is insufficient. It is impossible to use a single model to handle multi-task problems with different objectives, resulting in low efficiency in processing tasks.
[0031] In order to at least solve some of the above problems, the following Figure 1-Figure 9 The multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention is explained.
[0032] Figure 1 is one of the flow charts of the multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, such as Figure 1 As shown, the steps include but are not limited to the following steps 101-102.
[0033] Step 101 : determining context information, wherein the context information is used to indicate tasks to be performed by multiple robots.
[0034] The multi-task multi-robot motion control method based on deep reinforcement learning provided in the present embodiment can be executed by a multi-task multi-robot system (also referred to as a multi-task multi-agent system). The multi-task multi-robot system is a complex system covering multiple robots, which is characterized in that the multi-robot system is required to perform different tasks in different scenarios. Each task requires the robots in the system to work together to optimize the overall performance of the current task. Under this framework, each robot has independent cognitive and execution capabilities, and through interaction with other robots, they jointly solve the current task; the multi-task multi-robot motion control method based on deep reinforcement learning provided in the present embodiment can be applied to a hardware environment including multiple robots and servers, each of the multiple robots is communicated with the server, and the multi-task multi-robot system is integrated on the server. The multi-task multi-robot motion control method based on deep reinforcement learning in the present embodiment can be executed by the server.
[0035] The multi-task multi-robot motion control method based on deep reinforcement learning provided in this embodiment can be applied to the scenario of performing motion control on multiple robots based on tasks to be performed to complete the tasks to be performed.
[0036] For example, Figure 2 is a multi-task scene diagram of the multi-particle simulation environment provided by the present invention, such as Figure 2 As shown, the multi-task multi-robot motion control method based on deep reinforcement learning in this embodiment can be applied to a multiple particle simulation environment (Multiple Particle Environment, MPE), which may include Figure 2 The multiple tasks shown (the task to be executed in this embodiment can be any one of the multiple tasks), Figure 2 The robot in corresponds to the multiple robots in this embodiment.
[0037] Below Figure 2 The multiple tasks in are described.
[0038] (a) The Landmark Capture environment is similar to the “Simple Landmark Capture” in the official MPE. This task involves controlling a robot to capture a landmark. The number of robots n can be randomly selected from {3, 4, 5}. The number of landmarks is equal to the number of robots. When a robot is closer to a landmark, it will receive a higher reward.
[0039] (b) The formation environment includes controlling the robots to form a regular polygon near the landmark. The number of robots n can be randomly selected from {3, 4, 5}, and there is only one landmark. The robots will move to the target point located on a circle with a certain radius around the landmark.
[0040] (c) Landmark Contest Environment is similar to the “Simple Landmark Contest” in the official MPE, which involves a robot (corresponding to multiple robots in this example) and an enemy robot (corresponding to Figure 2 The number of robots n can be randomly selected from {3, 4, 5}. The number of enemy robots and the number of landmarks are equal to the number of robots. A robot will be rewarded when it approaches a landmark, while an enemy will be punished when it approaches a landmark. The enemy is controlled by a predefined script strategy: moving towards the specified landmark.
[0041] (d) Chasing the Enemy The environment is similar to the “Simple Pursuit” in the official MPE. This task involves controlling a robot to chase an enemy robot. There are 4 friendly robots and 2 enemy robots. There are 2 circular obstacles on the map. Our robots are slower and larger; the enemy robots are faster and smaller. Each time our robot collides with an enemy robot, our robot will receive a certain reward. The enemy robots are controlled by a predefined script strategy: stay away from the friendly robot that is closest to the enemy robot.
[0042] (e) Escaping the enemy environment is different from chasing the enemy. It involves controlling the robot to avoid the enemy robots. There are 2 friendly robots and 4 enemy robots. There are 2 obstacles on the map. Our robot is faster and smaller; the enemy robot is slower and larger. Every time our robot collides with an enemy robot, our robot will receive a certain negative reward. The enemy robots are controlled by a predefined script strategy: move towards the nearest friendly robot.
[0043] In this embodiment, context information is used to indicate the tasks to be performed by multiple robots. The tasks to be performed here can be any of the multiple tasks in the multi-particle simulation environment in the above example. The tasks to be performed can also be related tasks for controlling the movement of multiple robots in other scenarios. For example, in the field of robot transportation of goods, multiple transport robots are controlled to carry multiple goods to a designated location.
[0044] In the case where the multi-task multi-robot motion control method based on deep reinforcement learning in this embodiment is executed by a server, the context information may be task information preset by the server, or may be task information determined by the server receiving user input.
[0045] Optionally, the context information is defined in natural language to describe pending tasks that the multiple robots need to perform.
[0046] Step 102 : sending context information to each robot among the multiple robots to drive each robot among the multiple robots to perform a corresponding decision action.
[0047] Among them, the decision action corresponding to the individual robot is the individual decision action, the individual robot is any robot among multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent the feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks, and obstacles; inputting the current observation data and context information into the independent Q network to obtain the individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the value of multiple decision actions based on the input information; and determining the individual decision action from multiple decision actions based on the individual value estimation vector.
[0048] The current observation data may be obtained by an individual robot through its own sensors or sensing devices such as cameras. The current observation data is used to represent the characteristic information of multiple entities observed by the individual robot at the current moment. The multiple entities include at least one of other robots, landmarks, and obstacles. Here, other robots may include robots other than the individual robot among the multiple robots. Other robots may also include robots that do not belong to the multiple robots, for example, enemy robots in the above-mentioned MPE. Landmarks are the geographic coordinates of the locations that the multiple robots need to reach. Obstacles are the inaccessible locations detected by the robots during the movement of the multiple robots.
[0049] In this embodiment, the characteristic information of the entity is composed of the speed, position and category of the entity. For example, for other robots, its characteristic information includes the speed information, position coordinate information and category information (for example, friendly robot or enemy robot) of other robots; for landmarks and obstacles, since they cannot move, the characteristic information of landmarks and obstacles only includes their position coordinate information.
[0050] It should be noted that the current observation data is the feature information of multiple entities observed by the individual robot at the current moment, and the individual robot determines the decision action for the next moment based on the current observation data.
[0051] Furthermore, the individual robot inputs the current observation data and context information into an independent Q-network to obtain an individual value estimation vector output by the independent Q-network; Independent Q-Network in reinforcement learning, especially in the context of Multi-Agent Systems (MAS, corresponding to the multi-robot system in this embodiment) and Deep Reinforcement Learning (DRL), refers to maintaining a Q-network for each robot separately to evaluate the value of the robot's action in a given state. This design allows each robot to learn and optimize its strategy based on its own local observations and experience without relying on information from other robots.
[0052] Among them, the independent Q network can predict the value of multiple decision actions based on the input current observation data and context information, that is, the individual value estimation vector is used to express the expected return of the individual robot performing multiple decision actions in the decision-making process; further, the individual decision action is determined from multiple decision actions according to the individual value estimation vector, and a set of vector values contained in the individual value estimation vector can correspond to multiple decision actions respectively. The value of each decision action in the multiple decision actions can be determined based on the size of the vector value in the individual value estimation vector, thereby determining the individual decision action.
[0053] It should be noted that multiple decision actions are pre-set decision actions. For example, in MPE, the decision actions of individual robots may include up, down, left, right, and static (based on Figure 2 The five actions are shown in the top view perspective.
[0054] After determining the individual decision action, the individual robot performs the individual decision action, and each of the multiple robots performs the corresponding decision action to complete the task to be performed indicated by the context information; it should be noted that the completion of the task to be performed requires a certain amount of time. After each of the multiple robots performs its own individual decision action, the completion progress of the task to be performed can be improved, but it may not be able to directly complete the task to be performed. For example, to control the robot to move to a landmark position, the robot may need to move according to the individual decision actions obtained at multiple times until it finally reaches the landmark position.
[0055] Through the embodiments provided by the present application, context information is determined, wherein the context information is used to indicate tasks to be performed by multiple robots; the context information is sent to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision action; wherein the decision action corresponding to the individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks, and obstacles; inputting the current observation data and context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the values of multiple decision actions based on the input information; determining the individual decision action from the multiple decision actions based on the individual value estimation vector solves the problem that the motion control method of multiple tasks and multiple robots based on deep reinforcement learning in the related art is insufficient in generalization ability when facing different tasks, resulting in low efficiency in processing multiple tasks, thereby improving the efficiency of the multi-task processing process.
[0056] As an optional solution, the independent Q network includes a first multi-layer perceptron, a second multi-layer perceptron, a pre-trained language model, a first cross-attention mechanism module, a second cross-attention mechanism module, and a plurality of end-to-end self-attention mechanism modules.
[0057] The current observation data and context information are input into the independent Q network to obtain the individual value estimation vector output by the independent Q network, including S11-S16.
[0058] S11, input the current observation data into the first multi-layer perceptron to obtain the current feature embedding output by the first multi-layer perceptron.
[0059] S12, inputting the context information into the pre-trained language model to obtain the task encoding output by the pre-trained language model.
[0060] S13, input the current feature embedding as the query variable and the task encoding as the key and value into the first cross-attention mechanism module to obtain the specific task representation output by the first cross-attention mechanism module.
[0061] S14, inputting the specific task representation into multiple end-to-end self-attention mechanism modules to obtain entity features output by multiple end-to-end self-attention mechanism modules, wherein the self-attention mechanism modules use the same input as query variables, keys, and values.
[0062] S15, input the entity features as keys and values and the task codes as query variables into the second cross-attention mechanism module to obtain the hidden state output by the second cross-attention mechanism module.
[0063] S16, input the hidden state into the second multi-layer perceptron to obtain the individual value estimation vector output by the second multi-layer perceptron.
[0064] Figure 3 Schematic diagram of the structure of the independent Q network provided by the present invention, such as Figure 3 As shown, the independent Q network includes a first multi-layer perceptron, a second multi-layer perceptron, a pre-trained language model, a first cross attention mechanism module, a second cross attention mechanism module, and a plurality of end-to-end self-attention mechanism modules. Figure 3 This embodiment is described.
[0065] After the current observation data and context information are input into the independent Q network, the current observation data (corresponding to Figure 3 Individual observations in o a ) is input into the first multi-layer perceptron to obtain the current feature embedding output by the first multi-layer perceptron; here, Figure 3 The observation data shown includes the characteristic information of entities such as robots, enemies, and landmarks (only for example, the scope of observation data is not limited); the current observation data is ,in is the feature information of m entities, a is used to indicate that the individual robot is the ath robot among multiple robots. Figure 3 In the example, the value of m is 6.
[0066] The multilayer perceptron (MLP) in this embodiment is a basic feedforward artificial neural network model, which expands the function of a single-layer neural network by introducing one or more hidden layers, so that the neural network can handle more complex tasks. Here, the current observation data is input into the first multilayer perceptron to obtain the current feature embedding output by the first multilayer perceptron, that is, the current observation data o a Generate the current feature embedding of the individual robot through MLP mapping .
[0067] The context information is input into the pre-trained language model to obtain the task encoding output by the pre-trained language model; here, the pre-trained language model is a deep learning model trained on a large amount of text data. The pre-trained language model in this embodiment processes the task context information c to obtain the task encoding .
[0068] Since each entity may have different effects on the actions that individual robots need to perform in different tasks, the current feature embedding is used as the query variable and the task encoding is used as the key and value to input into the first cross-attention mechanism module to obtain the specific task representation output by the first cross-attention mechanism module. , which can be specifically expressed as:
[0069]
[0070] It should be noted that the Cross-Attention Mechanism is a commonly used mechanism in the fields of natural language processing and computer vision. Here, since the attention calculation does not depend on the input sequence, the current features of any length can be embedded into the same network.
[0071] Here, the attention mechanism is explained. The input of the attention mechanism is a set of query variables ,key Sum , where m is the length of the query, n is the length of the key and value, , , They represent their hidden dimensions respectively, R represents a real number, and R plus a superscript represents the corresponding real vector or real matrix. The output of the attention module can be specifically expressed as:
[0072]
[0073] in, , and are three learnable parameterized matrices. If the query comes from the same input source as the key and value, it is a self-attention module. If the query comes from a different input source than the key and value, it is a cross-attention module. Q, K, and V represent the query vector, key vector, and value vector in the attention mechanism calculation process, respectively. K T Represents the transpose of the key vector.
[0074] Furthermore, the specific task representation Input multiple end-to-end self-attention mechanism modules (corresponding to Figure 3 l self-attention mechanism modules in ), and obtain the entity features output by multiple self-attention mechanism modules connected end to end , which can be specifically expressed as:
[0075]
[0076] It should be noted that the self-attention mechanism module uses the same input as the query variable, key, and value.
[0077] Furthermore, the entity features As keys and values, task encoding As the query variable, it is input into the second cross-attention mechanism module to obtain the hidden state output by the second cross-attention mechanism module, which can be specifically expressed as:
[0078]
[0079] The hidden state is input into the second multi-layer perceptron to obtain the individual value estimation vector output by the second multi-layer perceptron; that is, based on the hidden state , the individual value estimation vector is obtained through MLP mapping (corresponding to Figure 3 Q a ).
[0080] Through this embodiment, based on the multi-layer perceptron and the self-attention mechanism, the individual value estimation vector corresponding to the individual robot is determined based on the current observation data and context information, which can effectively improve the accuracy and reliability of the obtained individual value estimation vector.
[0081] As an optional solution, the individual value estimation vector includes a set of individual values, and each individual value in the set of individual values corresponds to a decision action among the multiple decision actions.
[0082] Determine an individual decision action from multiple decision actions according to the individual value estimation vector, including S21.
[0083] S21, determining the decision action corresponding to the largest individual value in a group of individual values as the individual decision action.
[0084] In this embodiment, the individual value estimation vector includes a group of individual values, each individual value in a group of individual values corresponds to a decision action among multiple decision actions, the size of the individual value is positively correlated with the value of the decision action, and by comparing the sizes of the individual values in a group of individual values, the decision action corresponding to the largest individual value in a group of individual values is determined as the individual decision action, so that the decision action with the highest value can be obtained.
[0085] Through this embodiment, the accuracy of the determined individual decision-making actions can be improved.
[0086] As an optional solution, before determining the context information, the method further includes S31 - S36 .
[0087] S31, obtaining historical data, wherein the historical data includes context information, observation data, and decision actions performed by multiple robots in a historical time period.
[0088] S32, obtaining task regret, where the task regret is used to represent the gap between the actual reward and the optimal reward of each task in a plurality of tasks, and the plurality of tasks are tasks performed by a plurality of robots in a historical time period.
[0089] S33, determining a plurality of task learning weight parameters according to the task regret, wherein the plurality of task learning weight parameters correspond one-to-one to the plurality of tasks.
[0090] S34, sampling historical data based on multiple task learning weight parameters to obtain training data.
[0091] S35, initializing parameters of the robot network, wherein the robot network includes a plurality of independent Q networks with shared parameters, and the plurality of independent Q networks correspond one-to-one to the plurality of robots.
[0092] S36, using the target loss function to iteratively train the robot network based on the training data until the training termination condition is reached, thereby obtaining a trained robot network.
[0093] In this embodiment, before determining the context information, the robot network needs to be trained to obtain a trained independent Q network.
[0094] Task regret is used to represent the gap between the actual reward and the optimal reward of each task in multiple tasks, where the multiple tasks are tasks performed by multiple robots in a historical time period; optionally, task regret can be obtained by inputting the context information corresponding to the multiple tasks into a multi-task multi-robot system of multiple robots respectively, and the multi-task multi-robot system performs each of the multiple tasks, and calculates the task regret according to the task execution results; optionally, if the historical data contains the execution results of the current multi-task multi-robot system after executing multiple tasks, the task regret can be calculated based on the execution results in the historical data.
[0095] For example, the task regret can be expressed as: ,in, is the standardized performance of task c, such as winning rate, success rate or cumulative reward; here, for the cumulative reward, normalization is required:
[0096]
[0097] in, is the average cumulative reward of the random strategy. That is, for the cumulative reward, completing a task corresponds to a trajectory. There will be a reward at each moment on the trajectory. The rewards at all moments of the entire trajectory are accumulated as the cumulative reward. The cumulative rewards of multiple trajectories generated by the same task are calculated as an average value to eliminate the influence of random noise. is the cumulative reward for a single task after 200K steps of training using TransfQmix, a benchmark method for multi-agent reinforcement learning based on QMIX; It can be used as the standardized performance of the task in the task regret definition use.
[0098] It should be noted that in this embodiment, TransfQmix was used to perform single-task training for each task for 200,000 steps and 1 million steps, and its results were used as the baseline. The performance starting from 200,000 steps was used to normalize the cross-task rewards.
[0099] Multiple task learning weight parameters are determined according to task regret, wherein the multiple task learning weight parameters correspond to multiple tasks one by one; when the task regret value is large, it reflects that the performance of the task is poor. In this embodiment, a larger task learning weight can be assigned to the task with a large task regret value to balance the learning in all tasks. The formula for determining multiple task learning weight parameters according to task regret can be specifically:
[0100]
[0101] in, Determines the proportion of learning resources, i.e., experience samples, allocated to task c; is a fixed constant; a represents all tasks in multiple tasks.
[0102] Optionally, to prevent a task from having no training resources, the sampling probability (training probability) of task c is assigned as , where C represents a collection of multiple tasks, Represents the number of multiple tasks, p min Represents the minimum probability of task sampling; the sampling probability can give a higher probability for training tasks with poor performance.
[0103] Furthermore, historical data is sampled based on multiple task learning weight parameters to obtain training data; that is, the selection probability of each task is calculated based on the current learning weight, and historical data is sampled using each task selection probability to obtain training data.
[0104] The robot network is iteratively trained based on the training data using the target loss function until the training termination condition is reached to obtain the trained robot network.
[0105] Optional, Figure 4 This is the second flow chart of the multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention. Figure 4 As shown, the training process in this embodiment may further include steps 401-409.
[0106] Step 401, initialize the parameters of the robot network and initialize the memory space of the experience buffer pool. The experience buffer pool is a memory structure used to store the experience data generated during the interaction between the reinforcement learning robot and the environment. During the interaction between the robot and the environment, new experience data will be continuously generated, and these data will be collected and stored in the experience buffer pool.
[0107] Step 402, initializing learning weight parameters of multiple tasks.
[0108] Step 403: Calculate the selection probability of each task based on the learning weight parameter, and determine the current task to be trained according to each task selection probability.
[0109] In step 404, the multiple robots interact with the environment and store the generated trajectories in the experience buffer pool.
[0110] Step 405 , sample a small batch of trajectories from the experience buffer pool based on the task selection probability, and train the robot network and update its parameters according to the target loss function.
[0111] Step 406, determine whether the inner loop evaluation times have been reached, if so, execute step 407, otherwise, repeat steps 404-405.
[0112] Step 407 , evaluating the performance of each of the multiple tasks, and updating the task regret and the learning weight parameters of the multiple tasks based on the evaluation results.
[0113] Step 408, determine whether the training termination condition is met, if so, terminate the training, if not, repeat steps 403 to 407. The training termination condition includes reaching a preset number of training times or the target loss function converges.
[0114] Step 409, the training is finished, and the parameters of the robot network updated last time are used as the training results.
[0115] Figure 5 is an example training curve diagram provided by the present invention. The training steps in this embodiment are used to train a multi-robot multi-task multi-robot system. The normalized benefit curve during the training process is as follows: Figure 5 As shown; Figure 5 The rewards for different tasks of MPE during the training process and the average rewards for the five tasks of MPE are provided. Optionally, the learning time for each task in this embodiment is 200K steps.
[0116] It can be understood that the training method of this embodiment not only exhibits the best multi-task average performance, but also has the ability to benefit from different tasks. The method of this embodiment exceeds the performance of the benchmark algorithm training 200K steps. Compared with the benchmark algorithm, the training method of this embodiment can learn similar skills from related tasks, thereby improving the utilization efficiency of samples.
[0117] Through this embodiment, the training data is determined according to the task regret, and the target loss function is used to iteratively train the robot network based on the training data, which can effectively improve the performance of the robot network.
[0118] As an optional solution, the historical data also includes the global state and reward signals at all moments in the historical time period, and the robot network also includes a joint Q network, wherein the global state is used to represent the characteristic information of all robots; the robot network is iteratively trained based on the training data using the target loss function, including S41-S49.
[0119] S41, obtaining first training data and second training data based on the training data, wherein the first training data includes target context information, a first global state at a first moment, observation data and decision actions of each of the multiple robots at the first moment, and the second training data includes target context information, a second global state and a target reward signal at a second moment, observation data of each of the multiple robots at the second moment, and the second moment is the next moment of the first moment.
[0120] S42, inputting the first training data into a plurality of independent Q networks to obtain a plurality of first value estimation vectors output by the plurality of independent Q networks.
[0121] S43, determining multiple first values from multiple first value estimation vectors based on the decision action of each robot among the multiple robots at the first moment.
[0122] S44, inputting the plurality of first values, target context information, and the first global state into the joint Q network to obtain a first joint value output by the joint Q network.
[0123] S45, inputting the second training data into a plurality of independent Q networks to obtain a plurality of second value estimation vectors output by the plurality of independent Q networks.
[0124] S46, determining multiple optimal second values based on multiple second value estimation vectors.
[0125] S47, inputting the plurality of optimal second values, the target context information and the second global state into the joint Q network to obtain a second joint value output by the joint Q network. S48, determining a training target according to the second joint value and the target reward signal.
[0126] S49, iteratively train the robot network based on the training objective and the first joint value using the target loss function.
[0127] In this embodiment, the independent Q network in the robot network is trained by the joint Q network, training data and task regret under the temporal difference algorithm; Temporal difference reinforcement learning algorithm: Temporal difference reinforcement learning is a reinforcement learning method based on timing, which is used to learn the long-term cumulative rewards of the agent when taking different actions in the environment. The algorithm updates the estimate of the state value or action value by estimating the difference of the value function, so that the agent can gradually adjust its strategy. In a multi-agent system, each agent uses a similar temporal difference update rule.
[0128] Figure 6 is a schematic diagram of a multi-task multi-robot motion control method based on deep reinforcement learning provided by the present invention, such as Figure 6 As shown, Figure 6 The experience playback in the embodiment corresponds to the training data in this embodiment, and the first training data and the second training data are obtained from the training data. The first training data and the second training data both include the target context information, that is, this training is to train the task corresponding to the target context information; Figure 6 This embodiment is described.
[0129] The first training data includes target context information, a first global state at a first moment, and observation data of each of the multiple robots at the first moment. The first training data is input into multiple independent Q networks to obtain multiple first value estimation vectors output by the multiple independent Q networks. It should be noted that, here, the target context information in the first training data and the observation data of each of the multiple robots at the first moment are respectively input into multiple independent Q networks (corresponding to Figure 6 The general Enter the independent Q network, here refers to the number of robots among multiple robots, and each o represents the observation data of one robot among the multiple robots at the first moment), and obtains multiple first value estimation vectors output by multiple independent Q networks.
[0130] Furthermore, based on the decision-making action of each robot among the multiple robots at the first moment, multiple first values are determined from multiple first value estimation vectors; the first value estimation vector includes multiple first values, and each first value corresponds to a decision action. Here, based on the decision-making action of each robot at the first moment, the corresponding first value is selected from the first value estimation vector, and then multiple first values are obtained.
[0131] Further, the first values, the target context information and the first global state (corresponding to Figure 6 The state s in the joint Q network is input to obtain the first joint value output by the joint Q network (corresponding to Figure 6 Q tot ).
[0132] The second training data includes target context information, a second global state and a target reward signal at a second moment, and observation data of each of the multiple robots at the second moment, where the second moment is the next moment of the first moment; the second training data is input into multiple independent Q networks to obtain multiple second value estimation vectors output by the multiple independent Q networks; here, the target context information in the second training data and the observation data of each of the multiple robots at the second moment are respectively input into multiple independent Q networks to obtain multiple second value estimation vectors output by the multiple independent Q networks.
[0133] It should be noted that the target reward signal is the reward signal when multiple robots execute the task corresponding to the target context information, as well as the reward signal at the second moment; multiple robots completing a task corresponds to a trajectory, and each moment on the trajectory corresponds to a different reward signal. The rewards at all moments of the entire trajectory can be accumulated to obtain the cumulative reward.
[0134] Furthermore, multiple optimal second values are determined based on multiple second value estimation vectors; here, the largest second value in each second value estimation vector is determined as the optimal second value, thereby obtaining multiple optimal second values.
[0135] Furthermore, multiple optimal second values, target context information, and the first global state are input into the joint Q network to obtain a second joint value output by the joint Q network.
[0136] Further, a training target is determined according to the second joint value and the target reward signal; specifically, the training target ,in, is the observation data of each robot in the plurality of robots at the second moment; A decision action of each robot among the multiple robots at a second moment, which is obtained based on a second global state; is the second global state; are the parameters of the updated robot network at the second moment; is the task currently being trained, that is, the task corresponding to the target context information; is the learning rate; r is the target reward signal; Q tot is the joint value output by the joint Q network.
[0137] Further, the target loss function is used to iteratively train the robot network based on the training target and the first joint value; specifically, according to the loss function The parameters of the robot networks (independent Q-network and joint Q-network) are trained using gradient descent optimization, where is the observation data of each robot in the multiple robots at the first moment; A decision action of each robot among the plurality of robots at a first moment, which is obtained based on a second global state; is the first global state; is the parameter of the robot network at the first moment; D is the collection of historical data.
[0138] Through this embodiment, the robot network is iteratively trained based on training data using the target loss function in combination with the joint Q network, thereby improving the training efficiency of the robot network and optimizing the training results.
[0139] As an optional solution, the joint Q network includes a converter module and a combination network; multiple first values, target context information and a first global state are input into the joint Q network to obtain a first joint value output by the joint Q network, including S51-S52.
[0140] S51, input the target loop state, target context information, the first global state and multiple first hidden states into the converter module to obtain the first weight and the first deviation output by the converter module, wherein the target loop state is the loop state output by the converter module at the previous moment, and the multiple first hidden states are the intermediate values obtained by multiple independent Q networks in the process of obtaining multiple first value estimation vectors according to the first training data.
[0141] S52, inputting a plurality of first values into the combination network to obtain a first joint value output by the combination network based on the first weight and the first deviation.
[0142] Figure 7 is a schematic diagram of the structure of the joint Q network provided by the present invention, such as Figure 7 As shown, the joint Q network includes a converter module (corresponding to Figure 7 Transformer modules in ) and combined networks.
[0143] The Transformer module is a deep learning architecture for sequence modeling, originally used for natural language processing tasks. It achieves efficient and flexible representation learning through self-attention mechanism and fully connected layers when processing sequence data. The Transformer module consists of multiple layers, each of which contains a self-attention sublayer and a feedforward neural network sublayer. The stacking of these layers constitutes the entire module. In the self-attention mechanism, the Transformer module can dynamically capture the dependencies between different positions in the input sequence without recursive or convolutional operations.
[0144] In this embodiment, the target loop state, the target context information, the first global state and the first hidden states are input into the converter module to obtain the first weight and the first deviation output by the converter module; wherein the target context information corresponds to Figure 7 The task context information c in the first global state corresponds to Figure 7 In ,here, It is used to represent the number of all entities; the multiple first hidden states are intermediate values obtained by multiple independent Q networks in the process of obtaining multiple first value estimation vectors according to the first training data, which correspond to Figure 7 In ,here Refers to the number of robots in multiple robots.
[0145] It should be noted that the target loop state is the loop state output by the transformer module at the previous moment; the Transformer module takes the loop state, context information, global state, and multiple hidden states as input at each moment, outputs the first weight and the first deviation, and then outputs three new loop states. At the initial moment, the values of the three loop states are all 0. After other variables are input into the Transformer module, the Transformer module outputs the values of the three new loop states as the values of the loop state input into the Transformer module at the next moment.
[0146] See also Figure 7 , h1, h2, and h3 are the values of the three loop states output by the Transformer module at the previous moment. Here, the previous moment refers to the moment before the first moment. are the values of the three recurrent states output by the Transformer module based on h1, h2, and h3 at the first moment.
[0147] In this embodiment, the first weight includes Figure 7 In W1 and W2, the first deviation includes Figure 7 b1 and b2 in which multiple first hidden states are Input Transformer module to get hidden embedding ,Hidden Embed The fusion features of the task corresponding to the observation data and the target context information are further calculated by the absolute value function ; From ; Depend on The absolute value of is calculated; Depend on It is obtained by MLP mapping and Relu (Rectified Linear Unit) activation function in sequence.
[0148] Further, multiple first values are input into the combination network to obtain a first joint value output by the combination network based on the first weight and the first deviation; see Figure 7 , that is, multiple first values Input the combined network and obtain the first joint value of the combined network based on W1, W2, b1 and b2 outputs. .
[0149] It should be noted that since W1 and W2 take absolute values, the combined value of the output is , It is monotonic and therefore satisfies the IGM (Individual Global Max Principle) condition.
[0150] Here, the IGM condition means that for a joint value and individual action value function , if you follow:
[0151]
[0152] but and Satisfy the IGM conditions; among which, is a fixed constant.
[0153] Optionally, the structure of the joint Q network in this embodiment is also applicable to the process of inputting multiple second value estimation vectors, target context information and a second global state into the joint Q network to obtain a second joint value output by the joint Q network.
[0154] Through this embodiment, the accuracy of the obtained joint value can be improved.
[0155] Figure 8 is a block diagram of a multi-task multi-robot motion control system based on deep reinforcement learning provided by the present invention. Figure 8As shown, including but not limited to the following modules:
[0156] The determination module 801 is used to determine context information, wherein the context information is used to indicate tasks to be performed by multiple robots.
[0157] The sending module 802 is used to send the context information to each robot among the multiple robots to drive each robot among the multiple robots to perform a corresponding decision-making action.
[0158] Among them, the decision action corresponding to the individual robot is the individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent the feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks and obstacles; inputting the current observation data and the context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the values of multiple decision actions based on the input information; and determining the individual decision action from the multiple decision actions based on the individual value estimation vector.
[0159] Through the embodiments of the present application, context information is determined, wherein the context information is used to indicate tasks to be performed by multiple robots; the context information is sent to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision-making action; wherein the decision action corresponding to the individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent the feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks, and obstacles; inputting the current observation data and context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the values of multiple decision actions based on the input information; determining the individual decision action from the multiple decision actions based on the individual value estimation vector solves the problem that the motion control method of multiple tasks and multiple robots based on deep reinforcement learning in the related art is insufficient in generalization ability when facing different tasks, resulting in low efficiency in processing multiple tasks, thereby improving the efficiency of the multi-task processing process.
[0160] It should be noted that the multi-task multi-robot motion control system based on deep reinforcement learning provided by the present invention can execute the multi-task multi-robot motion control method based on deep reinforcement learning described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.
[0161] Fig. 9 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig. 9 As shown, the electronic device may include: a processor (processor) 910 , a communication interface (Communications Interface) 920 , a memory (memory) 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute a multi-task multi-robot motion control method based on deep reinforcement learning, the method comprising: determining context information, wherein the context information is used to indicate the tasks to be performed by multiple robots; sending the context information to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision action; wherein the decision action corresponding to the individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent the feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks, and obstacles; inputting the current observation data and context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the value of multiple decision actions based on the input information; and determining the individual decision action from multiple decision actions based on the individual value estimation vector.
[0162] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0163] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multi-task multi-robot motion control method based on deep reinforcement learning provided by the above-mentioned embodiments, the method including: determining context information, wherein the context information is used to indicate the tasks to be performed by multiple robots; sending context information to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision-making action; wherein the decision action corresponding to the individual robot is an individual decision action, and the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent the feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks and obstacles; inputting the current observation data and context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the value of multiple decision actions based on the input information; and determining the individual decision action from the multiple decision actions based on the individual value estimation vector.
[0164] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-task multi-robot motion control method based on deep reinforcement learning provided in the above-mentioned embodiments, the method comprising: determining context information, wherein the context information is used to indicate the tasks to be performed by multiple robots; sending context information to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision-making action; wherein the decision action corresponding to the individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: obtaining current observation data, wherein the current observation data is used to represent the feature information of multiple entities observed by the individual robot at the current moment, and the multiple entities include at least one of other robots, landmarks and obstacles; inputting the current observation data and the context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the values of multiple decision actions based on the input information; and determining the individual decision action from the multiple decision actions based on the individual value estimation vector.
[0165] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multi-task multi-robot motion control method based on deep reinforcement learning, It is characterized in that include: Determining context information, wherein the context information is used to indicate tasks to be performed by the plurality of robots; Sending the context information to each of the multiple robots to drive each of the multiple robots to perform a corresponding decision action; The decision action corresponding to the individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: Acquire current observation data, wherein the current observation data is used to represent feature information of multiple entities observed by the individual robot at a current moment, and the multiple entities include at least one of other robots, landmarks, and obstacles; Inputting the current observation data and the context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the values of multiple decision actions based on the input information; determining the individual decision action from the plurality of decision actions according to the individual value estimation vector; Before determining the context information, the method further includes: Acquiring historical data, wherein the historical data includes the context information, observation data, and decision actions performed by the multiple robots in a historical time period; Obtaining task regret, wherein the task regret is used to represent the gap between the actual reward and the optimal reward of each task in a plurality of tasks, wherein the plurality of tasks are tasks performed by the plurality of robots in the historical time period; Determine a plurality of task learning weight parameters according to the task regret, wherein the plurality of task learning weight parameters correspond one-to-one to the plurality of tasks; Sampling the historical data based on the multiple task learning weight parameters to obtain training data; Initializing parameters of a robot network, wherein the robot network includes a plurality of independent Q networks with shared parameters, and the plurality of independent Q networks correspond one to one to the plurality of robots; Iteratively training the robot network based on the training data using a target loss function until a training termination condition is reached to obtain a trained robot network; The historical data also includes the global state and reward signal at all times in the historical time period, and the robot network also includes a joint Q network, wherein the global state is used to represent the characteristic information of all robots; The iterative training of the robot network based on the training data using the target loss function includes: Acquire first training data and second training data based on the training data, wherein the first training data includes target context information, a first global state at a first moment, observation data and decision actions of each of the multiple robots at the first moment, and the second training data includes the target context information, a second global state and a target reward signal at a second moment, and observation data of each of the multiple robots at the second moment, and the second moment is a moment next to the first moment; Inputting the first training data into the multiple independent Q networks to obtain multiple first value estimation vectors output by the multiple independent Q networks; Determining a plurality of first values from the plurality of first value estimation vectors based on a decision action of each of the plurality of robots at the first moment; Inputting the plurality of first values, the target context information, and the first global state into the joint Q network to obtain a first joint value output by the joint Q network; Inputting the second training data into the multiple independent Q networks to obtain multiple second value estimation vectors output by the multiple independent Q networks; determining a plurality of optimal second values based on the plurality of second value estimate vectors; Inputting the multiple optimal second values, the target context information, and the second global state into the joint Q network to obtain a second joint value output by the joint Q network; determining a training target according to the second joint value and the target reward signal; The robot network is iteratively trained based on the training objective and the first joint value using the objective loss function.
2. The multi-task multi-robot motion control method based on deep reinforcement learning according to claim 1, It is characterized in that The independent Q network includes a first multi-layer perceptron, a second multi-layer perceptron, a pre-trained language model, a first cross-attention mechanism module, a second cross-attention mechanism module, and a plurality of end-to-end self-attention mechanism modules; The step of inputting the current observation data and the context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network includes: Inputting the current observation data into the first multi-layer perceptron to obtain a current feature embedding output by the first multi-layer perceptron; Inputting the context information into the pre-trained language model to obtain a task code output by the pre-trained language model; Input the current feature embedding as a query variable and the task encoding as a key and a value into the first cross-attention mechanism module to obtain a specific task representation output by the first cross-attention mechanism module; Inputting the specific task representation into the multiple end-to-end self-attention mechanism modules to obtain entity features output by the multiple end-to-end self-attention mechanism modules, wherein the self-attention mechanism modules use the same input as query variables, keys, and values; Input the entity feature as a key and a value and the task code as a query variable into the second cross-attention mechanism module to obtain a hidden state output by the second cross-attention mechanism module; The hidden state is input into the second multi-layer perceptron to obtain the individual value estimation vector output by the second multi-layer perceptron.
3. The multi-task multi-robot motion control method based on deep reinforcement learning according to claim 1, It is characterized in that The individual value estimation vector includes a set of individual values, each individual value in the set of individual values corresponds to one of the multiple decision actions; The step of determining the individual decision action from the plurality of decision actions according to the individual value estimation vector comprises: The decision action corresponding to the largest individual value in the group of individual values is determined as the individual decision action.
4. The multi-task multi-robot motion control method based on deep reinforcement learning according to claim 1, It is characterized in that The joint Q network includes a converter module and a combining network; The step of inputting the plurality of first values, the target context information, and the first global state into the joint Q network to obtain a first joint value output by the joint Q network includes: Inputting a target loop state, the target context information, the first global state, and a plurality of first hidden states into the converter module to obtain a first weight and a first deviation output by the converter module, wherein the target loop state is a loop state output by the converter module at a previous moment, and the plurality of first hidden states are intermediate values obtained by the plurality of independent Q networks in a process of obtaining the plurality of first value estimation vectors according to the first training data; The multiple first values are input into the combination network to obtain a first joint value output by the combination network based on the first weight and the first deviation.
5. A multi-task multi-robot motion control system based on deep reinforcement learning, It is characterized in that include: A determination module, configured to determine context information, wherein the context information is used to indicate tasks to be performed by multiple robots; A sending module, configured to send the context information to each of the multiple robots, so as to drive each of the multiple robots to perform a corresponding decision action; The decision action corresponding to the individual robot is an individual decision action, the individual robot is any robot among the multiple robots, and the individual decision action is determined by the individual robot through the following steps: Acquire current observation data, wherein the current observation data is used to represent feature information of multiple entities observed by the individual robot at a current moment, and the multiple entities include at least one of other robots, landmarks, and obstacles; Inputting the current observation data and the context information into an independent Q network to obtain an individual value estimation vector output by the independent Q network, wherein the independent Q network is used to predict the values of multiple decision actions based on the input information; determining the individual decision action from the plurality of decision actions according to the individual value estimation vector; Before determining the context information, the method further includes: Acquiring historical data, wherein the historical data includes the context information, observation data, and decision actions performed by the multiple robots in a historical time period; Obtaining task regret, wherein the task regret is used to represent the gap between the actual reward and the optimal reward of each task in a plurality of tasks, wherein the plurality of tasks are tasks performed by the plurality of robots in the historical time period; Determine a plurality of task learning weight parameters according to the task regret, wherein the plurality of task learning weight parameters correspond one-to-one to the plurality of tasks; Sampling the historical data based on the multiple task learning weight parameters to obtain training data; Initializing parameters of a robot network, wherein the robot network includes a plurality of independent Q networks with shared parameters, and the plurality of independent Q networks correspond one to one to the plurality of robots; Iteratively training the robot network based on the training data using a target loss function until a training termination condition is reached to obtain a trained robot network; The historical data also includes the global state and reward signal at all times in the historical time period, and the robot network also includes a joint Q network, wherein the global state is used to represent the characteristic information of all robots; The iterative training of the robot network based on the training data using the target loss function includes: Acquire first training data and second training data based on the training data, wherein the first training data includes target context information, a first global state at a first moment, observation data and decision actions of each of the multiple robots at the first moment, and the second training data includes the target context information, a second global state and a target reward signal at a second moment, and observation data of each of the multiple robots at the second moment, and the second moment is a moment next to the first moment; Inputting the first training data into the multiple independent Q networks to obtain multiple first value estimation vectors output by the multiple independent Q networks; Determining a plurality of first values from the plurality of first value estimation vectors based on a decision action of each of the plurality of robots at the first moment; Inputting the plurality of first values, the target context information, and the first global state into the joint Q network to obtain a first joint value output by the joint Q network; Inputting the second training data into the multiple independent Q networks to obtain multiple second value estimation vectors output by the multiple independent Q networks; determining a plurality of optimal second values based on the plurality of second value estimate vectors; Inputting the multiple optimal second values, the target context information, and the second global state into the joint Q network to obtain a second joint value output by the joint Q network; determining a training target according to the second joint value and the target reward signal; The robot network is iteratively trained based on the training objective and the first joint value using the objective loss function.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the multi-task multi-robot motion control method based on deep reinforcement learning as described in any one of claims 1 to 4 is implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the multi-task multi-robot motion control method based on deep reinforcement learning as described in any one of claims 1 to 4 is implemented.
8. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the multi-task multi-robot motion control method based on deep reinforcement learning as described in any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Text information-based deep reinforcement learning interactive recommendation method and system
CN111104595A
Robot behavior decision-making method and device based on hierarchical deep reinforcement learning model
CN115409158A
Multi-machine searching method and system based on multi-agent deep reinforcement learning
CN115860107A
Reinforcement learning continuous action control method based on discrete-continuous heterogeneous Q network
CN116224794A
Path planning method fusing deep neural network and reinforcement learning method
CN116448117A
