A collaborative exploration method for unmanned equipment clusters equipped with reinforcement learning agents

Abstract

The present invention belongs to the field of unmanned equipment automation control, and specifically provides an unmanned equipment cluster collaborative exploration method equipped with a reinforcement learning intelligent agent, which can effectively enhance the unmanned cluster's collaborative exploration capability for unknown environments; the present invention adopts a stacked autoencoder to encode similar states into similar feature vectors, so that the intelligent agent can more clearly identify similar environmental states; a self-supervised classifier is used to calculate a matching score between the intelligent agent and the visited environmental state, determine whether the intelligent agent should visit this state, guide the differentiation of the intelligent agent's access state, and prompt different intelligent agents to explore different areas, thereby enhancing the overall exploration capability of a multi-agent system for the environment, realizing collaborative exploration of the multi-agent system, and thereby strengthening the synergy of different devices in the unmanned equipment cluster, improving the effective coordination between different devices, and improving the comprehensive decision-making capability of the cluster.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned equipment automation control and involves algorithm intelligent decision-making. It specifically provides an unmanned equipment cluster collaborative exploration method equipped with a reinforcement learning agent. Background Art

[0002] Reinforcement learning is a common method in the field of intelligent decision-making, especially deep reinforcement learning, which combines the decision-making advantages of reinforcement learning with the perception advantages of deep learning. It has greatly promoted the development of modern artificial intelligence and is hailed as the most likely method to achieve general artificial intelligence. Reinforcement learning agents also have more powerful capabilities than traditional agents, especially strong learning and environmental adaptability. They can autonomously explore various unknown environments, and play a key role in the intelligent decision-making and autonomous operation of unmanned equipment in completely unknown environments such as the wild, the open sea, and space. Specifically, unmanned equipment equipped with reinforcement learning agents has stronger decision-making capabilities than traditional unmanned equipment, and a cluster of unmanned equipment equipped with reinforcement learning agents can be considered a multi-agent reinforcement learning system.

[0003] However, the reinforcement learning algorithm requires that the task rewards provided by the environment cannot be too sparse, otherwise it is still necessary to manually set reward targets to guide the agent to explore the unknown environment and decide the optimal strategy. Traditional reinforcement learning agents use brute force to explore unknown environments. If the environment task rewards are too sparse, the agent needs to spend a lot of computing power on exploring rewards, and may not even be able to correctly explore any rewards at all. Artificial reward shaping often requires high costs. This problem makes the training of reinforcement learning agents costly, and this cost cannot even be compensated by the benefits brought by the decision-making advantages of reinforcement learning agents, which makes it difficult for reinforcement learning algorithms to be implemented in actual industrial production.

[0004] At this stage, reinforcement learning exploration is mainly divided into two methods: building intrinsic motivation and introducing prior knowledge. The former mainly constructs other reward logics in addition to environmental task goals through algorithms, such as curiosity mechanism to encourage agents to explore environmental areas that have never been visited; post-experience replay effectively utilizes failed explorations to enhance the agent's explorable range, but the exploration strategy in the single-agent field is only partially effective for multi-agent systems and cannot completely solve the problem of collaborative exploration of multiple agents; the latter enables agents to directly learn strategies that have been formulated by human experts. This method requires experts to directly study and judge the unknown environment, and cannot give full play to the decision-making advantages of reinforcement learning. Summary of the invention

[0005] The purpose of the present invention is to address the defects of the above-mentioned prior art and provide a collaborative exploration method for unmanned equipment clusters equipped with reinforcement learning agents, which can effectively enhance the collaborative exploration capabilities of unmanned clusters in unknown environments; the present invention aims to train the independence between different agents, encourage different agents to explore different environmental areas, accelerate the social division of labor and differentiation of multi-agent systems, greatly shorten the overall exploration time, save the computing power overhead in the exploration stage of multi-agent reinforcement learning training, and then reflect on the unmanned equipment cluster, which can improve the effective coordination of different devices between clusters and improve the comprehensive decision-making capabilities of the cluster.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A collaborative exploration method for an unmanned device cluster equipped with a reinforcement learning agent, wherein the unmanned device cluster is regarded as a multi-agent system, and the unmanned device cluster equipped with a reinforcement learning agent is regarded as a multi-agent reinforcement learning system, including the following steps:

[0008] S1. Encode the feature vector of each agent in the multi-agent system according to the agent attributes to obtain the agent feature vector of each agent;

[0009] S2, the multi-agent system interacts with the environment, and each agent obtains the next observation state and task reward from the environment;

[0010] S3, construct a stacked autoencoder and perform unsupervised training on it according to the detected state in the environment to obtain an autoencoder model;

[0011] S4, encoding the observation state of each agent in the multi-agent system at the current time step through the autoencoder model to obtain the state feature vector of each agent;

[0012] S5, using a classifier to calculate a matching score for the agent feature vector and state feature vector of each agent in the multi-agent system at the current time step;

[0013] S6. Calculate the matching score reward for each agent in the multi-agent system at the current time step according to the matching score;

[0014] S7, calculating the total reward for each agent in the multi-agent system at the current time step according to the matching score reward and the task reward;

[0015] S8. According to the agent’s current observation state, action, next observation state and total reward, train the multi-agent reinforcement learning algorithm and update the reinforcement learning agent strategy model.

[0016] Furthermore, in step S1, the feature vector encoding process is:

[0017] S11. The total action space in the Markov decision process {A i} i∈I can be divided into n independent parts: {z1, z2, ..., z n}, using n-dimensional vectors to encode agent types, if the executable action space A of agent i i and partition space z i If they overlap, then z i The corresponding vector bit is recorded as 1, and the other vector bits are recorded as 0, and the agent type characteristic vector of agent i is obtained;

[0018] S12. For each type of agent, encode the agent serial number by one-hot encoding to obtain the agent serial number feature vector of agent i;

[0019] S13. Cascade the agent type feature vector and the agent sequence number feature vector to obtain the agent feature vector of agent i.

[0020] Furthermore, in step S2, at each time step, each agent in the multi-agent system interacts with the environment in turn: agent i interacts with the environment according to the current observation state o i and reinforcement learning policy model π i Decision to take action a i , the environment is in the current state s of agent i i Execute action a i , transfer agent i to state s i ′, and the task reward and the next observation state o i ′ is fed back to agent i.

[0021] Further, in step S3, specifically: S31, in each time step, collect the observation state of each agent access state in the multi-agent system and store it in the observation state buffer area B of size buffsize s ;

[0022] S32, construct a stacked autoencoder Coder that is sufficient to analyze the entire environment state space through a multi-layer perceptron;

[0023] S33. At the end of each epsiode of reinforcement learning, from the observation state buffer area B s The data is sampled and the mean square error loss function is used to perform unsupervised training on the stacked autoencoder Coder to obtain a converged autoencoder model.

[0024] Furthermore, in step S4, specifically: the next observation state obtained by the agent i from the environment is input into the autoencoder model, and the state feature vector of the agent i is calculated by the encoding module of the autoencoder model.

[0025] Furthermore, in step S5, specifically:

[0026] S51, inputting the agent feature vector and state feature vector of agent i into a classifier, and the classifier outputs a matching score;

[0027] S52. Build temporary labels: <feature i ,code i ; 1> and in, Represents the feature vector of other agents except agent i, - is the counterfactual label, and 1 represents the feature vector feature of the agent i and state feature vector code i The fact combination of is a positive sample, and 0 represents the feature vector of other agents except agent i. and state feature vector code i The counterfactual combination of is a negative sample; the network parameters of the classifier are updated through the temporary labels.

[0028] Furthermore, in step S6, specifically: calculate the state matching reward that agent i should obtain

[0029]

[0030] Among them, β is the reward weight, r b is the basic reward, and score is the matching score;

[0031] And match the status to reward Feedback to agent i.

[0032] Furthermore, in step S7, specifically:

[0033] S71. Calculate the state matching reward of agent i and other intrinsic rewards using curiosity mechanism or post-experience replay to obtain the total intrinsic reward of agent i

[0034] S72. The total intrinsic reward of agent i Add to the task reward to get the total reward r i , and the total reward r i Feedback to agent i.

[0035] Based on the above technical solution, the beneficial effects of the present invention are:

[0036] 1. Using stacked autoencoders to encode similar states into similar feature vectors enables the intelligent agent to more clearly identify similar environmental states and obtain more information from the environment, thereby actually enhancing the comprehensive information acquisition capabilities of the unmanned equipment cluster;

[0037] 2. Use the self-supervised classifier to calculate the matching score between the agent and the visited environmental state, and determine whether the agent should visit this state. As the number of reinforcement learning training episodes increases, the differentiation of the agent's access state is guided, prompting different agents to explore different areas, enhancing the overall exploration capability of the multi-agent system for the environment, and realizing the collaborative exploration of the multi-agent system, thereby strengthening the coordination of different devices in the unmanned equipment cluster, improving the effective coordination between different devices, and improving the comprehensive decision-making ability of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of the intelligent agent collaborative exploration method based on reinforcement learning in the present invention.

[0039] Figure 2 Schematic diagram of the interaction process between the multi-agent system and the environment in the present invention.

[0040] Figure 3 It is a structural diagram of the autoencoder model in the present invention.

[0041] Figure 4 It is a structural diagram of the self-supervised classifier network model in the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with embodiments and drawings.

[0043] This embodiment aims to propose a collaborative exploration method for an unmanned equipment cluster equipped with a reinforcement learning agent. The method provides new intrinsic motivations for the agents by constructing state matching rewards, accelerates the environmental social division of labor among multiple agents, improves the comprehensive exploration efficiency of the multi-agent system, and thus solves the problem that the unmanned equipment cluster equipped with reinforcement learning agents has difficulty in autonomous learning in unknown environments. The specific process is as follows: Figure 1 As shown, the following steps are included:

[0044] S1. Encode the feature vector of each agent in the multi-agent system according to the agent attributes to obtain the agent feature vector of each agent. The process of encoding the feature vector is as follows:

[0045] S11. The total action space in the Markov decision process {Ai} i∈I can be divided into n independent parts: {z1,z2,…,z n}, using n-dimensional vectors to encode agent types, if the executable action space A of agent i i and partition space z i If they overlap, then z i The corresponding vector bit is recorded as 1, and the other vector bits are recorded as 0, and the agent type characteristic vector of agent i is obtained I represents the set of agents;

[0046] S12. For each type of agent, encode the agent number by one-hot encoding (if the agent number is too large, use segmented one-hot encoding) to obtain the agent number feature vector of agent i

[0047] S13, concatenate the agent type feature vector and the agent sequence number feature vector to obtain the agent feature vector feature of agent i i ;

[0048] S2, the multi-agent system interacts with the environment, and each agent obtains the next observation state and task reward from the environment;

[0049] The process of interaction between the multi-agent system and the environment is as follows: Figure 2 As shown in the figure, in each time step, each agent in the multi-agent system interacts with the environment in turn: agent i interacts with the environment according to the current observation state o i and reinforcement learning policy model π i Decision to take action a i , the environment is in the current state s of agent i i Execute action a i , transfer agent i to state s i ′, and the task reward and the next observation state o i ′ is fed back to agent i;

[0050] S3, construct a stacked autoencoder and perform unsupervised training on it according to the detected state in the environment to obtain an autoencoder model;

[0051] The stacked autoencoder model is as follows Figure 3 As shown, it is divided into an encoder and a decoder, where the encoder includes an input layer and a hidden layer, and the decoder includes a hidden layer and an output layer. The encoder and the decoder share the hidden layer network parameters to speed up training. The calculation formula is:

[0052] h w，b (x) = x

[0053] decoder(enCoder(k))=x

[0054]

[0055] Among them, h w,b (.) represents the autoencoder, x represents the agent observation state vector, encoder and decoder are the encoder and decoder respectively, loss is the loss function, y i and Represent the true value and the network predicted value respectively;

[0056] Specifically: S31, at each time step, collect the observation state of each agent access state in the multi-agent system (observation state o i ′), stored in the observation state buffer B of size buffsize s ;

[0057] S32, construct a stacked autoencoder Coder that is sufficient to analyze the entire environment state space through a multi-layer perceptron;

[0058] S33. At the end of each epsiode of reinforcement learning, from the observation state buffer area B s The data is sampled and the stacked autoencoder Coder is trained unsupervised using the mean square error loss function to obtain a converged autoencoder model.

[0059] S4, the observation state (observation state o) of each agent in the multi-agent system at the current time step is accessed through the autoencoder model i ′) to encode and obtain the state feature vector of each agent;

[0060] Specifically: the next observation state o obtained by agent i from the environment i ′ is input into the autoencoder model, and the state feature vector code of agent i is calculated by the encoding module encoder of the autoencoder model i ;

[0061] S5, using a classifier to calculate a matching score for each pair of agent feature vectors and state feature vectors in the multi-agent system at the current time step;

[0062] The network model of the classifier is as follows Figure 4 As shown, it includes input layer, hidden layer and output layer, and the score calculation formula is:

[0063] score=sigmoid(W4(W3(W1cfeature i )+W2(code i )))

[0064] Among them, W1 and W2 are the network parameters of the agent feature vector and state feature vector in the input layer, W3 is the hidden layer network parameter, W4 and sigmoid are the output layer network parameter and activation function respectively;

[0065] During the training phase, the classifier is trained through the agent feature vector feature i and state feature vector code i , the sigmoid activation function is used to calculate the matching score, and the agent feature vector feature i and state feature vector code i Generate self-supervised learning labels and update network parameters in a self-supervised manner by minimizing the cross entropy loss function;

[0066] In the execution phase, specifically:

[0067] S51, the agent feature vector feature of agent i i and the state feature vector code i Input the classifier Classifier, and the classifier Classifier outputs the agent feature vector feature i and state feature vector code i The matching score score;

[0068] S52. Build temporary labels: <feature i ,code i ; 1> and in, Represents the feature vector of other agents except agent i, - is the counterfactual label, and 1 represents the feature vector feature of the agent i and state feature vector code i The fact combination of is a positive sample, and 0 represents the feature vector of other agents except agent i. and state feature vector code i The counterfactual combination is a negative sample; the network parameters of the classifier are updated through temporary labels;

[0069] S6. Calculate the matching score reward for each agent in the multi-agent system at the current time step;

[0070] Specifically: Calculate the state matching reward that agent i should obtain

[0071]

[0072] Among them, β is the reward weight, rb As a basic reward;

[0073] And match the status to reward Feedback to agent i;

[0074] S7, calculate the total reward for each agent in the multi-agent system at the current time step;

[0075] Specifically:

[0076] S71. Match the state of agent i with the reward The total intrinsic reward of agent i is calculated by using curiosity mechanism or hindsight experience replay with other intrinsic rewards

[0077] S72. The total intrinsic reward of agent i Rewards Add up to get the total reward r i : The total reward r i Feedback to agent i;

[0078] S8. According to the agent's current observation state, action, next observation state and total reward, the multi-agent reinforcement learning algorithm is trained at an appropriate time to update the reinforcement learning agent strategy model; specifically, if there is an experience replay area, the agent stores the observation state, the next observation state of the action and the total reward in the experience replay area, and samples from the replay area for training during the model update phase; if online learning is used for updating, the agents use this experience to update the strategy model and discard the experience.

[0079] The above reinforcement learning agent is divided into a training phase and an execution phase. In the training phase, S1 is implemented in the initialization phase, S2, S4-S8 are iteratively implemented as time steps advance, and S3 is implemented at the end of each episode; in the execution phase, the agent feature vector encoding method has been established, the autoencoder Coder and classifier Classifier have been trained, and S2, S4-S7 are iteratively implemented as time steps advance.

[0080] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A collaborative exploration method for an unmanned device cluster equipped with a reinforcement learning agent, wherein the unmanned device cluster is regarded as a multi-agent system, and the unmanned device cluster equipped with a reinforcement learning agent is regarded as a multi-agent reinforcement learning system, including the following steps: S1. Encode the feature vector of each agent in the multi-agent system according to the agent attributes to obtain the agent feature vector of each agent; The process of encoding the feature vector is as follows: S11. The total action space in the Markov decision process {A i } i∈I is divided into n independent parts: {z1,z2,…,z n }, using n-dimensional vectors to encode agent types, if the executable action space A of agent i i and partition space z i If they overlap, then z i The corresponding vector bit is recorded as 1, and the other vector bits are recorded as 0, and the agent type characteristic vector of agent i is obtained; S12. For each type of agent, encode the agent serial number by one-hot encoding to obtain the agent serial number feature vector of agent i; S13, concatenating the agent type feature vector and the agent sequence number feature vector to obtain an agent feature vector of agent i; S2, the multi-agent system interacts with the environment, and each agent obtains the next observation state and task reward from the environment; S3, construct a stacked autoencoder and perform unsupervised training on it according to the detected state in the environment to obtain an autoencoder model; S4, encoding the observation state of each agent in the multi-agent system at the current time step through the autoencoder model to obtain the state feature vector of each agent; S5, using a classifier to calculate a matching score for the agent feature vector and state feature vector of each agent in the multi-agent system at the current time step; S6. Calculate the matching score reward for each agent in the multi-agent system at the current time step according to the matching score; S7. Calculate the total reward for each agent in the multi-agent system at the current time step based on the matching score reward and task reward; specifically: S71. Calculate the state matching reward of agent i and other intrinsic rewards using curiosity mechanism or post-experience replay to obtain the total intrinsic reward of agent i S72. The total intrinsic reward of agent i Add to the task reward to get the total reward r i , and the total reward r i Feedback to agent i; S8. According to the agent’s current observation state, action, next observation state and total reward, train the multi-agent reinforcement learning algorithm and update the reinforcement learning agent strategy model.

2. The unmanned equipment cluster collaborative exploration method equipped with a reinforcement learning agent according to claim 1, characterized in that: In step S2, at each time step, each agent in the multi-agent system interacts with the environment in turn: agent i interacts with the environment according to the current observation state o. i and reinforcement learning policy model π i Decision to take action a i , the environment is in the current state s of agent i i Execute action a i , transfer agent i to state s i ′, and the task reward and the next observation state o i ′ is fed back to agent i.

3. The unmanned equipment cluster collaborative exploration method equipped with a reinforcement learning agent according to claim 1, characterized in that: In step S3, specifically: S31. In each time step, collect the observation state of each agent in the multi-agent system and store it in the observation state buffer area B with a size of buffsize. s ; S32, construct a stacked autoencoder Coder that is sufficient to analyze the entire environment state space through a multi-layer perceptron; S33. At the end of each epsiode of reinforcement learning, from the observation state buffer area B s The data is sampled and the mean square error loss function is used to perform unsupervised training on the stacked autoencoder Coder to obtain a converged autoencoder model.

4. The unmanned equipment cluster collaborative exploration method equipped with a reinforcement learning agent according to claim 1, characterized in that: In step S4, specifically: the next observation state obtained by agent i from the environment is input into the autoencoder model, and the state feature vector of agent i is calculated by the encoding module of the autoencoder model.

5. The unmanned equipment cluster collaborative exploration method equipped with a reinforcement learning agent according to claim 1, characterized in that: In step S5, specifically: S51, inputting the agent feature vector and state feature vector of agent i into a classifier, and the classifier outputs a matching score; S52. Build temporary labels: <feature i ,code i ; 1> and in, Represents the feature vector of other agents except agent i, - is the counterfactual label, and 1 represents the feature vector feature of the agent i and state feature vector code i The fact combination of is a positive sample, and 0 represents the feature vector of other agents except agent i. and state feature vector code i The counterfactual combination of is a negative sample; the network parameters of the classifier are updated through the temporary labels.

6. The unmanned equipment cluster collaborative exploration method equipped with a reinforcement learning agent according to claim 1, characterized in that: In step S6, specifically: calculate the state matching reward that agent i should obtain Among them, β is the reward weight, r b is the basic reward, and score is the matching score; And match the status to reward Feedback to agent i.

Images