Multi-robot cooperative work method based on contrastive learning role representation reinforcement learning algorithm

Abstract

The application discloses a kind of multi-robot cooperative work methods based on contrast learning role representation reinforcement learning algorithm, belong to multi-agent reinforcement learning field.It includes: each robot obtains the observation information of current environment, respectively carries out time series modeling, role modeling, space modeling, selects action and obtains reward value, generates state value;And in this process, the observation information of each robot, the global state information of current environment and reward value, role representation etc.are stored in an experience replay pool, repeat the above steps finally utilize the data stored in experience replay pool to complete model optimization, solve the agent behavior homogenization problem and reputation allocation problem in the process of multi-robot cooperative work, can effectively quantify the collaboration contribution between robots, to encourage robots to work better in future tasks, so that robot preferentially selects the behavior that is beneficial to the whole system, improves overall operating efficiency.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent reinforcement learning, specifically relating to a multi-robot collaborative working method based on a contrastive learning role representation reinforcement learning algorithm. Background Technology

[0002] Existing multi-agent reinforcement learning methods often focus only on current observations, making it difficult to capture long-term temporal dependencies. Leveraging the sequence processing capabilities of the Transformer architecture, researchers have introduced it into the MARL field, such as in MAT. Although Transformer-based MARL architectures have shown great potential in capturing long-term dependencies, their application remains limited by two core challenges.

[0003] First, the homogeneity of agent behavior limits the effectiveness of diverse collaborative systems. Existing methods generally employ parameter sharing mechanisms to improve learning efficiency, but this inevitably leads to convergence of agent strategies, making it difficult to adapt to complex and ever-changing collaborative tasks. To address this issue, researchers have introduced role-sharing mechanisms, such as Romat using predefined roles and SMART deriving roles through action abstraction, to preserve individual specificity in shared representations. However, the roles in these methods are often static or pre-defined, failing to fully respond to dynamic environmental changes, and their flexibility and adaptability remain insufficient.

[0004] Secondly, insufficient accuracy in credit allocation hinders further improvements in learning efficiency. Under the framework of shared global rewards, accurately evaluating the individual contribution of each agent remains a critical challenge. While numerous methods have addressed this, such as COMA using counterfactual baselines, VDN and QMIX using value function decomposition, and LICA and DuelMIX using attention-weighted mechanisms to differentiate credit allocation, most lack detailed modeling of the individual decision-making context. Their credit allocation mechanisms remain crude, lacking targeted analysis for heterogeneous agents, making it difficult for agent policy optimization to reach global optimum. Therefore, a method capable of adaptively learning dynamic roles and achieving accurate individualized credit assessment is urgently needed to comprehensively improve the collaborative performance of multi-agent systems. Summary of the Invention

[0005] To address the issues of homogenized agent behavior and reputation allocation in multi-robot collaborative decision-making, which lead to similar robot behaviors and potential decision-making errors or strategy conflicts, and to resolve the problems of evaluating and allocating the responsibilities and contributions of each robot in the overall environment—issues where a robot's responsibilities and contributions are influenced not only by its own behavior but also by the actions of other robots and environmental changes—this invention provides a multi-robot collaborative working method based on contrastive learning role representation, comprising the following steps:

[0006] S1: Each robot acquires its own observation information at the current moment and merges the observation information of all robots at the current moment into a global state; constructs a role representation learning module, and performs temporal modeling and role modeling on each robot through the role representation learning module based on each observation information, generating the corresponding temporal representation and role representation at the current moment;

[0007] S2: Based on the temporal and role representations of each robot at the current moment, construct the Encoder module to perform spatial modeling for each robot and generate information after the current robot interacts with each other.

[0008] S3: Based on the interaction information obtained in S2, construct the Decoder module, make decisions according to the serialization principle, and make decision selections for each robot in an autoregressive manner;

[0009] S4: The role representation learning module is optimized and trained based on the contrastive learning algorithm to obtain the optimized role representation learning module; based on the observation information, global state and role representation obtained in S1, a state value evaluation network is constructed to evaluate the current state of the robot; the Encoder module and Decoder module are trained based on the state value to obtain the optimized Encoder module and Decoder module; the optimized role representation learning module, Encoder module and Decoder module are integrated into the CRMAT network model;

[0010] S5: Decompose the task to be processed into several steps, and process each step using the CRMAT network model until the task is completed.

[0011] Furthermore, in step S1, the specific steps of the role representation learning module in performing temporal modeling and role modeling for each robot are as follows:

[0012] S1.1: In At any given moment, each robot receives observational information about its current environment, represented as... ,in Indicates the first A robot Real-time observation information, including its own state, surrounding environment, and task-related information; and simultaneously, by integrating all robot data. The observation information at each moment constructs a global state representation of the current environment. , to observe information With global state representation Store them together in the experience replay pool middle;

[0013] S1.2: Observation information After being processed by the MLP, the data is input into the GRU (Government Recurrent Neural Network) for extracting temporal information, while simultaneously processing the temporal representation from the previous time step. It is also input into the recurrent neural network GRU to obtain the robot's... Temporal representation of time The temporal representation of all robots constitutes , The learning formula is as follows:

[0014]

[0015] in, Indicates the first A robot Temporal representation of moments; For the first A robot The temporal representation of each moment is initialized as a 0 vector;

[0016] S1.3: Temporal representation of any robot The robot's role representation is obtained by inputting the data into a multilayer perceptron (MLP). The role representation of all robots constitutes , The learning formula is as follows:

[0017]

[0018] in, Indicates the first A robot The role representation of time.

[0019] Furthermore, in step S2, the robot is spatially modeled by constructing an Encoder module to generate information about the robot's interactions with each other, helping the robot to make more accurate decisions by fully utilizing the information from its teammates. Specifically, this includes:

[0020] Temporal representation of any robot With role representation The data is concatenated and input into the Encoder module. After processing by the attention mechanism and the Multilayer Perceptron (MLP), the spatially modeled interaction information is obtained. The specific formula is shown below:

[0021]

[0022]

[0023]

[0024] in, This represents a function that concatenates temporal representations with role representations; This represents the information obtained by concatenating the temporal representation and the role representation; , , These represent the parameter matrices used to generate the query vector, key vector, and value vector in the self-attention mechanism, respectively. The dimension of the query vector is represented by the softmax function, which calculates the distribution of attention weights. Representation layer normalization; This represents the attention-weighted features calculated through a self-attention mechanism; This represents the final interactive information output after spatial modeling.

[0025] Furthermore, in step S3, the specific steps for making decisions and selections for each robot using the Decoder module are as follows:

[0026] First, initialize an empty action sequence. Each decision action generated by the robot is stored in this sequence using one-hot encoding, with a length equal to the action space dimension plus 1. Initially, the first bit of the sequence is set to 1, and the remaining bits are 0. At each time step, the previous... The robot's action sequence The input is fed into the first-layer masked self-attention network, which uses a masking mechanism to ensure that the model is in focus. Only able to perceive the past The robot's motion information is used to output integrated information. This provides the conditions for subsequent strategy generation:

[0027]

[0028] in, Represents the mask matrix;

[0029] Interactive information output by the Encoder module The query vector, used as the second-layer attention network, simultaneously incorporates the features extracted by the first-layer masked attention network. With action sequence After concatenation, they together serve as the key and value vectors of the second-layer attention network, outputting the information processed by the second-layer attention mechanism. :

[0030]

[0031] After that and splicing together information The information is then processed by an MLP and output as the processed information. :

[0032]

[0033]

[0034] Then and After concatenation, the data is processed by an MLP to output the final decision information. And convert it into one-hot encoding and put it back into the action sequence:

[0035]

[0036] in, yes Time before The action sequence information of each robot; This represents the information processed by the first layer of the attention network; This represents the information processed by the second-layer attention network; Indicates will and Information after splicing; Indicates will Information after MLP processing; Indicates the first The decision-making of a robot;

[0037] After obtaining the actions of all robots Afterwards, the action was performed and obtained Momentary Rewards and will and All are stored in the experience replay pool middle.

[0038] Furthermore, in step S4, the role representation learning module is optimized and trained based on the contrastive learning algorithm. The training process includes:

[0039] From the experience replay pool Mid-sampling yields data for a complete round. ,in ;

[0040] For each moment Execute step S1 to obtain the temporal representation of all robots. and role representation ;

[0041] Regularly The robots are clustered based on their temporal representations. Cluster ,in For the first A role cluster is formed, encouraging roles within the same cluster to move closer to each other while maintaining distinction from robots in other clusters; for the first... A robot, representing its role Let q be the query, and the role representations of other robots be denoted as the key set. ; belongs to the same cluster as the query (i.e. The key is designated as a positive key. Keys from different clusters are assigned as negative keys. The bilinear product is used as the scoring function, where the similarity between the query and the key is determined by... Calculate W, a learnable parameter matrix; calculate the contrastive loss function and update the network according to the following formula:

[0042]

[0043]

[0044] in, Indicates the positive sample index. This represents the positive sample role. Indicates the negative sample index. This represents the role of the negative sample.

[0045] Furthermore, in step S4, the specific steps for the state value assessment network to assess the state value of the robot are as follows:

[0046] Observation information Global status information and character representation The observation information is input into the state value assessment network, which then processes it as follows: After processing by the MLP, the data is input into the GRU (Generative Recurrent Neural Network) for extracting temporal information, generating the robot's temporal representation. The global state representation is then processed by the MLP and input into the GRU again, generating a global state temporal representation. The state temporal representation is then used as a query vector, and the role representation is used as a key and value vector, respectively, and input into an attention network. Finally, the output of the attention network is concatenated with the temporal representation and processed by the MLP to output the state value. The specific formulas for these steps are as follows:

[0047]

[0048]

[0049]

[0050] in, express Time of the first Temporal representation of robot observations; express The global state temporal representation at any given moment; express The global state temporal representation at any given moment; This represents the output information of the attention network; Indicates the first A robot The value of a state at any given moment.

[0051] Furthermore, in step S4, the training of the Encoder module and Decoder module based on state values ​​is performed as follows:

[0052] From the experience replay pool Mid-sampling yields data for a complete round. , , , ,in , ;

[0053] Calculate the advantage function and the target state value:

[0054]

[0055]

[0056]

[0057] in, and They represent Time and The value of a state at any given moment; It is a discount factor. ; express The reward value at any given moment; Indicates timing difference error; Represents the dominance function; Indicates the value of the target state; It is the attenuation factor;

[0058] Calculate the loss function and update the entire network:

[0059]

[0060] in, Operations should be implemented to prevent excessive updates; Strategy entropy is used to enhance exploration; The entropy regularization coefficient; This refers to the cropping area.

[0061] A second aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the electronic device performs the multi-robot collaborative work method based on contrastive learning role representation.

[0062] A third aspect of the present invention provides a storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to execute the multi-robot collaborative work method based on contrastive learning role representation.

[0063] The beneficial effects of this invention are as follows: This invention introduces a role learning module during execution, combining GRU and Transformer architectures, and employs an autoregressive approach when action selection follows a sequential strategy. By modeling the robot's observation sequence, the robot can combine historical observation information with current observation information, helping it learn more suitable and distinctive roles. Spatial modeling allows the robot to fully utilize information from teammates, enabling it to be more distinctive and obtain more comprehensive information, reducing problems caused by behavioral homogenization, which is crucial for the robot to make correct decisions. This significantly improves the robot's decision-making ability, enabling it to effectively coordinate and cooperate with other robots in complex dynamic scenarios. Furthermore, this invention introduces contrastive learning during training, allowing the robot to learn more distinctive roles and greatly improving the heterogeneity among robots. Simultaneously, an attention mechanism is introduced into the state value evaluation network, guiding the global state to focus on the role representation of each robot based on the attention mechanism, resulting in more accurate reputation allocation during state value evaluation. By addressing the issue of reputation allocation, the collaborative contributions among robots can be effectively quantified, thereby incentivizing robots to work together better in future tasks. This will encourage robots to prioritize behaviors that benefit the system as a whole, thus improving overall operational efficiency. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the process steps of the present invention.

[0065] Figure 2 This is a schematic diagram of the architecture of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, this embodiment of the invention provides a multi-robot collaborative work method based on contrastive learning role representation in a photovoltaic factory maintenance scenario, including the following steps:

[0068] S1: Each robot acquires its own observation information at the current moment and merges the observation information of all robots at the current moment into a global state; a role representation learning module is constructed, and based on the observation information, the role representation learning module performs temporal and role modeling on each robot, generating the corresponding temporal and role representations at the current moment. The specific process is as follows:

[0069] S1.1: In At any given moment, each robot receives observational information about its current environment, represented as... ,in Indicates the first Individual testing robots The system collects real-time observation information, including its own state information: the robot's current position, orientation, and speed; the type of task it is currently performing (e.g., cleaning, inspection, or maintenance); the type and status of its carried tools or equipment; and the robot's remaining battery power. It also collects surrounding environment information: the positions of other robots; the distribution and orientation of the photovoltaic panel array; the location of obstructions or obstacles; and the terrain ruggedness. Task-related information includes the photovoltaic panel number or area to be cleaned or repaired; the task priority; and the specific location and anomaly type of faulty equipment. Simultaneously, it integrates the observation information from all robots to construct a global state representation of the current environment. The observation information and the global state representation are stored together in the experience replay pool. middle.

[0070] S1.2: Observation information After being processed by the MLP, the data is input into the GRU (Government Recurrent Neural Network) for extracting temporal information, while simultaneously processing the temporal representation from the previous time step. It is also input into the recurrent neural network GRU to obtain the robot's... Temporal representation of time The temporal representation of all robots constitutes , The learning formula is as follows:

[0071]

[0072] in, Indicates the first A robot Temporal representation of moments; For the first A robot The temporal representation of each moment is initialized as a 0 vector.

[0073] S1.3: Temporal representation of any robot The robot's role representation is obtained by inputting the data into a multilayer perceptron (MLP). The role representation of all robots constitutes , The learning formula is as follows:

[0074]

[0075] in, Indicates the first A robot The role representation of time.

[0076] S2: Based on the temporal and role representations of each robot at the current moment, an Encoder module is constructed to perform spatial modeling for each robot, generating information about the current robot's interactions with each other. This helps the robot fully utilize the information from its teammates to make more accurate decisions, specifically including:

[0077] Temporal representation of any robot With role representation The data is concatenated and input into the Encoder module. After processing by the attention mechanism and the Multilayer Perceptron (MLP), the spatially modeled interaction information is obtained. The specific formula is shown below:

[0078]

[0079]

[0080]

[0081] in, This represents a function that concatenates temporal representations with role representations; This represents the information obtained by concatenating the temporal representation and the role representation; , , These represent the parameter matrices used to generate the query vector, key vector, and value vector in the self-attention mechanism, respectively. The dimension of the query vector is represented by the softmax function, which calculates the distribution of attention weights. Representation layer normalization; This represents the attention-weighted features calculated through a self-attention mechanism; This represents the final interactive information output after spatial modeling.

[0082] S3: Based on the interaction information obtained in S2, construct the Decoder module, follow the serialization principle to make decisions, and use an autoregressive approach to select the best option for each robot; the specific steps are as follows:

[0083] First, initialize an empty action sequence. Each decision action generated by the robot is stored in this sequence using one-hot encoding, with a length equal to the action space dimension plus 1. Initially, the first bit of the sequence is set to 1, and the remaining bits are 0. At each time step, the previous... The robot's action sequence The input is fed into the first-layer masked self-attention network, which uses a masking mechanism to ensure that the model is in focus. Only able to perceive the past The robot's motion information is used to output integrated information. This provides the conditions for subsequent strategy generation:

[0084]

[0085] in, This represents the mask matrix.

[0086] Interactive information output by the Encoder module The query vector, used as the second-layer attention network, simultaneously incorporates the features extracted by the first-layer masked attention network. With action sequence After concatenation, they together serve as the key and value vectors of the second-layer attention network, outputting the information processed by the second-layer attention mechanism. :

[0087]

[0088] After that and splicing together information The information is then processed by an MLP and output as the processed information. :

[0089]

[0090]

[0091] Then and After concatenation, the data is processed by an MLP to output the final decision information. And convert it into one-hot encoding and put it back into the action sequence:

[0092]

[0093] in, yes Time before The action sequence information of each robot; This represents the information processed by the first layer of the attention network; This represents the information processed by the second-layer attention network; Indicates will and Information after splicing; Indicates will Information after MLP processing; Indicates the first The decision-making of a robot.

[0094] After obtaining the actions of all robots Afterwards, the action was performed and obtained Momentary Rewards and will and All are stored in the experience replay pool middle.

[0095] S4: The character representation learning module is optimized and trained based on a contrastive learning algorithm to obtain an optimized character representation learning module; the training process includes:

[0096] From the experience replay pool Mid-sampling yields data for a complete round. ,in ;

[0097] For each moment Execute step S1 to obtain the temporal representation of all robots. and role representation ;

[0098] Regularly The robots are clustered based on their temporal representations. Cluster ,in For the first A role cluster is formed, encouraging roles within the same cluster to move closer to each other while maintaining distinction from robots in other clusters; for the first... A robot, representing its role Let q be the query, and the role representations of other robots be denoted as the key set. ; belongs to the same cluster as the query (i.e. The key is designated as a positive key. Keys from different clusters are assigned as negative keys. The bilinear product is used as the scoring function, where the similarity between the query and the key is determined by... Calculate W, a learnable parameter matrix; calculate the contrastive loss function and update the network according to the following formula:

[0099]

[0100]

[0101] in, Indicates the positive sample index. This represents the positive sample role. Indicates the negative sample index. This represents the role of the negative sample.

[0102] Based on the observation information, global state, and role representation obtained from S1, a state value assessment network is constructed to assess the state value of the robot's current state. The specific steps are as follows:

[0103] Observation information Global status information and character representation The observation information is input into the state value assessment network, which then processes it as follows: After processing by the MLP, the data is input into the GRU (Generative Recurrent Neural Network) for extracting temporal information, generating the robot's temporal representation. The global state representation is then processed by the MLP and input into the GRU again, generating a global state temporal representation. The state temporal representation is then used as a query vector, and the role representation is used as a key and value vector, respectively, and input into an attention network. Finally, the output of the attention network is concatenated with the temporal representation and processed by the MLP to output the state value. The specific formulas for these steps are as follows:

[0104]

[0105]

[0106]

[0107] in, express Time of the first Temporal representation of robot observations; express The global state temporal representation at any given moment; express The global state temporal representation at any given moment; This represents the output information of the attention network; Indicates the first A robot The value of a state at any given moment.

[0108] The Encoder and Decoder modules are trained based on state values ​​to obtain optimized Encoder and Decoder modules; the optimized role representation learning module, Encoder module, and Decoder module are integrated into a CRMAT network model; the process of training the Encoder and Decoder modules based on state values ​​is as follows:

[0109] From the experience replay pool Mid-sampling yields data for a complete round. , , , ,in , ;

[0110] Calculate the advantage function and the target state value:

[0111]

[0112]

[0113]

[0114] in, and They represent Time and The value of a state at any given moment; It is a discount factor. ; express The reward value at any given moment; Indicates timing difference error; Represents the dominance function; Indicates the value of the target state; It is the attenuation factor;

[0115] Calculate the loss function and update the entire network:

[0116]

[0117] in, Operations should be implemented to prevent excessive updates; Strategy entropy is used to enhance exploration; The entropy regularization coefficient; This refers to the cropping area.

[0118] S5: Decompose the task to be processed into several steps, and process each step using the CRMAT network model until the task is completed.

[0119] Experimental verification

[0120] This invention proposes a multi-robot collaborative working method based on contrastive learning role representation. It utilizes contrastive learning and attention mechanisms to address the issues of homogenized agent behavior and credit allocation, improving the algorithm's performance in multi-agent collaborative tasks. Performance evaluation was conducted in two cooperative multi-agent environments: StarCraft (SMAC) and Google Football (GRF), as shown in Table 1. The environment name is in the first column, the task scenario and difficulty level are in the second and third columns respectively, and the evaluation data are listed in the following columns. "CR-MAT" represents the reinforcement learning algorithm based on contrastive learning role representation, which is the algorithm proposed in this invention; "MAT" represents the reinforcement learning algorithm using Transformer based on sequence modeling; "QMIX" represents the algorithm combining Q-values ​​for cooperative multi-agent learning; "ROMA" represents the reinforcement learning algorithm that assigns roles based on each agent's current observation; and "GoMARL" represents the reinforcement learning algorithm using an automatic grouping mechanism. This experiment used three different random seeds for training. During training, each method was evaluated 32 times every 10,000 time steps, and the average data was recorded. For each random seed, the median of the last ten evaluations is calculated to represent the performance of that seed. Finally, the mean and standard deviation of these medians across the three seeds represent the overall performance. The values ​​following the SMAC environment represent the win rate, the values ​​following the GRF environment represent the score, the values ​​in parentheses represent the standard deviation, and the bolded values ​​indicate the optimal value obtained by the algorithm.

[0121] Table 1

[0122]

[0123] As shown in Table 1, the algorithm proposed in this invention outperforms the other four comparative algorithms in both the SMAC and GRF environments. The advantages of the CR-MAT algorithm compared to other comparative methods mainly lie in its dynamic discriminative role modeling and refined credit allocation mechanism. This method combines contrastive learning with a Transformer structure to obtain highly discriminative and adaptively updatable role embeddings, effectively alleviating the homogenization problem of agents caused by traditional parameter sharing. Simultaneously, role-based attention state value estimation enables more accurate credit allocation at the global level, allowing for a reasonable balance of contributions from each agent at different task stages. Furthermore, role information is explicitly introduced during action generation, making the strategy more aligned with role responsibilities and improving collaboration efficiency.

[0124] Finally, it should be noted that the above embodiments are intended to illustrate the technical solutions of the present invention and do not constitute any limitation on the present invention. Those skilled in the art should fully understand that modifications to the technical solutions described in the foregoing embodiments or equivalent substitutions for any part or all of the technical features are entirely feasible. Such modifications or substitutions, as long as they do not depart from the scope of protection defined by the claims of the present invention, should be considered reasonable extensions of the present invention.

Claims

1. A multi-robot collaborative working method based on contrastive learning role representation reinforcement learning algorithm, characterized in that, include: S1: Each robot acquires its own observation information at the current moment and merges the observation information of all robots at the current moment into a global state; A role representation learning module is constructed. Based on the observation information, the role representation learning module is used to perform temporal modeling and role modeling for each robot, and generate the corresponding temporal representation and role representation at the current moment. S2: Based on the temporal and role representations of each robot at the current moment, construct the Encoder module to perform spatial modeling for each robot and generate information after the current robot interacts with each other. S3: Based on the interaction information obtained in S2, construct the Decoder module, make decisions according to the serialization principle, and make decision selections for each robot in an autoregressive manner; S4: The character representation learning module is optimized and trained based on the contrastive learning algorithm to obtain the optimized character representation learning module; Based on the observation information, global state, and role representation obtained from S1, a state value assessment network is constructed to assess the current state of the robot. The Encoder and Decoder modules are trained based on the state value to obtain optimized Encoder and Decoder modules. The optimized role representation learning module, Encoder module, and Decoder module are integrated into a CRMAT network model. S5: Decompose the task to be processed into several steps, and process each step using the CRMAT network model until the task is completed.

2. The multi-robot collaborative working method based on contrastive learning role representation reinforcement learning algorithm according to claim 1, characterized in that, The specific steps of the role representation learning module in performing temporal modeling and role modeling for each robot are as follows: exist At any given moment, each robot receives observational information about its current environment, represented as... ,in Indicates the first A robot Real-time observation information, including its own state, surrounding environment, and task-related information; and simultaneously, by integrating all robot data. The observation information at each moment constructs a global state representation of the current environment. , will observe information With global state representation Store them together in the experience replay pool middle; Observation information After being processed by the MLP, the data is input into the GRU (Government Recurrent Neural Network) for extracting temporal information, while simultaneously processing the temporal representation from the previous time step. It is also input into the recurrent neural network GRU to obtain the robot's... Temporal representation of time The temporal representation of all robots constitutes , The learning formula is as follows: in, Indicates the first A robot Temporal representation of moments; For the first A robot The temporal representation of each moment is initialized as a 0 vector; Temporal representation of any robot The robot's role representation is obtained by inputting the data into a multilayer perceptron (MLP). The role representation of all robots constitutes , The learning formula is as follows: in, Indicates the first A robot The role representation of time.

3. The multi-robot collaborative working method based on contrastive learning role representation reinforcement learning algorithm according to claim 2, characterized in that, The Encoder module performs spatial modeling of the robot, generating information about the robot's interactions with each other, specifically including: Temporal representation of any robot With role representation The data is concatenated and input into the Encoder module. After processing by the attention mechanism and the Multilayer Perceptron (MLP), the spatially modeled interaction information is obtained. The specific formula is shown below: in, This represents a function that concatenates temporal representations with role representations; This represents the information obtained by concatenating the temporal representation and the role representation; , , These represent the parameter matrices used to generate the query vector, key vector, and value vector in the self-attention mechanism, respectively. The dimension of the query vector is represented by the softmax function, which calculates the distribution of attention weights. Representation layer normalization; This represents the attention-weighted features calculated through a self-attention mechanism; This represents the final interactive information output after spatial modeling.

4. The multi-robot collaborative working method based on contrastive learning role representation reinforcement learning algorithm according to claim 3, characterized in that, The specific steps for constructing the Decoder module to make decisions for each robot are as follows: First, initialize an empty action sequence. Each decision action generated by the robot is stored in this sequence using one-hot encoding, with a length equal to the action space dimension plus 1. Initially, the first bit of the sequence is set to 1, and the remaining bits are 0. At each time step, the previous... The robot's action sequence The input is fed into the first-layer masked attention network, and the output is the integrated information. : in, Represents the mask matrix; Interactive information output by the Encoder module This serves as the query vector for the second-layer attention network, while simultaneously incorporating features extracted from the first-layer masked attention network. With action sequence After concatenation, they together serve as the key and value vectors of the second-layer attention network, outputting the information processed by the second-layer attention mechanism. : After that and splicing together information The information is then processed by an MLP and output as the processed information. : Then and After concatenation, the data is processed by an MLP to output the final decision information. And convert it into one-hot encoding and put it back into the action sequence: in, yes Time before The action sequence information of each robot; Indicates the first The decision-making of a robot; After obtaining the actions of all robots Afterwards, the action was performed and obtained Momentary Rewards and will and All are stored in the experience replay pool middle.

5. The multi-robot collaborative working method based on contrastive learning role representation reinforcement learning algorithm according to claim 4, characterized in that, The optimization training of the role representation learning module based on the contrastive learning algorithm includes the following training process: From the experience replay pool Mid-sampling yields data for a complete round. ,in ; For each moment Execute step S1 to obtain the temporal representation of all robots. and role representation ; Regularly The robots are clustered based on their temporal representations. Cluster ,in For the first A role cluster is formed, encouraging roles within the same cluster to move closer to each other while maintaining distinction from robots in other clusters; for the first... A robot, representing its role Let q be the query, and the role representations of other robots be denoted as the key set. Keys belonging to the same cluster as the query are specified as positive keys. Keys from different clusters are assigned as negative keys. The bilinear product is used as the scoring function, where the similarity between the query and the key is determined by... Calculate W, a learnable parameter matrix; calculate the contrastive loss function and update the network: in, Indicates the positive sample index. This represents the positive sample role. Indicates the negative sample index. This represents the role of negative samples.

6. The multi-robot collaborative working method based on contrastive learning role representation reinforcement learning algorithm according to claim 5, characterized in that, The specific steps of the state value assessment network in assessing the state value of the robot are as follows: Observation information Global status information and character representation The observation information is input into the state value assessment network, which then processes it as follows: After being processed by the MLP, the data is input into the GRU (Government Recurrent Neural Network) for extracting temporal information to generate the robot's temporal representation; the global state representation is also processed by the MLP and then input into the GRU for extracting temporal information to generate the global state temporal representation. The state temporal representation is then used as the query vector, and the role representation is used as the key vector and value vector, which are then input into the attention network. The output of the attention network is then concatenated with the temporal representation and passed through an MLP to output the state value; the formula is as follows: in, express Time of the first Temporal representation of robot observations; express The global state temporal representation at any given moment; express The global state temporal representation at any given moment; This represents the output information of the attention network; Indicates the first A robot The value of a state at any given moment.

7. The multi-robot collaborative working method based on contrastive learning role representation reinforcement learning algorithm according to claim 6, characterized in that, The process of training the Encoder and Decoder modules based on state values ​​is as follows: From the experience replay pool Mid-sampling yields data for a complete round. , , , ,in , ; Calculate the advantage function and the target state value: in, and They represent Time and The value of a state at any given moment; It is a discount factor. ; express The reward value at any given moment; Indicates timing difference error; Represents the dominance function; Indicates the value of the target state; It is the attenuation factor; Calculate the loss function and update the entire network: in, Operations should be implemented to prevent excessive updates; Strategy entropy is used to enhance exploration; The entropy regularity coefficient; This refers to the cropping area.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that, When the processor executes the computer program, it causes the electronic device to perform the multi-robot collaborative work method based on the contrastive learning role representation reinforcement learning algorithm as described in any one of claims 1-7.

9. A storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, the electronic device performs the multi-robot collaborative work method based on the contrastive learning role representation reinforcement learning algorithm as described in any one of claims 1-7.

Images