A multi-agent centralized decision-making method and system based on hierarchical representation and a storage medium

Abstract

The application discloses a kind of multi-agent centralized decision-making method, system and storage medium based on hierarchical representation.The method is realized based on reinforcement learning policy network, specifically includes: the situation information perceived by multi-agent is input into policy network, the first module of policy network is based on the hierarchical feature extraction of the data type of situation information, each layer situation information data is processed respectively and then feature fusion is carried out, and the processed situation information is output;The second module of policy network further processes the processed situation information;The third module of policy network carries out hierarchical decision-making according to action parameter type, and each action parameter is output by processing in turn, to obtain the joint action of final output.The high-dimensional situation information and action information brought by multi-agent centralized decision-making are reduced in dimension by adopting hierarchical method based on type, and the search efficiency in high-dimensional space is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a decision optimization method and system based on multi-agent reinforcement learning and hierarchical representation and a storage medium. BACKGROUND

[0002] In recent years, intelligent technologies centered on deep learning (DL) have made great progress. In particular, intelligent methods represented by deep reinforcement learning (DRL) have achieved numerous results that surpass human level in solving decision-making problems such as Atari games, chess game confrontation, and real-time strategy (RTS) games, making it possible for autonomous cognitive intelligent decision-making to be further developed and achieve a key breakthrough. Reinforcement learning (RL) is an important method for solving sequential decision-making problems. It learns strategies through trial and error with the environment, which coincides with the experience learning and decision-making thinking of humans, and can effectively solve the problem of difficult sample data acquisition.

[0003] Single-agent systems cannot realize mutual cooperation or competition between multiple decision-makers, and often need to be solved by multiple agents through cooperation. Multi-agent systems (MAS) are a research hotspot in the current field of distributed AI. They mainly study the cooperation and interaction between agents to solve collaborative problems such as task scheduling, resource allocation, behavior coordination, and conflict resolution in complex real-time dynamic multi-agent environments.

[0004] Multi-agent systems can generally be divided into multi-agent decentralized decision-making and multi-agent centralized decision-making. Document 1 discloses a multi-agent decentralized decision-making method. In this method, multiple agents have high autonomy to process information, plan and make decisions, and execute instructions, and communicate with other agents to coordinate their behaviors, which makes the method inefficient in cooperation and difficult to guarantee the realization of global goals. Multi-agent centralized decision-making can effectively improve decision-making efficiency, but it still faces the following two problems when solving similar real-time strategy games or war game decision-making problems.

[0005] One is the problem of high-dimensional state-action space. In the problem of real-time strategy games or war game deduction, due to the wide range of geographical environment, multi-source situation information data, numerous combat entities and long time span, the state-action space of decision-making will grow exponentially, which will make the efficiency of traditional multi-agent reinforcement learning method low. Literature 2 proposes a hierarchical reinforcement learning (HRL) method which can effectively solve the problem of dimension disaster, which discloses the hierarchical learning mode based on options, hierarchical abstraction, value function decomposition and end-to-end. However, the above-mentioned disclosed mode still has the problem of how to maximize the search efficiency of high-dimensional space in the problem of real-time strategy games or war game deduction, which is a difficult problem to be solved in intelligent decision-making.

[0006] Two is the balance between exploration and utilization. In the deep reinforcement learning method, the exploration of unknown strategy space and the utilization of known strategy space are a contradictory problem. At the same time, in the high-dimensional unknown battlefield space and the strange and variable opponent, efficient exploration and trial of different actions can collect more battlefield unknown area information and opponent strategy information, and the best decision-making under the known strategy information can seize the opportunity in the enemy game confrontation process, improve the situation advantage, and then obtain the competitive advantage. Therefore, in the process of intelligent decision-making based on deep reinforcement learning, how to balance the exploration of unknown strategy space and the utilization of known strategy space, and improve the diversity and robustness of strategy, is a key problem to be solved.

[0007] Prior art documents:

[0008] Document 1: CN114896899A, "Multi-agent decentralized decision-making method and system based on information interaction";

[0009] Document 2: "Multi-agent hierarchical reinforcement learning review", Yin Changsheng et al., "Journal of Intelligent Systems", Vol. 15, No. 4, pp. 646-655, July 2020. SUMMARY

[0010] In view of at least one defect or improvement demand of the prior art, the present application provides a multi-agent centralized decision-making method, system and storage medium based on hierarchical representation, to solve at least one of the problems of high-dimensional space search efficiency and the balance between exploration and utilization.

[0011] To achieve the above-mentioned purpose, according to the first aspect of the present application, a multi-agent centralized decision-making method based on hierarchical representation is provided, characterized in that the decision-making method is realized by a policy network based on reinforcement learning, and the decision-making method specifically comprises the following steps:

[0012] S1. inputting the situation information perceived by the multi-agent into the policy network, a first module of the policy network performing hierarchical feature extraction based on the data type of the situation information, performing feature fusion after processing the situation information data of each layer respectively, and outputting the processed situation information;

[0013] S2. a second module of the policy network further processing the processed situation information;

[0014] S3. a third module of the policy network performing hierarchical decision-making according to the action parameter type, sequentially processing and outputting each action parameter, and obtaining the final output joint action.

[0015] Further, the multi-agent centralized decision-making method based on hierarchical representation, the step S1 specifically comprises:

[0016] S11. layering the situation information according to the data type to obtain the layered situation information;

[0017] S12. designing different neural networks for processing according to the data characteristics of the layered situation information, and extracting the corresponding feature vectors of each layer of situation information;

[0018] S13. parallelly outputting the feature vectors, and inputting the feature vectors into a deep long short-term memory network to output the processed situation information.

[0019] Further, the multi-agent centralized decision-making method based on hierarchical representation, the step S11 specifically comprises:

[0020] simplifying and abstracting the situation information into layers, wherein the layers at least include geographic information, entity information and scalar information;

[0021] The geographic information at least includes the combat region, the terrain, the elevation; the entity information at least includes the combat entity type, the position information and the state information; and the scalar information at least includes the combat time, the combat damage situation of the enemy and ourselves, and the combat task and completion situation.

[0022] Further, the multi-agent centralized decision-making method based on hierarchical representation, the step S12 specifically comprises: at least using a convolutional neural network to process the geographic information, at least using a deep self-attention network to process the entity information, and at least using a multi-layer perception to process the scalar information.

[0023] Further, the multi-agent centralized decision-making method based on hierarchical representation, the step S3 specifically comprises:

[0024] receiving the information processed by the second module in the step S2 and performing hierarchical design, which is divided into NEach decision-making action, among which N ≥2 and N It is a positive integer;

[0025] For the i Layer decision action (1≤ i ≤ N Based on the characteristics of the actions at this layer, a corresponding neural network is designed for processing to obtain the first... i Decision actions output by the layer a t i ;Will a t i Pass to the i +1 layer performs corresponding processing; the joint action is represented as A t =( a t 1 , a t 2 ,..., a t N ).

[0026] Furthermore, in the above-mentioned multi-agent centralized decision-making method based on hierarchical representation, step S3 further includes:

[0027] The information processed by the second module is divided into at least action type, execution time, execution unit, execution region, and action parameters. The above action type, execution time, execution unit, execution region, and action parameters are trained sequentially using residual multilayer perceptron, multilayer perceptron, pointer network, deconvolution network, and multilayer perceptron network, respectively. Finally, decisions are made in sequence to form the joint action.

[0028] Furthermore, the aforementioned multi-agent centralized decision-making method based on hierarchical representation also includes:

[0029] The policy network outputs the joint action using a random policy; the random policy refers to the output distribution of the policy network being... ,in The mean m ,variance n The normal distribution S t for t The situation at any moment, θ μ For θ Policy network with parameters μ The method for obtaining the joint action is based on probability. ε Randomly select action information, or from The action information is obtained by sampling.

[0030] Further, the multi-agent centralized decision-making method based on hierarchical representation further comprises:

[0031] On the basis of the random strategy, maximum strategy entropy is introduced; the maximum strategy entropy is introduced in a manner of modifying the reinforcement learning target, so that the reinforcement learning target is changed to maximize the cumulative expected return while maximizing the strategy entropy, that is, the target function can be represented as:

[0032]

[0033] Wherein, r ( s t , a t ) represents the action taken by the agent at the moment t s t under the situation a t The obtained immediate reward, represents the expectation of the action probability distribution of the agent at the moment t s t under the situation is the strategy π The strategy entropy of the state s t ; α is a temperature coefficient, used to control the importance of the strategy entropy in the target function.

[0034] According to the second aspect of the present application, a multi-agent centralized decision-making system based on hierarchical representation is also provided, which comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of any one of the above-mentioned methods.

[0035] According to the third aspect of the present application, a computer readable storage medium is also provided, which stores a computer program executable by an access authentication device, and when the computer program runs on the access authentication device, the access authentication device executes the steps of any one of the above-mentioned methods.

[0036] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0037] ​​(1) The application provides a multi-agent centralized decision-making method based on hierarchical representation, which respectively from the situation information input and the policy action output two aspects, adopts the action parameter type based on the situation information data type and the policy action to carry out hierarchical, carries out the targeted processing to different layers respectively, can effectively reduce the neural network complexity, compared with the hierarchical mode in the prior art can maximize the search efficiency of high-dimensional space.

[0038] (2) The application provides a multi-agent centralized decision-making method based on hierarchical representation, adopts the maximum entropy random policy gradient method, encourages the agent to explore the policy space as much as possible through the randomness policy, and integrates the maximum policy entropy into the objective function, so that the agent considers the richness of the policy and the maximization of the benefit, enhances the diversity and robustness of the decision policy, and avoids the single and targeted agent policy. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings required to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0040] Figure 1 A flowchart of a multi-agent centralized decision-making method based on hierarchical representation provided by the embodiment of the application is shown.

[0041] Figure 2 A basic flowchart of a multi-agent centralized decision-making method based on hierarchical representation provided by the embodiment of the application is shown.

[0042] Figure 3 A general flowchart of a multi-agent centralized decision-making method based on hierarchical representation provided by the embodiment of the application is shown.

[0043] Figure 4 A classification situation feature extraction and fusion strategy diagram provided by the embodiment of the application is shown.

[0044] Figure 5 A decision-making process diagram for action layer provided by the embodiment of the application is shown.

[0045] Figure 6 A multi-agent centralized decision-making implementation architecture diagram based on hierarchical representation provided by the embodiment of the application is shown. DETAILED DESCRIPTION

[0046] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0047] The terms "first", "second", "third" and the like in the description, claims, and drawings of the present application are used to distinguish different objects, and are not used to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0048] As shown in Figures 1-3 The present application provides a multi-agent decentralized decision-making method based on information interaction, which is realized by a policy network based on reinforcement learning. The decision-making method specifically includes the following steps:

[0049] S1. Input the situation information perceived by the multi-agent into the policy network. The first module of the policy network performs hierarchical feature extraction based on the data type of the situation information. After processing each layer of the situation information data, the features are fused, and the processed situation information is output.

[0050] S2. The second module of the policy network further processes the processed situation information.

[0051] S3. The third module of the policy network performs hierarchical decision-making according to the action parameter type, sequentially processes and outputs each action parameter, and obtains the final output joint action.

[0052] The perception and action ability of a single agent (Agent) are relatively limited. The multi-agent system shares information such as actions, states and strategies through interaction, which can effectively improve system efficiency and complete more complex tasks, and has unique advantages in dealing with complex environmental problems such as real-time strategy games or war game problems.

[0053] In real-time strategy games or war game problems, due to the wide geographical environment, multi-source situation information data, numerous combat entities and long time span, the situation information perceived by the multi-agent and the decision-making action information output by the multi-agent grow exponentially, i.e. the dimension disaster problem, which will cause the traditional reinforcement learning method to be inefficient.

[0054] To solve the above problems, in the embodiment of the present application, the situation information perceived by the multi-agent and the output information of the strategy network are respectively layered based on the data type of the situation information and the action parameter type of the strategy action, and different layers are processed respectively, which can effectively reduce the complexity of the neural network and maximize the search efficiency of the high-dimensional space compared with the layering method in the prior art.

[0055] As shown in Figure 4 In this embodiment, according to the characteristics of the situation information data in the real-time strategy game or the war game problem, the present application proposes a specific situation information layering method as follows:

[0056] S11. Layering the situation information according to the data type to obtain the layered situation information;

[0057] S12. Designing different neural networks to process according to the data characteristics of the layered situation information, and extracting the corresponding feature vectors of each layer of situation information;

[0058] S13. Parallel output of the feature vectors, and common input of the feature vectors into a deep long short-term memory network to output the processed situation information.

[0059] Further, step S11 in this embodiment specifically includes:

[0060] simplifying and abstracting the situation information into layers, wherein the layers include at least geographic information, entity information and scalar information;

[0061] The geographic information includes at least the combat region, the terrain, the elevation, the entity information includes at least the combat entity type, the position information and the state information, and the scalar information includes at least the combat time, the enemy and friendly forces damage situation, the combat task and the completion situation.

[0062] The step S12 specifically includes: at least using a convolutional neural network to process the geographic information, at least using a deep self-attention network to process the entity information, and at least using a multilayer perception to process the scalar information.

[0063] The situation information data in the common real-time strategy game or war game problem includes at least geographic information, entity information and scalar information. The geographic information includes combat region, terrain, elevation and other geographic environment information, and the characteristic of the geographic information is that it is often presented in the form of graphics or images. Convolutional neural network is a common core algorithm in image recognition field, therefore, in this embodiment, convolutional neural network is used to process the geographic information.

[0064] The entity information includes combat entity types, position information, state information, and combat entity information of both sides, which is characterized by being local information, and information is associated with each other, for example, there is a combat entity of a certain type at a certain position, which is in a healthy / sick state. For the above information, the deep self-attention network processing can achieve good results in the embodiment.

[0065] The scalar information includes combat time, enemy and friendly war damage, combat tasks and completion, and some situation measurement information for describing, which is characterized by being global information, and information is relatively independent, so the multi-layer perception machine is used to process global information in the embodiment.

[0066] Finally, the geographical information processed by the convolutional neural network, the entity information processed by the deep self-attention network, and the global information processed by the multi-layer perception machine need to be combined into processed situation information, i.e. the first information, through the deep long short-term memory network.

[0067] Further, as shown in Figure 5 , the step S3 further includes:

[0068] The information processed by the second module in the step S2 is received and designed in layers, which is divided into N decision actions, wherein N ≥2 and N is a positive integer;

[0069] For the first i layer decision action (1≤ i ≤ N ), a corresponding neural network is designed according to the characteristics of the layer action to process and obtain the decision action of the first i layer output a t i ; and a t i is transmitted to the i +1 layer for corresponding processing; the joint action is represented as A t =( a t 1 , a t 2 ,..., a t N ).

[0070] Further, the step S3 in the embodiment further includes:

[0071] The information processed by the second module is divided into at least action type, execution time, execution unit, execution region, and action parameters. The above action type, execution time, execution unit, execution region, and action parameters are trained sequentially using residual multilayer perceptron, multilayer perceptron, pointer network, deconvolution network, and multilayer perceptron network, respectively. Finally, decisions are made in sequence to form the joint action.

[0072] This embodiment employs a multi-agent centralized decision-making approach, ensuring that the information processed by the second module remains high-dimensional. Therefore, this embodiment adopts a similar approach, dividing this high-dimensional information into multiple layers of decision-making action information based on action parameter types. Different neural networks are designed according to the characteristics of each layer of actions to process the corresponding decision-making action information, thereby improving decision-making efficiency.

[0073] The difference between this second module and the first module's hierarchical approach lies in the sequence of information processed by the second module and used as decision-making actions; each earlier decision-making action influences the next. Therefore, as... Figure 4 As shown, when designing the hierarchical information after the second module has processed the information, the results of the previous layer's decision-making actions need to be taken as a factor in the subsequent layer's decision-making actions, that is, as an input variable for the subsequent layer's decision-making actions.

[0074] Those skilled in the art will understand that the layering method described in this invention is not limited to the examples given in this embodiment. That is, the layering method for situational information is not limited to dividing situational information into three categories: geographic information, entity information, and scalar information. Similarly, the layering method for decision-making actions is not limited to dividing them into five categories: action type, execution time, execution unit, execution region, and action parameters. As long as the layering is based on the situational information data type and the action parameter type of the strategy action described in this invention, the technical effect of effectively reducing the complexity of the neural network and maximizing the search efficiency in high-dimensional space can be achieved.

[0075] Furthermore, this embodiment also includes:

[0076] The policy network outputs the joint action using a random policy; the random policy refers to the output distribution of the policy network being... ,in The mean m ,variance n The normal distribution S t for t The situation at any moment, θ μ For θ Policy network with parameters μ The method for obtaining the joint action is based on probability.ε randomly selecting action information, or sampling action information from .

[0077] Further, the multi-agent centralized decision-making method based on hierarchical representation described above further comprises:

[0078] On the basis of the random strategy, maximum strategy entropy is introduced; the introduction of the maximum strategy entropy is in the form of modifying the goal of reinforcement learning, so that the goal of the reinforcement learning becomes maximizing the cumulative expected return while maximizing the strategy entropy, that is, the objective function can be represented as:

[0079]

[0080] wherein, r ( s t , a t ) represents t the action taken by the agent at the moment s t under the state a t the immediate reward obtained, the expectation of the action probability distribution at the moment t under the state s t . the strategy entropy of the strategy π under the state s t . α the temperature coefficient, used to control the importance of the strategy entropy in the objective function.

[0081] The embodiment adopts a strategy selection based on random action, that is, the output distribution of the strategy network is , wherein is a normal distribution with mean m and variance n , and the joint action output by the strategy network is obtained in the form of randomly selecting action information with a probability ε or sampling the hierarchical joint action from . By outputting the action information in the form of random selection instead of fixed correspondence, the randomness of the strategy can be effectively improved, the agent is encouraged to explore the strategy space as much as possible, and the richness of the strategy is improved.

[0082] Further, on the basis of the random strategy, maximum strategy entropy is introduced in the embodiment; the introduction of the maximum strategy entropy is in the form of modifying the goal of reinforcement learning, so that the goal of the reinforcement learning becomes maximizing the cumulative expected return while maximizing the strategy entropy, that is, the objective function can be represented as:

[0083]

[0084] The objective function incorporates a policy entropy parameter, namely... For strategy π In state s t strategy entropy; α The temperature coefficient controls the importance of policy entropy in the objective function, i.e., the degree of randomness in the agent's policy. α If the value is 0, it becomes a reinforcement learning algorithm that does not consider policy entropy.

[0085] Furthermore, such as Figure 6 As shown, the policy network structure in this embodiment also includes a value network, which adopts an Actor-Critic (AC) model and is divided into an online policy network, a target policy network, an online value network, and a target value network. The agent is in a joint state... S t At that time, the action will first be output through online policy network decision-making. μ ( S t ), at this time μ ( S t This is a probability distribution of a series of actions, which needs to be sampled to form the final combined actions acting on the simulation environment. A t The intelligent agent receives a reward after interacting with the simulation environment. R t State at the next moment S t+1 This process is repeated continuously to generate trial-and-error sample tuples. S t , A t , R t , S t+1 The data is then stored in the experience replay pool. If the experience replay pool is full, previous data is overwritten. During each training session, Minibatch data is randomly sampled from the experience replay pool to update the parameters.

[0086] Online policy network output policy distribution With probability ε Randomly select actions A t Otherwise from Mid-sampling yields hierarchical joint actions. A t, as the policy output of the online policy network, and store the data into the sample experience pool. Randomly sample a sample data from the experience pool to obtain a global value K Q’ total , and add the real-time global reward to obtain the target network value output:

[0087]

[0088] Use the mean squared error (MSE) of the target value network output and the online value network output as the loss function:

[0089]

[0090] Then use the optimizer to update the online value network, and update the online Q i Use the data randomly sampled from the storage buffer (Replay memory buffer) to perform unbiased estimation on the policy:

[0091]

[0092] Update the online policy network according to the gradient calculation: μ i The parameter update calculation method is shown in the formula:

[0093]

[0094] Wherein θ i Q , θ i μ Respectively represent the neural network parameters of the value network and the policy network.

[0095] The application also provides a multi-agent centralized decision system based on hierarchical representation, which comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of any one of the above-mentioned methods.

[0096] ​The application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the above method. The computer readable storage medium can include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nano system (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0097] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-agent centralized decision-making method based on hierarchical representation, characterized in that, The decision-making method is implemented by a policy network based on reinforcement learning, and specifically comprises the following steps: S1. inputting the situation information perceived by the multi-agent into the policy network, performing hierarchical feature extraction on the data types of the situation information by a first module of the policy network, performing feature fusion after processing each layer of the situation information data, and outputting the processed situation information; S2. further processing the processed situation information by a second module of the policy network; S3. performing hierarchical decision-making according to the action parameter types by a third module of the policy network, sequentially processing and outputting each action parameter to obtain a final output joint action; The step S1 specifically comprises: S11. layering the situation information according to the data types to obtain layered situation information; S12. designing different neural networks according to the data characteristics of the layered situation information to process and extract the corresponding feature vectors of each layer of situation information; S13. parallelly outputting the feature vectors and inputting the feature vectors into a deep long short-term memory network to output the processed situation information; The step S11 specifically comprises simplifying and abstracting the situation information, wherein the layering at least includes geographic information, entity information and scalar information; wherein the geographic information at least includes the combat region, the terrain, the elevation; the entity information at least includes the combat entity type, the position information, the state information; and the scalar information at least includes the combat time, the enemy and friendly war damage, the combat task and the completion situation.

2. The hierarchical representation-based multi-agent centralized decision-making method according to claim 1, wherein, The step S12 specifically comprises: At least a convolutional neural network is used to process the geographic information, at least a deep self-attention network is used to process the entity information, and at least a multilayer perceptron is used to process the scalar information.

3. The hierarchical representation based multi-agent centralized decision-making method of claim 1, wherein, The step S3 specifically comprises: Receiving the information processed by the second module in the step S2 and carrying out hierarchical design, which is divided into N decision actions, wherein N ≥2 and N is a positive integer; For the i Layer decision action (1≤ i ≤ N Based on the characteristics of the actions at this layer, a corresponding neural network is designed for processing to obtain the first... i Decision actions output by the layer a t i ;Will a t i Pass to the i +1 layer performs corresponding processing; the joint action is represented as A t =( a t 1 , a t 2 ,..., a t N ).

4. The multi-agent centralized decision-making method based on hierarchical representation according to claim 3, wherein, The step S3 further comprises: The information processed by the second module is at least divided into action type, execution time, execution unit, execution region and action parameter; the above-mentioned action type, execution time, execution unit, execution region and action parameter are sequentially trained by using residual multilayer perceptron, multilayer perceptron, pointer network, deconvolution network and multilayer perceptron network respectively, and finally sequentially decided in order to form the joint action.

5. The multi-agent centralized decision-making method based on hierarchical representation according to claim 1, characterized in that: The policy network outputs the joint action in a manner of a random policy; the random policy refers to that an output distribution of the policy network is , wherein is a mean m , and a variance n is a normal distribution S t is a state at a moment t , and θ μ is a policy network with θ as a parameter μ ; the joint action is obtained in a manner of randomly selecting action information with a probability ε , or sampling the action information from .

6. The multi-agent centralized decision-making method based on hierarchical representation according to claim 5, characterized in that: On the basis of the random policy, a maximum policy entropy is introduced; the maximum policy entropy is introduced by modifying the goal of reinforcement learning, so that the goal of reinforcement learning is to maximize the cumulative expected return while maximizing the policy entropy, that is, the objective function can be represented as: wherein r s t , a t represents t the time instant s t an action is taken under the situation a t the immediate reward obtained, represents the expectation of the action probability distribution under the situation t the time instant s t when the policy is employed, is the policy π at state s t the policy entropy; α is a temperature coefficient, used to control the importance of the policy entropy in the objective function.​ 7. A multi-agent centralized decision system based on hierarchical representation, characterized in that, A computer program product comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program which, when executed by the processing unit, causes the processing unit to perform the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, which stores a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images