A decision-making control method, device and system for intelligent group interaction

Abstract

The present invention discloses a decision control method, device and system for group interaction of intelligent agents. The decision control device includes an initial interaction unit, a model training unit and a decision control unit. The decision control system also includes a decision control module and a data storage module. By constructing an initial decision control model including a top-level learning model and a bottom-level learning model, and performing top-level and bottom-level fusion training on the initial decision control model, thereby obtaining a final decision control model and then performing decision control, the decision control method, device and system improve the effectiveness of decision control during group interaction of intelligent agents.

Description

Technical Field

[0001] The present invention relates to the field of decision-making control of intelligent agent group interaction, and relates to a decision-making control method, device and system for intelligent agent group interaction. Background Art

[0002] In large-scale group interaction scenarios, such as massively multiplayer online role-playing games, equity trading markets, online advertising auctions, urban traffic, and military intelligence clusters, massive individuals act concurrently in the same environment and adjust their strategies in real time. The dynamics and scale of these environments pose new challenges to multi-agent reinforcement learning algorithms.

[0003] In the prior art, group interaction is usually controlled by making decisions through the MADDPG algorithm based on the centralized training distributed execution (CTDE) learning model, the VDN algorithm based on the value decomposition idea, or the learning method based on the mean field theory; among them, the MADDPG algorithm based on the centralized training distributed execution (CTDE) learning model uses the centrally controlled Critic network to obtain the status, behavior and target strategy of all individuals in the training phase, and each agent Actor makes decisions based on local information in the execution phase; in the VDN algorithm based on the value decomposition idea, each agent maximizes the global benefit function by maximizing the local benefit function, thereby achieving the coordination of multiple agents (both of which achieve the coordination between agents by characterizing the interaction between individuals); the learning method based on the mean field theory can perform a macroscopic representation of state information and action information from the group level, thereby better solving the dimensionality curse and complex interaction problems in group decision-making.

[0004] However, the existing technology still has the following defects: it is unable to simultaneously take into account the coordination between individuals, the coordination between individuals and neighboring intelligent agents, and the coordination between groups, resulting in poor decision-making control effects during group interaction.

[0005] Therefore, there is a need for a decision-making control method, device and system for group interaction of intelligent agents, so as to overcome the above-mentioned defects in the prior art. Summary of the invention

[0006] In view of the above-mentioned existing technical problems, the purpose of the present invention is to provide a decision control method, device and system for group interaction of intelligent agents, so as to improve the effectiveness of decision control during group interaction of intelligent agents.

[0007] The present invention provides a decision-making control method for group interaction of intelligent agents, the decision-making control method comprising: obtaining a preset initial decision-making control model, allowing the group of intelligent agents to perform group interaction according to the initial decision-making control model, thereby obtaining an initial decision-making control data group; the initial decision-making control model comprises a top-level learning model and a bottom-level learning model; using the initial decision-making control data group, training the top-level learning model and the bottom-level learning model, thereby obtaining a final decision-making control model; and performing decision-making control on the group interaction of the intelligent agents according to the final decision-making control model.

[0008] In one embodiment, a preset initial decision control model is obtained, and a group of agents is made to perform group interaction according to the initial decision control model, thereby obtaining an initial decision control data group, specifically including: obtaining a preset initial decision control model and a preset opponent model of an opponent, initializing a preset group interaction platform, and obtaining a first state of the agent and a second state of the opponent; the initial decision control model includes a local neural network; inputting the first state into the local neural network to obtain a first behavior and a first reward, inputting the second state into the opponent model to obtain a second behavior and a second reward, and storing the first state, the second state, the first behavior, the second behavior, the first reward, and the second reward into the initial decision control data group; inputting the first behavior and the second behavior into the group interaction platform, thereby correspondingly obtaining a third state of the agent and a fourth state of the opponent; inputting the third state into the local neural network to obtain a third behavior and a third reward, inputting the fourth state into the opponent model to obtain a fourth behavior and a fourth reward, and storing the third state, the fourth state, the third behavior, the fourth behavior, the third reward, and the fourth reward into the initial decision control data group.

[0009] In one embodiment, the initial decision control data group is used to train the top-level learning model and the bottom-level learning model to obtain a final decision control model, specifically comprising: dividing the agent group into a corresponding number of small groups according to a preset number of groups, obtaining the average behavior value and reward and value of each small group according to the initial decision control data group; obtaining a learning target according to the average behavior value and reward and value of each small group; training the top-level learning model according to the learning target and the initial decision control data group to obtain a first top-level model and a corresponding first average neural network, and training the bottom-level learning model according to the first average neural network and the initial decision control data group, and recording the number of training times; judging whether the number of training times reaches a preset number threshold; when the number of training times reaches the preset number threshold, stopping the training and outputting the final decision control model.

[0010] In one embodiment, after determining whether the number of training times reaches a preset number threshold, the method further includes: when the number of training times does not reach the preset number threshold, continuing the model training.

[0011] The present invention also provides a decision control device for group interaction of intelligent agents, the decision control device comprising an initial interaction unit, a model training unit and a decision control unit, wherein the initial interaction unit is used to obtain a preset initial decision control model, so that the group of intelligent agents performs group interaction according to the initial decision control model, thereby obtaining an initial decision control data group; the model training unit is used to use the initial decision control data group to train a preset top-level learning model and a preset bottom-level learning model, thereby obtaining a final decision control model; the decision control unit is used to perform decision control on the group interaction of the intelligent agents according to the final decision control model.

[0012] In one embodiment, the initial interaction unit is also used to: obtain a preset initial decision control model and a preset opponent model of the opponent, initialize a preset group interaction platform, and obtain a first state of the agent and a second state of the opponent; the initial decision control model includes a local neural network; input the first state into the local neural network to obtain a first behavior and a first reward, input the second state into the opponent model to obtain a second behavior and a second reward, and store the first state, the second state, the first behavior, the second behavior, the first reward and the second reward in an initial decision control data group; input the first behavior and the second behavior into the group interaction platform, thereby correspondingly obtaining a third state of the agent and a fourth state of the opponent; input the third state into the local neural network to obtain a third behavior and a third reward, input the fourth state into the opponent model to obtain a fourth behavior and a fourth reward, and store the third state, the fourth state, the third behavior, the fourth behavior, the third reward and the fourth reward in the initial decision control data group.

[0013] In one embodiment, the model training unit is also used to: divide the intelligent agent group into a corresponding number of small groups according to a preset number of groups, and obtain the average behavior value and reward and value of each small group according to the initial decision control data group; obtain the learning target according to the average behavior value and reward and value of each small group; train the top-level learning model according to the learning target and the initial decision control data group to obtain the first top-level model and the corresponding first average neural network, and train the bottom-level learning model according to the first average neural network and the initial decision control data group, and record the number of training times; determine whether the number of training times reaches a preset number threshold; when the number of training times reaches the preset number threshold, stop training and output the final decision control model.

[0014] The present invention also provides a decision-making control system for group interaction of intelligent agents, which also includes a decision-making control module and a data storage module. The decision-making control module is communicatively connected to the data storage module, wherein the decision-making control module is used to perform decision-making control on the group interaction of the intelligent agent group according to the decision-making control method for group interaction of intelligent agents as described above, and the data storage module is used to store all data.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0016] The present invention provides a decision control method, device and system for group interaction of intelligent agents. An initial decision control model including a top-level learning model and a bottom-level learning model is constructed, and top-level and bottom-level fusion training is performed on the initial decision control model to obtain a final decision control model for decision control. The decision control method, device and system improve the effectiveness of decision control during group interaction of intelligent agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with the accompanying drawings, wherein:

[0018] Figure 1 A flowchart of an embodiment of a decision-making control method for group interaction of intelligent agents according to the present invention is shown;

[0019] Figure 2 A schematic diagram showing the training of the top-level learning model and the bottom-level learning model is shown;

[0020] Figure 3 A structural diagram showing an embodiment of a decision control device for group interaction of intelligent agents according to the present invention;

[0021] Figure 4A structural diagram of an embodiment of a decision-making control system for group interaction of intelligent agents according to the present invention is shown. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment 1

[0024] The embodiment of the present invention firstly describes a decision-making control method for group interaction of intelligent agents. Figure 1 A flow chart of an embodiment of a decision-making control method for group interaction of intelligent agents according to the present invention is shown.

[0025] like Figure 1 As shown, the decision control method includes the following steps:

[0026] S1: Obtain a preset initial decision control model, and enable the intelligent agent group to perform group interaction according to the initial decision control model, thereby obtaining an initial decision control data group.

[0027] The initial decision control model includes a top-level learning model and a bottom-level learning model.

[0028] In one embodiment, a preset initial decision control model is obtained, and a group of agents is made to perform group interaction according to the initial decision control model, thereby obtaining an initial decision control data group, specifically including: obtaining a preset initial decision control model and a preset opponent model of an opponent, initializing a preset group interaction platform, and obtaining a first state of the agent and a second state of the opponent; the initial decision control model includes a local neural network; inputting the first state into the local neural network to obtain a first behavior and a first reward, inputting the second state into the opponent model to obtain a second behavior and a second reward, and storing the first state, the second state, the first behavior, the second behavior, the first reward, and the second reward into the initial decision control data group; inputting the first behavior and the second behavior into the group interaction platform, thereby correspondingly obtaining a third state of the agent and a fourth state of the opponent; inputting the third state into the local neural network to obtain a third behavior and a third reward, inputting the fourth state into the opponent model to obtain a fourth behavior and a fourth reward, and storing the third state, the fourth state, the third behavior, the fourth behavior, the third reward, and the fourth reward into the initial decision control data group.

[0029] S2: Using the initial decision control data set, train the top-level learning model and the bottom-level learning model to obtain a final decision control model.

[0030] To further illustrate the initial decision control model, Figure 2 A schematic diagram of training the top-level learning model and the bottom-level learning model is shown.

[0031] Among them, the top-level learning coordinates the collaboration between several groups based on a macro perspective. Using the CTDE learning model, the learned collaborative information is applied to the mean field Q value network of each group, that is, the mean field information of multiple groups is used to obtain the mean field Q values ​​corresponding to multiple groups, and the sum is obtained to obtain a global Q value. At the same time, a global reward is obtained based on the real reward value obtained by each agent, and the goal of top-level learning is formed. Based on this goal, the parameters of the mean field Q value network are updated, and the updated mean field Q value network is passed to the bottom-level group for learning. The bottom-level model realizes the collaboration between agents in the group based on the mean field idea and the attention mechanism. The bottom-level learning is the collaborative learning of agents in the same group. Therefore, in this process, only the mean field Q value network updated by the top layer is received, and no information from agents in other groups is received. Each group contains multiple agents.

[0032] In practical applications, when in the top-level learning stage, based on the state information set o of each group i , action information set a i , the top layer relies on the weight matrix w(x i ) Statistical group information μ(x i ), where i is the group number, k is the agent number, and x i Equivalent to concatenate(o i ,a i ), which is the concatenation of state information and action information. i (x) is calculated as follows:

[0033]

[0034] Where N(i) represents the number of agents in the i-th group, Equivalent to Represents the state information of the kth agent in the i-th group; Represents the action information of the kth agent in the ith group.

[0035] Then, the statistical information μ(x i ) is passed to the mean field Q value network to calculate the Q corresponding to each group MF (μ(xi )), and finally obtain a global Q value Q through the summation module tot :

[0036]

[0037] In the formula, m is the number of groups.

[0038] After obtaining the global Q value, we need to minimize the loss function L based on the CTDE idea. tot Optimize Q tot , based on the loss function L tot Optimize Q tot , and then optimize Q MF (μ(x i )). Among them, the loss function formula is:

[0039]

[0040] In the formula,

[0041] The principle of Independent Global Max (IGM) is adopted here, that is, when the average field Q value Q of each group is MF (μ(x i )) reaches the maximum, the global Q value Q tot This mode is also based on the CTDE idea. In the top-level learning process, the average field Q value Q of other groups is combined MF (μ(x i )) belongs to the central training, but only the average field Q value Q of the group is used in the bottom group MF (μ(x i )), the average field Q value of all groups will not be collected.

[0042] In practical applications, when in the underlying learning stage, the mean information of the group As the central node, Calculate the importance weight of each agent in the i-th group based on the Attention mechanism where is the number of the k agent.

[0043] Among them, the importance weight The formula is:

[0044]

[0045] Where W K K key The parameters of the network, W Q K query The parameters of the network; Kkey Network and K query The network is the internal embedded network of the attention module. Each agent obtains t represents time.

[0046] Then, the mean field Q value Q passed by the top layer needs to be fused MF (μ(x i )), and then a group local Q value is obtained through the weighted summation module The local Q value of the group The formula is:

[0047]

[0048] In the formula, Refers to the importance of the kth agent in group i. By grouping local Q value This can achieve collaboration among multiple agents within the group, as well as collaboration between the bottom and top layers.

[0049] Based on the above, we need to minimize the loss function Training group Q function:

[0050]

[0051] In the formula, Among them, Q MF (μ(x i )) is the guidance from the top layer to the bottom layer, indicating the mean field Q value transmitted at the top layer. i is the true reward value returned by the environment for group i. is the training target for group i.

[0052] In one embodiment, the initial decision control data group is used to train the top-level learning model and the bottom-level learning model to obtain a final decision control model, specifically comprising: dividing the agent group into a corresponding number of small groups according to a preset number of groups, obtaining the average behavior value and reward and value of each small group according to the initial decision control data group; obtaining a learning target according to the average behavior value and reward and value of each small group; training the top-level learning model according to the learning target and the initial decision control data group to obtain a first top-level model and a corresponding first average neural network, and training the bottom-level learning model according to the first average neural network and the initial decision control data group, and recording the number of training times; judging whether the number of training times reaches a preset number threshold; when the number of training times reaches the preset number threshold, stopping the training and outputting the final decision control model.

[0053] S3: Perform decision control on the group interaction of the intelligent agents according to the final decision control model.

[0054] In one embodiment, after determining whether the number of training times reaches a preset number threshold, the method further includes: when the number of training times does not reach the preset number threshold, continuing the model training.

[0055] An embodiment of the present invention describes a decision control method for group interaction of intelligent agents. An initial decision control model including a top-level learning model and a bottom-level learning model is constructed, and top-level and bottom-level fusion training is performed on the initial decision control model to obtain a final decision control model for decision control. The decision control method improves the effectiveness of decision control during group interaction of intelligent agents. Specific embodiment 2

[0057] In addition to the above method, the embodiment of the present invention also describes a decision-making control device for group interaction of intelligent agents. Figure 3 A structural diagram of an embodiment of a decision-making control device for group interaction of intelligent agents according to the present invention is shown.

[0058] As shown in the figure, the decision control device includes an initial interaction unit 11, a model training unit 12 and a decision control unit 13.

[0059] The initial interaction unit 11 is used to obtain a preset initial decision control model, so that the agent group performs group interaction according to the initial decision control model, thereby obtaining an initial decision control data group. In one embodiment, the initial interaction unit 11 is also used to: obtain a preset initial decision control model and a preset opponent model of the opponent, initialize a preset group interaction platform, and obtain a first state of the agent and a second state of the opponent; the initial decision control model includes a local neural network; input the first state into the local neural network to obtain a first behavior and a first reward, input the second state into the opponent model to obtain a second behavior and a second reward, and store the first state, the second state, the first behavior, the second behavior, the first reward and the second reward into the initial decision control data group; input the first behavior and the second behavior into the group interaction platform, thereby correspondingly obtaining a third state of the agent and a fourth state of the opponent; input the third state into the local neural network to obtain a third behavior and a third reward, input the fourth state into the opponent model to obtain a fourth behavior and a fourth reward, and store the third state, the fourth state, the third behavior, the fourth behavior, the third reward and the fourth reward into the initial decision control data group.

[0060] The model training unit 12 is used to train the preset top-level learning model and the preset bottom-level learning model using the initial decision control data group, so as to obtain the final decision control model. In one embodiment, the model training unit 12 is also used to: divide the agent group into a corresponding number of groups according to the preset number of groups, obtain the average behavior value and reward and value of each group according to the initial decision control data group; obtain the learning target according to the average behavior value and reward and value of each group; train the top-level learning model according to the learning target and the initial decision control data group, so as to obtain the first top-level model and the corresponding first average neural network, and train the bottom-level learning model according to the first average neural network and the initial decision control data group, and record the number of trainings; determine whether the number of trainings reaches the preset number threshold; when the number of trainings reaches the preset number threshold, stop training and output the final decision control model.

[0061] The decision control unit 13 is used to perform decision control on the group interaction of the intelligent agents according to the final decision control model.

[0062] An embodiment of the present invention describes a decision control device for group interaction of intelligent agents. By constructing an initial decision control model including a top-level learning model and a bottom-level learning model, and performing top-level and bottom-level fusion training on the initial decision control model, a final decision control model is obtained and decision control is then performed. The decision control device improves the effectiveness of decision control during group interaction of intelligent agents. Specific embodiment three

[0064] In addition to the above methods and devices, the present invention also describes a decision-making control system for group interaction of intelligent agents. Figure 4 A structural diagram of an embodiment of a decision-making control system for group interaction of intelligent agents according to the present invention is shown.

[0065] As shown in the figure, the decision control system also includes a decision control module 1 and a data storage module 2, wherein the decision control module 1 is communicatively connected to the data storage module 2, wherein the decision control module 1 is used to perform decision control on the group interaction of the intelligent agent group according to the decision control method for the group interaction of the intelligent agent as described above, and the data storage module 3 is used to store all data.

[0066] In practical applications, the decision control module 1 and the data storage module 2 are respectively connected to the agent group for communication, so that the decision control module 1 can perform group interactive decision control on the agent group.

[0067] An embodiment of the present invention describes a decision control system for group interaction of intelligent agents. By constructing an initial decision control model including a top-level learning model and a bottom-level learning model, and performing top-level and bottom-level fusion training on the initial decision control model, a final decision control model is obtained and decision control is then performed. The decision control system improves the effectiveness of decision control during group interaction of intelligent agents.

[0068] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A decision-making control method for group interaction of intelligent agents, characterized in that: The decision control method comprises: Obtaining a preset initial decision control model, allowing the intelligent agent group to perform group interaction according to the initial decision control model, thereby obtaining an initial decision control data group; the initial decision control model includes a top-level learning model and a bottom-level learning model; wherein obtaining a preset initial decision control model, allowing the intelligent agent group to perform group interaction according to the initial decision control model, thereby obtaining an initial decision control data group, specifically includes: Acquire a preset initial decision control model and a preset opponent model of the opponent, initialize a preset group interaction platform, and acquire a first state of the intelligent agent and a second state of the opponent; the initial decision control model includes a local neural network; Input the first state into the local neural network to obtain a first behavior and a first reward, input the second state into the opponent model to obtain a second behavior and a second reward, and store the first state, the second state, the first behavior, the second behavior, the first reward and the second reward into an initial decision control data set; Inputting the first behavior and the second behavior into the group interaction platform, thereby correspondingly obtaining a third state of the agent and a fourth state of the opponent; Inputting the third state into the local neural network to obtain a third behavior and a third reward, inputting the fourth state into the opponent model to obtain a fourth behavior and a fourth reward, and storing the third state, the fourth state, the third behavior, the fourth behavior, the third reward and the fourth reward into an initial decision control data set; Using the initial decision control data group, training the top-level learning model and the bottom-level learning model, thereby obtaining a final decision control model; According to the final decision control model, decision control is performed on the group interaction of the intelligent agents.

2. The decision-making control method for group interaction of intelligent agents according to claim 1, characterized in that: Using the initial decision control data set, training the top-level learning model and the bottom-level learning model to obtain a final decision control model specifically includes: According to the preset number of groups, the agent group is divided into a corresponding number of groups, and the average behavior value and reward and value of each group are obtained according to the initial decision control data group; Get learning goals based on the average behavior values, rewards and values ​​of each group; According to the learning objective and the initial decision control data group, the top-level learning model is trained to obtain a first top-level model and a corresponding first average neural network, and according to the first average neural network and the initial decision control data group, the bottom-level learning model is trained, and the number of training times is recorded; Determining whether the number of training times reaches a preset number threshold; When the number of training times reaches a preset number threshold, the training is stopped and the final decision control model is output.

3. The decision-making control method for group interaction of intelligent agents according to claim 2, characterized in that: After determining whether the number of training times reaches a preset number threshold, the method further includes: When the number of training times does not reach the preset number threshold, the model training continues.

4. A decision-making control device for group interaction of intelligent agents, characterized in that: The decision control device includes an initial interaction unit, a model training unit and a decision control unit, wherein: The initial interaction unit is used to obtain a preset initial decision control model, so that the intelligent agent group performs group interaction according to the initial decision control model, thereby obtaining an initial decision control data group; the initial interaction unit is also used to: Acquire a preset initial decision control model and a preset opponent model of the opponent, initialize a preset group interaction platform, and acquire a first state of the intelligent agent and a second state of the opponent; the initial decision control model includes a local neural network; Input the first state into the local neural network to obtain a first behavior and a first reward, input the second state into the opponent model to obtain a second behavior and a second reward, and store the first state, the second state, the first behavior, the second behavior, the first reward and the second reward into an initial decision control data set; Inputting the first behavior and the second behavior into the group interaction platform, thereby correspondingly obtaining a third state of the agent and a fourth state of the opponent; Inputting the third state into the local neural network to obtain a third behavior and a third reward, inputting the fourth state into the opponent model to obtain a fourth behavior and a fourth reward, and storing the third state, the fourth state, the third behavior, the fourth behavior, the third reward and the fourth reward into an initial decision control data set; The model training unit is used to train a preset top-level learning model and a preset bottom-level learning model using the initial decision control data group, so as to obtain a final decision control model; The decision control unit is used to perform decision control on the group interaction of the intelligent agents according to the final decision control model.

5. The decision-making control device for group interaction of intelligent agents according to claim 4, characterized in that: The model training unit is also used for: According to the preset number of groups, the agent group is divided into a corresponding number of groups, and the average behavior value and reward and value of each group are obtained according to the initial decision control data group; Get learning goals based on the average behavior values, rewards and values ​​of each group; According to the learning objective and the initial decision control data group, the top-level learning model is trained to obtain a first top-level model and a corresponding first average neural network, and according to the first average neural network and the initial decision control data group, the bottom-level learning model is trained, and the number of training times is recorded; Determining whether the number of training times reaches a preset number threshold; When the number of training times reaches a preset number threshold, the training is stopped and the final decision control model is output.

6. A decision-making control system for group interaction of intelligent agents, characterized in that: The decision control system also includes a decision control module and a data storage module, and the decision control module is communicatively connected to the data storage module, wherein the decision control module is used to perform decision control on the group interaction of the intelligent agent group according to the decision control method for group interaction of intelligent agents as described in any one of claims 1 to 3, and the data storage module is used to store all data.

Images