Data-driven group intelligent management and control method based on reinforcement learning algorithm

Abstract

The invention discloses a reinforcement learning algorithm-based data-driven group intelligent management and control method. The method comprises the steps of establishing a big data lake; a deep reinforcement learning model is developed and optimized; a Deep Q-Network deep reinforcement learning algorithm is adopted to learn a complex decision strategy; a multi-agent framework is built, wherein an Autogen framework is adopted to achieve communication among multiple agents, and a Deep Q-Network reinforcement learning algorithm is adopted as a tool capable of being called by a main agent to plan actions of all the agents; the main agent is responsible for issuing an instruction, collecting environment feedback of each agent, and calling a deep reinforcement learning algorithm to adjust the action of each agent. According to the data-driven swarm intelligence management and control method based on the reinforcement learning algorithm, reinforcement learning and a swarm intelligence strategy are combined, so that the group can adapt to environmental changes more effectively in the strategy searching process, and falling into a local optimal solution is avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a data-driven group intelligent management and control method based on a reinforcement learning algorithm. Background Art

[0002] With the rapid development of information technology, data collection and storage have become easier. The purpose of data-driven methods is to use large amounts of data to improve existing algorithms and technologies, such as optimizing search strategies through historical log data and improving the coordination between swarm agents. Swarm intelligence methods have shown good results in multiple engineering fields by simulating complex collective behaviors, such as logistics distribution, energy management, smart cities, etc. Classic swarm intelligence methods, including ant colony algorithms and particle swarm algorithms, have shown certain performance in dealing with complex problems and have certain application potential in some fields, but still have the following disadvantages:

[0003] (1) The solution convergence speed is slow. In some complex problems, simple rules and feedback methods are used to direct the search process. When searching for the optimal solution, it takes a long time to converge. It is highly sensitive to parameters. If the search space is more complex during the search process, there is still a risk of falling into the local optimal solution.

[0004] (2) It is easy to fall into local optimality. The algorithm relies on the mutual influence and interaction between individuals in the group. If the group is concentrated in a certain area, the search plan will lose diversity, making it difficult to search for more solutions, and the optimal solution space may be missed.

[0005] (3) Difficulty in handling dynamic problems. Most of the above swarm intelligence algorithms are designed for static environment scenarios, that is, they assume that the environmental conditions of the problem remain unchanged throughout the entire problem-solving process. However, in actual scenarios, environmental conditions often change dynamically, and traditional swarm intelligence methods are difficult to adapt to such dynamic changes.

[0006] Therefore, there is an urgent need for a data-driven swarm intelligent management and control method based on reinforcement learning algorithm. Summary of the invention

[0007] The purpose of the present invention is to provide a data-driven swarm intelligent management and control method based on a reinforcement learning algorithm to solve the problems in the above-mentioned prior art, and to combine reinforcement learning with swarm intelligence strategies so that the swarm can more effectively adapt to environmental changes and avoid falling into local optimal solutions in the process of searching for strategies, thereby achieving more efficient and accurate swarm control.

[0008] The present invention provides a data-driven group intelligent management and control method based on a reinforcement learning algorithm, which includes:

[0009] Steps to build a big data lake: Use Kafka and Flink batch data stream collection components to access and collect large amounts of data;

[0010] Deep reinforcement learning model development optimization steps: Use Deep Q-Network deep reinforcement learning algorithm, combine Q-Learning and deep learning, and learn complex decision-making strategies;

[0011] Steps to build a multi-agent framework: Use the Autogen framework to achieve communication between multiple agents, use the Tongyi big model API interface as the message passing mechanism, and use the Deep Q-Network reinforcement learning algorithm as a tool that the main agent can call to globally plan the actions of all agents;

[0012] Swarm agent collaboration steps: The main agent is responsible for issuing instructions, collecting environmental feedback from each agent, and then calling the deep reinforcement learning algorithm to adjust the actions of each agent so that the group benefit can be maximized. After receiving the instructions from the main agent, a single agent will perform actions to collect environmental feedback and convey the environmental feedback to the agent.

[0013] In the above-mentioned data-driven group intelligent management and control method based on reinforcement learning algorithm, preferably, the big data lake construction step specifically includes:

[0014] We use the parallelized large-scale data analysis tool Presto to simultaneously analyze multiple different data sources in a unified manner, and support large-scale data storage and dynamic expansion of data capacity. For sensitive confidential data, we use desensitized data to encrypt the data, and provide a data interface through a Webservice service interface.

[0015] In the above-mentioned data-driven group intelligent management and control method based on reinforcement learning algorithm, preferably, the big data lake construction step specifically includes:

[0016] Kafka is used for data collection and transmission, and as a data transmission pipeline to transmit data to the data lake. The Flink stream processing framework is used to implement complex data logic conversion through Flink, and the pre-processed data is stored in the data lake.

[0017] Hadoop HDFS is used as the storage component of the data lake, and multi-source data backup is implemented based on DataNode nodes;

[0018] Large-scale data query is achieved based on the big data query engine Presto.

[0019] The data-driven group intelligent management and control method based on the reinforcement learning algorithm as described above, wherein, preferably, the deep reinforcement learning model development optimization step specifically includes: sequentially executing the definition of the task environment, designing the deep learning framework, collecting feedback results and updating the network,

[0020] When designing the deep learning framework, the PER sampling method is adopted, and a dual-channel network is designed to estimate actions and action values ​​respectively, and a dynamic attention mechanism layer is added, while combining the Dueling Network architecture and the maximum entropy exploration strategy.

[0021] As described above, the data-driven group intelligent management and control method based on the reinforcement learning algorithm, wherein, preferably, the dual-channel network in the deep reinforcement learning model development optimization step specifically includes:

[0022] The two channels in the dual-channel mechanism correspond to different functions. One set of channels corresponds to the selection of actions, and the other set of channels calculates the target value to ensure that all actions bring the maximum benefit.

[0023] In the process of implementing the dual-channel mechanism, the training of two networks is maintained, namely the online network and the target network, and the parameters learned by the online network are periodically copied to the target network;

[0024] When calculating the target value of the target network, we first use the online network of one channel to select an optimal action state, and then use the target network of another channel to evaluate the possible value of this action. The target network is only used to calculate the target value and does not participate in the update.

[0025] As described above, in the data-driven group intelligent control method based on the reinforcement learning algorithm, preferably, the PER sampling method specifically includes:

[0026] Different weight coefficients are assigned to different experience training samples, representing the priority between different samples. The larger the weight coefficient, the greater the possibility of being selected for training.

[0027] In the data-driven group intelligent control method based on the reinforcement learning algorithm as described above, preferably, the dynamic attention mechanism layer specifically includes:

[0028] An additional dynamic attention mechanism layer is added before calculating the target value to dynamically adjust the impact of the action according to changes in the environment, thereby reducing interference from the external environment, and aggregating the results of the dynamic attention mechanism layer to the target value estimation channel, after which the intelligent agent balances and selects the optimal action.

[0029] As described above, the data-driven group intelligent management and control method based on the reinforcement learning algorithm, wherein preferably, the Dueling Network architecture and the maximum entropy exploration strategy specifically include:

[0030] The target value prediction function is decomposed into two parts, one for evaluating the average value of any action in the preset environment, and the other for evaluating the advantage of the average value of the action in the environment;

[0031] The maximum entropy exploration strategy is to directly process the original loss function and promote the agent to continuously explore different actions in a dynamic environment by adding an entropy regularization term.

[0032] In the data-driven group intelligent management and control method based on the reinforcement learning algorithm as described above, preferably, the multi-agent framework building step specifically includes:

[0033] Autogen is used as the agent decision-making system. The UserAgent class in Autogen is used to create the main agent, which is used to issue instructions to the global agent and automatically generate various agent configurations and action plans. The remaining agents are created using AssistantAgent to receive instructions from the main agent and perform corresponding actions.

[0034] In the data-driven group intelligent management and control method based on the reinforcement learning algorithm as described above, preferably, the group intelligent agent collaboration step specifically includes:

[0035] Set up a main agent to call the tools in the tool library and the historical data in the knowledge base, analyze the historical data based on the deep reinforcement learning model tool, output the action plan of the global agent, and guide the actions of other agents in the form of instructions. After receiving the instructions, the other agents will execute the actions and collect feedback to pass it to the main agent through the message pipeline. The main agent will readjust the plan based on the global feedback until the global optimal effect is achieved.

[0036] During this process, the main agent sets up a broadcast mechanism to ensure that the other agents can receive instructions. The remaining agents need to set up a monitoring mechanism to listen to the main agent's broadcast information and establish a message passing pipeline to ensure that feedback is directly transmitted to the main agent.

[0037] The present invention provides a data-driven group intelligent management and control method based on a reinforcement learning algorithm, which combines an intelligent agent framework with reinforcement learning, allowing multiple intelligent agents to dynamically adjust strategies according to changes in environmental feedback; multiple intelligent agents can continuously learn under the coordination of a main intelligent agent, and communicate with each other to improve the performance of the entire group; at the same time, based on the technology of a big data lake to store historical data, the reinforcement learning framework Deep Q-Network can use historical data to accelerate the training process, and through the experience replay sampling technology, the intelligent agent can also continuously learn from past experience, thereby improving the quality of strategy generation; based on the reinforcement learning algorithm, the ability of group intelligent management and control is improved, the flexibility and anti-interference of group intelligent collaboration are improved, and the shortcomings of low time efficiency and slow solution speed in traditional group management and control methods are avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0039] Figure 1 A flowchart of an embodiment of a data-driven group intelligent management and control method based on a reinforcement learning algorithm provided by the present invention;

[0040] Figure 2 It is a flowchart of data processing for big data lake;

[0041] Figure 3 This is a diagram of the big data lake architecture;

[0042] Figure 4 This is the architecture diagram of the deep reinforcement learning model;

[0043] Figure 5 It is a multi-agent framework diagram;

[0044] Figure 6 This is the structure diagram of the dynamic attention mechanism layer. DETAILED DESCRIPTION

[0045] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The description of the exemplary embodiments is merely illustrative and is in no way intended to limit the present disclosure and its application or use. The present disclosure can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to make the present disclosure thorough and complete and to fully express the scope of the present disclosure to those skilled in the art. It should be noted that unless otherwise specifically stated, the relative arrangement of the components and steps, the composition of the materials, the numerical expressions and the numerical values ​​set forth in these embodiments should be interpreted as being merely exemplary and not as limiting.

[0046] The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different parts. The words "include" or "comprises" and similar words mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of including other elements. "Up", "down" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0047] In the present disclosure, when a specific component is described as being located between a first component and a second component, there may or may not be an intermediate component between the specific component and the first component or the second component. When a specific component is described as being connected to other components, the specific component may be directly connected to the other components without an intermediate component, or may not be directly connected to the other components but have an intermediate component.

[0048] All terms (including technical terms or scientific terms) used in the present disclosure have the same meanings as those understood by ordinary technicians in the field to which the present disclosure belongs, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries, such as general dictionaries, should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined herein.

[0049] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0050] Reinforcement learning, as a machine learning algorithm, allows agents to learn optimal strategies through interaction with the environment. It allows agents to learn optimal strategies through trial and error in a dynamically changing environment. Therefore, it is suitable for dealing with problems in dynamic environments and uncertain factors, and helps to avoid falling into local optimality. By combining deep learning, it can approximate the search function to achieve the purpose of improving time efficiency. By integrating reinforcement learning with swarm agent collaboration methods, the present invention can design a modular and flexible system architecture to improve the flexibility and scalability of swarm intelligence collaboration. The present invention integrates technologies such as reinforcement learning, multi-agent collaboration, and big data lakes, and is particularly suitable for reinforcement learning swarm intelligence management and control driven by big data.

[0051] like Figure 1 As shown, the data-driven group intelligent management and control method based on the reinforcement learning algorithm provided in this embodiment includes the following steps in actual execution:

[0052] Step S1, big data lake construction steps: use Kafka and Flink batch data stream collection components to achieve large-scale data access and collection.

[0053] Specifically, the parallel large-scale data analysis tool Presto is used to perform unified analysis on multiple different data sources, and supports large-scale data storage and dynamic expansion of data capacity. For sensitive confidential data, desensitized data is used to encrypt the data, thereby solving data security issues and providing a data interface through a Webservice service interface.

[0054] like Figure 2 and Figure 3 As shown, in one embodiment of the data-driven group intelligent management and control method based on the reinforcement learning algorithm of the present invention, the step S1 may specifically include:

[0055] Step S11: Use Kafka for data collection and transmission as a data transmission pipeline to deliver data to the data lake. Use the Flink stream processing framework to implement complex data logic conversions such as filtering and aggregation through Flink, and store the preprocessed data in the data lake.

[0056] The present invention selects the distributed real-time data stream reading platform Kafka (distributed processing real-time data stream platform) to collect raw data from the data source and stream it to the big data lake. Then the distributed stream processing framework Flink (distributed stream processing framework) is used to filter the data for different scenarios. In order to improve efficiency, the data can be divided into different partitions for parallel processing in this process.

[0057] Specifically, in the process of reading data, by setting up a message queue system through Kafka (a distributed real-time data stream processing platform), you can collect raw data from websites, devices, applications and other channels, and stream it to the big data lake. During the transmission process, the data is stored persistently to avoid data loss caused by failures. Kafka is suitable for high-throughput scenarios and can better handle real-time data streams.

[0058] In the process of real-time data processing, choose Flink (distributed stream processing framework) stream processing framework, read the data stream directly from the message queue, and perform preliminary filter processing on the data to meet different data processing requirements. Flink divides the data into multiple partitions for parallel processing and writes the data to the target system through sink.

[0059] Step S12: Use Hadoop HDFS as the storage component of the data lake, and perform multi-source backup of data based on DataNode nodes.

[0060] HDFS provides a multi-source massive data storage function, which can realize the storage and analysis of different types of data. Data storage uses the distributed file processing system HDFS (distributed file processing system) framework to create different DataNodes to back up data and ensure the safe storage of data.

[0061] Specifically, in the process of storing batch data, the processing of multi-source large data volumes is the core technology of the data lake. The present invention uses Hadoop HDFS (distributed file processing system) to store data. The main steps are as follows: i) Initialize the write request and perform the write operation through the create() method. ii) Establish a connection through the NameNode, perform a metadata check operation, determine the storage start address location of the written data, and feedback the location information to the client. iii) Establish a data transmission pipeline with the DataNode (data node) and start writing data. Pipelines are also established between DataNodes. Data can be transmitted to the following DataNodes in sequence through the pipeline. The last DataNode is responsible for writing its own storage data and providing confirmation information forward. iv) When the data transmission is completed and the replica data storage is completed, the DataNode will confirm with the client and then use the close() method to end the write operation.

[0062] Step S13: Implement large-scale query of data based on the big data query engine Presto.

[0063] Data query uses the big data query engine Presto to automatically parse query statements, generate query steps and execute them, and finally return the query results in a visual interface Web UI.

[0064] Specifically, in the process of batch querying data, Presto (big data query engine) is used to query data. It is necessary to modify the Presto configuration file, and then start the coordinator to receive the client's query and parse the SQL of the data query. The coordinator will parse the query to generate an execution plan and execute it. After the query is executed, the query results can be viewed in PrestoCLl, and the execution results can be limited in the Web UI.

[0065] Step S2, deep reinforcement learning model development optimization step: Use the Deep Q-Network deep reinforcement learning algorithm, combine Q-Learning and deep learning, and learn complex decision-making strategies.

[0066] The present invention selects the deep reinforcement learning algorithm Deep Q-Network. Different from the traditional group intelligent management method, the Deep Q-Network algorithm uses deep learning to approximate the state selection function to guide the action execution of the intelligent agent, which can not only learn more complex strategies, but also effectively improve time efficiency. In one embodiment of the present invention, step S2 includes: defining the task environment, designing the deep learning framework, collecting feedback results and updating the network in sequence. Figure 4 A deep reinforcement learning model architecture diagram is shown.

[0067] The core of step S2 is to build a deep neural network to approximate the function of the state value, which is used to guide the possible action value of each intelligent agent. In order to improve the ability of the reinforcement learning network, when designing the deep learning framework, the present invention adopts the PER sampling method, and designs a dual-channel network to estimate the action and action value respectively, so as to avoid the predicted value of the action being evaluated by a group of networks, thereby improving the quality of the Deep Q-Network strategy. A layer of dynamic attention mechanism is added, and the Dueling Network architecture and the maximum entropy exploration strategy are combined. Figure 6 The structure diagram of the dynamic attention mechanism layer is shown. Through the dynamic attention mechanism layer, the incentive weight corresponding to each action can be dynamically adjusted to expand or reduce the impact of the action, so as to better estimate the target value of the network. Combining the Dueling Network architecture and the maximum entropy exploration strategy, it can better adapt to the changes in the dynamic environment.

[0068] Specifically, the dual-channel network in the deep reinforcement learning model development optimization step specifically includes:

[0069] The two channels in the dual-channel mechanism correspond to different functions. One set of channels corresponds to the selection of actions, and the other set of channels calculates the target value to ensure that all actions bring the maximum benefit.

[0070] In the process of implementing the dual-channel mechanism, the training of two networks is maintained, namely the online network and the target network, and the parameters learned by the online network are periodically copied to the target network;

[0071] When calculating the target value of the target network, we first use the online network of one channel to select the best action state, and then use the target network of the other channel to evaluate the value that this action may produce. The target network is only used to calculate the target value and does not participate in the update. The dual-channel mechanism is conducive to improving the evaluation of actions and target values, and can improve the stability of DeepQ-Network learning.

[0072] Furthermore, the PER sampling method specifically includes:

[0073] Different weight coefficients are assigned to different empirical training samples, representing the priority between different samples. The larger the weight coefficient, the greater the possibility of being selected for training. The present invention adopts PER method for resampling, and different samples may be assigned different coefficients, which is conducive to better use of the empirical value of historical data, and can optimize the data sampling method, thereby effectively improving the ability of Deep Q-Network algorithm. In addition, a dual-channel mechanism is designed to predict the target value and action respectively, which can improve the evaluation ability of action and target value.

[0074] Specifically, unlike the traditional sample experience playback method, the PER sampling method sets different priorities for different samples. TD is used to calculate the error, which represents the difference between the current predicted target value and the actual target value. The specific calculation method is as follows: (1)

[0075] in, represents the empirical error of a sample, represents the corresponding reward coefficient, represents the discount factor, and They represent the target value prediction results of the dual-channel network. Based on formula (1), the probability calculation method of sampling is as follows: (2)

[0076] When the agent interacts with the environment to obtain feedback, it calculates the empirical TD error of the current sample, calculates the current priority according to the above formulas (1) and (2), and saves it in the experience pool. After each round of training, the new TD is used to update the priority of the corresponding sample in the current experience pool.

[0077] Then, Deep Q-Network was developed based on the dual-channel mode. At this time, the target value is calculated by the collaborative calculation of two channels:

[0078] (3)

[0079] Formula (3) shows how the two channels evaluate the target value. Indicates the action currently performed by the agent, which is determined by the channel The target value is selected by the network training of another channel. The training process between the two channels does not affect each other.

[0080] Furthermore, the dynamic attention mechanism layer specifically includes:

[0081] Before calculating the target value, an additional dynamic attention mechanism layer is added to assign different weights to actions according to different environmental states, so as to dynamically adjust the impact of the action according to changes in the environment, thereby reducing interference from the external environment, and aggregating the results of the dynamic attention mechanism layer to the target value estimation channel, after which the agent balances and selects the optimal action. At this time, the dynamic changes of the environment are taken into account, the impact of invalid actions can be eliminated, the anti-interference ability of the model can be enhanced, the prediction of the target value will be more realistic, the stability of the algorithm is improved, and the stability of the model in a dynamic environment is improved.

[0082] Specifically, after completing formula (3), a dynamic attention mechanism layer is added to the target value estimation channel, and the attention calculation function is as follows:

[0083] (s,a,α,ɡ) (4)

[0084] Among them, s represents the current environment state, a represents the action corresponding to the current state, α represents the network parameter corresponding to the dynamic attention mechanism layer, and ɡ represents the attention weight coefficient corresponding to the dynamic attention mechanism layer.

[0085] The final target value calculation function consists of two parts:

[0086] (5)

[0087] At this time, the target value is additionally added with the result of the dynamic attention mechanism layer. The purpose is to adjust the quality of the action through different weights, which can effectively correct the impact of invalid actions, especially increase the anti-interference ability of the model in a dynamic environment.

[0088] Furthermore, the Dueling Network architecture and maximum entropy exploration strategy specifically include:

[0089] The target value prediction function is decomposed into two parts. One part is used to evaluate the average value of any action in the preset environment, and the other part is used to evaluate the advantage of the average value of the action in the environment. This can overcome the limitations of the dynamic and randomly changing environment.

[0090] The maximum entropy exploration strategy is to directly process the original loss function and promote the agent to continuously explore different actions in a dynamic environment by adding an entropy regularization term. This can promote the agent to continuously explore different actions in a dynamic environment and avoid falling into a local optimal solution.

[0091] In order to prevent the model from falling into a local optimal solution, the present invention breaks down the target value into two prediction parts, one predicting the average value of actions in a certain state, and the other predicting the advantage of the average value of these actions. At the same time, an entropy regularization term is added to the loss function, and the parameters are set to adjust the influence of the regularization term to prevent the agent from falling into a local optimal solution during the search process.

[0092] Specifically, in order to improve the quality of target value prediction, the target value is decomposed into two parts of prediction, one of which is the average value under the environment state. , and the other part is to estimate the advantage of the average value of this set of actions , then the final objective function is: (6)

[0093] in: (7)

[0094] In order to increase the random exploration ability of the model, add the maximum entropy regularization term to the loss function:

[0095] (8)

[0096] in, represents the parameter that controls the influence of the entropy regularization term, represents the added entropy regularization term.

[0097] In addition, it is necessary to define an environment according to the specific task. Its functional modules mainly include the following parts. 1) Reset the environment and return to the initial state of the environment. 2) Execute the action and return the current state, action execution result, whether to stop and other information. 3) Design the neural network. The main purpose of this step is to approximate the state estimation function, that is, the value of taking an action under a certain state. The neural network consists of three parts: input layer, hidden layer, and output layer. The input layer determines the dimension according to the state of different environments. The hidden layer consists of multiple fully connected layers. The number of nodes in the output layer is the same as the number of environmental agents.

[0098] Step S3, multi-agent framework construction steps: The Autogen framework is used to realize the communication between multiple agents. The message passing mechanism adopts the method of the Tongyi large model API interface. The Deep Q-Network reinforcement learning algorithm is used as a tool that can be called by the main agent to globally plan the actions of all agents.

[0099] The core of step S3 is to select a suitable agent framework, which should have the characteristics of convenient development, high stability and high flexibility. The group should set up a main agent that can issue instructions to all agents, so there must be an efficient communication mechanism between multiple agents. The present invention selects the Autogen framework to realize communication between multiple agents. The message passing mechanism adopts the method of the Tongyi large model API interface. The Deep Q-Network reinforcement learning algorithm is used as a tool that can be called by the main agent to globally plan the actions of all agents. Specifically, Autogen is used as the agent decision-making system, and the UserAgent class in Autogen is used to create the main agent, which is used to issue instructions to the global agent and automatically generate a variety of agent configurations and action plans; the remaining agents are created using AssiantAgent to receive the main agent instructions and perform corresponding actions. Figure 5 A multi-agent framework diagram is shown.

[0100] The present invention designs multiple different intelligent agents to collaboratively handle complex problems. Based on Autogen, a multi-agent framework is built, and different intelligent agents complete the same task through communication and collaboration. The framework sets up a main intelligent agent to issue instructions to the whole world, and guide the actions of other intelligent agents based on the deep reinforcement learning model in the tool library. Other intelligent agents need to perform actions and pass the received environmental feedback to the main intelligent agent. Asynchronous interaction is used to transmit messages between intelligent agents, and a unified message format and transmission mechanism are defined.

[0101] Specifically, the present invention builds a multi-agent collaboration framework, where each agent can autonomously perceive the environment and has its own state and behavior. The multi-agent collaboration framework is built by Autogen, and the action guidance of the agent is guided by the reinforcement learning model, which mainly includes the following steps:

[0102] Design an agent collaboration framework, develop a main agent to issue instructions to the global agent, measure the benefits of the overall solution, and make adjustments in real time. A single agent needs to perform corresponding actions according to the instructions issued by the main agent, obtain interactive feedback from the environment, and pass the feedback information to the main agent in real time;

[0103] Model the environment, use a dictionary to record different elements in the environment and their current corresponding states, including the resources available in the current environment, the state of the agent, time, etc. Then define the rules for the agent to update the state, affect the state of the environment by sensing the environment and performing actions, define the state transfer function to describe how the agent's behavior changes according to the change of the environment state, the agent can sense the change of the environment and feedback to the main agent;

[0104] Agent interaction: The interaction between agents adopts asynchronous interaction mode. The message format is defined as several parts including sender, receiver, message content, message type, sending time, sending method, etc. The process of agent interaction includes three steps: agent request, agent response, and agent confirmation. The collaboration strategy adopts the leader-follower mode. A main agent is selected from all agents, and the actions of other agents are specified by the main agent.

[0105] Step S4, group agent collaboration step: The main agent is responsible for issuing instructions, collecting environmental feedback from each agent, and then calling the deep reinforcement learning algorithm to adjust the actions of each agent so that the group benefit can be maximized. After receiving the instructions from the main agent, a single agent will perform actions to collect environmental feedback and convey the environmental feedback to the agent.

[0106] Specifically, a main agent is set up to call the tools in the tool library and the historical data in the knowledge base, analyze the historical data based on the deep reinforcement learning model tool, output the action plan of the global agent, and guide the actions of other agents in the form of instructions. After receiving the instructions, the other agents execute the actions, collect feedback and pass it to the main agent through the message pipeline. The main agent readjusts the plan based on the global feedback until the global optimal effect is achieved.

[0107] During this process, the main agent sets up a broadcast mechanism to ensure that the other agents can receive instructions. The remaining agents need to set up a monitoring mechanism to listen to the main agent's broadcast information and establish a message passing pipeline to ensure that feedback is directly transmitted to the main agent.

[0108] In step S4, the main agent issues instructions to the global agent, which are sent to other agents via a broadcast mechanism. Other agents listen to the instructions in real time, perform corresponding actions, and feedback the current status of the main agent. The main agent collects global feedback and calls the Deep Q-Network algorithm to adjust the corresponding actions of each agent in real time to ensure the maximum group benefit.

[0109] The present invention realizes group intelligent management and control based on the intelligent agent collaboration system of the reinforcement learning framework Deep Q-Network. The defined intelligent agent collaboration system can automatically perform a series of tasks. The basic structure of the intelligent agent is as follows: (6)

[0110] Among them, Action represents the action performed by the agent. represents the environmental factors that affect the agent, Represents the defined environment, Tools represents the tool library of the agent, and the present invention adds the trained Deep Q-Network framework as a tool to the tool library of the main agent. The update principle of the agent can be as follows:

[0111] (7)

[0112] in, Represents the learning rate, which is used to control the influence of new environmental factors on the current agent's actions. After receiving the main agent's instructions, the agent performs the action You will get a reward , then update the agent Target value. In order to conveniently implement the multi-agent collaboration framework, the present invention quickly builds an agent group management and control system based on the Autogen framework, uses the basic class Agent to define different agent classes, defines specific behaviors and corresponding roles for each class, and defines the agent's message sending and message receiving through send_message and receive_message. The main agent assigns tasks to other agents based on a polling mechanism, and uses the chat group Groupchat to promote communication and collaboration between agents. In actual use, the trained Deep Q-Network model is deployed to the production environment, and the developed collaborative group management and control model is encapsulated into a container using Docker.

[0113] The data-driven group intelligent management and control method based on the reinforcement learning algorithm provided by the embodiment of the present invention combines the intelligent agent framework and reinforcement learning, allowing multiple intelligent agents to dynamically adjust strategies according to changes in environmental feedback; multiple intelligent agents can continuously learn under the coordination of the main intelligent agent, and communicate with each other to improve the performance of the entire group; at the same time, based on the technology of big data lake to store historical data, the reinforcement learning framework Deep Q-Network can use historical data to accelerate the training process, and through the experience replay sampling technology, the intelligent agent can also continuously learn from past experience, thereby improving the quality of strategy generation; based on the reinforcement learning algorithm, the ability of group intelligent management and control is improved, the flexibility and anti-interference of group intelligent collaboration are improved, and the shortcomings of low time efficiency and slow solution speed in traditional group management and control methods are avoided.

[0114] So far, various embodiments of the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0115] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A data-driven group intelligent management and control method based on reinforcement learning algorithm, characterized in that: include: Steps to build a big data lake: Use Kafka and Flink batch data stream collection components to access and collect large amounts of data; Deep reinforcement learning model development optimization steps: Use Deep Q-Network deep reinforcement learning algorithm, combine Q-Learning and deep learning, and learn complex decision-making strategies; Steps to build a multi-agent framework: Use the Autogen framework to achieve communication between multiple agents, use the Tongyi big model API interface as the message passing mechanism, and use the Deep Q-Network reinforcement learning algorithm as a tool that the main agent can call to globally plan the actions of all agents; Swarm agent collaboration steps: The main agent is responsible for issuing instructions, collecting environmental feedback from each agent, and then calling the deep reinforcement learning algorithm to adjust the actions of each agent so that the group benefit can be maximized. After receiving the instructions from the main agent, a single agent will perform actions to collect environmental feedback and convey the environmental feedback to the agent.

2. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 1 is characterized in that: The steps of building the big data lake specifically include: We use the parallelized large-scale data analysis tool Presto to simultaneously analyze multiple different data sources in a unified manner, and support large-scale data storage and dynamic expansion of data capacity. For sensitive confidential data, we use desensitized data to encrypt the data, and provide a data interface through a Webservice service interface.

3. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 2 is characterized in that: The steps for building a big data lake specifically include: Kafka is used for data collection and transmission, and as a data transmission pipeline to transmit data to the data lake. The Flink stream processing framework is used to implement complex data logic conversion through Flink, and the pre-processed data is stored in the data lake. Hadoop HDFS is used as the storage component of the data lake, and multi-source data backup is implemented based on DataNode nodes; Large-scale data query is achieved based on the big data query engine Presto.

4. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 1 is characterized in that: The deep reinforcement learning model development optimization steps specifically include: sequentially executing the definition of the task environment, designing the deep learning framework, collecting feedback results and updating the network, When designing the deep learning framework, the PER sampling method is adopted, and a dual-channel network is designed to estimate actions and action values ​​respectively, and a dynamic attention mechanism layer is added, while combining the Dueling Network architecture and the maximum entropy exploration strategy.

5. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 4 is characterized in that: The dual-channel network in the deep reinforcement learning model development optimization step specifically includes: The two channels in the dual-channel mechanism correspond to different functions. One set of channels corresponds to the selection of actions, and the other set of channels calculates the target value to ensure that all actions bring the maximum benefit. In the process of implementing the dual-channel mechanism, the training of two networks is maintained, namely the online network and the target network, and the parameters learned by the online network are periodically copied to the target network; When calculating the target value of the target network, we first use the online network of one channel to select an optimal action state, and then use the target network of another channel to evaluate the possible value of this action. The target network is only used to calculate the target value and does not participate in the update.

6. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 4 is characterized in that: The PER sampling method specifically includes: Different weight coefficients are assigned to different experience training samples, representing the priority between different samples. The larger the weight coefficient, the greater the possibility of being selected for training.

7. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 4 is characterized in that: The dynamic attention mechanism layer specifically includes: An additional dynamic attention mechanism layer is added before calculating the target value to dynamically adjust the impact of the action according to changes in the environment, thereby reducing interference from the external environment, and aggregating the results of the dynamic attention mechanism layer to the target value estimation channel, after which the intelligent agent balances and selects the optimal action.

8. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 4 is characterized in that: The Dueling Network architecture and maximum entropy exploration strategy specifically include: The target value prediction function is decomposed into two parts, one for evaluating the average value of any action in the preset environment, and the other for evaluating the advantage of the average value of the action in the environment; The maximum entropy exploration strategy is to directly process the original loss function and promote the agent to continuously explore different actions in a dynamic environment by adding an entropy regularization term.

9. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 1 is characterized in that: The steps of building the multi-agent framework specifically include: Autogen is used as the agent decision-making system. The UserAgent class in Autogen is used to create the main agent, which is used to issue instructions to the global agent and automatically generate various agent configurations and action plans. The remaining agents are created using AssistantAgent to receive instructions from the main agent and perform corresponding actions.

10. The data-driven group intelligent management and control method based on reinforcement learning algorithm according to claim 1 is characterized in that: The group agent collaboration steps specifically include: Set up a main agent to call the tools in the tool library and the historical data in the knowledge base, analyze the historical data based on the deep reinforcement learning model tool, output the action plan of the global agent, and guide the actions of other agents in the form of instructions. After receiving the instructions, the other agents will execute the actions and collect feedback to pass it to the main agent through the message pipeline. The main agent will readjust the plan based on the global feedback until the global optimal effect is achieved. During this process, the main agent sets up a broadcast mechanism to ensure that the other agents can receive instructions. The remaining agents need to set up a monitoring mechanism to listen to the main agent's broadcast information and establish a message passing pipeline to ensure that feedback is directly transmitted to the main agent.

Images