Multi-agent cooperative scheduling method and system based on maximum entropy reinforcement learning

Abstract

The invention discloses a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning, and the method comprises the steps: generating an environment state vector according to the first environment data of a production line; generating an initial production task according to the environment state vector; performing task scheduling operation on the initial production task, and generating a path plan; storing the environment state vector, the initial production task and the path planning to obtain empirical data; according to the empirical data, training an initial strategy model through a maximum entropy reinforcement learning algorithm to obtain a target strategy model; and obtaining a multi-agent collaborative target scheduling strategy according to the target strategy model. The multi-agent cooperation efficiency can be improved, so that the production efficiency is improved, and the method can be widely applied to the technical field of agent scheduling.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agent scheduling technology, and in particular to a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning. Background Art

[0002] With the development of industry, intelligent manufacturing technology has been widely used in various production scenarios. As an important part of intelligent manufacturing, flexible production lines can realize rapid adjustment of production processes and diversified production of product types according to changes in production needs. Flexible production lines usually require the collaborative work of multiple robots and automated equipment to complete tasks such as material handling, assembly, processing and testing. However, traditional production line scheduling systems usually rely on fixed rules or preset paths for task allocation and planning. When faced with changes in production needs or uncertainties in the production line environment, such methods often appear to be unresponsive and difficult to effectively adapt to changes in the production site. In addition, traditional methods lack an in-depth understanding of the collaborative relationship between multiple robots, which can easily lead to resource conflicts and path interference between robots, thereby reducing production efficiency. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning, in order to solve at least one of the problems of the prior art. The present invention can improve the efficiency of multi-agent collaboration to improve production efficiency.

[0004] To achieve the above object, an embodiment of the present invention provides a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning, which includes the following steps:

[0005] generating an environmental state vector according to first environmental data of the production line;

[0006] generating an initial production task according to the environmental state vector;

[0007] Performing task scheduling operations on the initial production tasks and generating path planning;

[0008] The environmental state vector, the initial production task and the path planning are stored to obtain experience data;

[0009] According to the empirical data, the initial strategy model is trained by a maximum entropy reinforcement learning algorithm to obtain a target strategy model;

[0010] According to the target strategy model, a target scheduling strategy for multi-agent collaboration is obtained.

[0011] In some embodiments, a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning further includes the following steps:

[0012] Collecting second environmental data of the production line, adjusting the initial scheduling strategy of the target strategy model, and obtaining an intermediate scheduling strategy;

[0013] Through a hybrid cloud-edge collaborative computing architecture, the target strategy model is trained according to the intermediate scheduling strategy and the second environment data to generate the target scheduling strategy;

[0014] According to the target scheduling strategy, the target production task is executed to obtain feedback information.

[0015] In some embodiments, performing a task scheduling operation on the initial production task and generating a path plan includes the following steps:

[0016] Performing time series analysis according to the first environmental data and the initial production task to obtain predicted load fluctuation data;

[0017] Constructing a pre-allocation mechanism for the initial production tasks;

[0018] Through the multi-level graph neural network communication mechanism, the communication mechanism of multi-agents is divided into local communication mechanism and global broadcast mechanism;

[0019] Performing task scheduling on the initial production task according to the predicted load fluctuation data, the pre-allocation mechanism, the local communication mechanism, and the global broadcast mechanism;

[0020] The path planning is generated through the path interaction avoidance mechanism and the multi-level graph neural network communication mechanism.

[0021] In some embodiments, after the step of storing the environment state vector, the initial production task and the path planning to obtain experience data, the following steps are also included:

[0022] Construct feature weights of multi-agents;

[0023] Constructing a strategy parameter loss function through KL divergence and the feature weights;

[0024] The strategy parameter loss function is used to weight the process of learning the experience data.

[0025] In some embodiments, the training of the initial policy model by a maximum entropy reinforcement learning algorithm based on the empirical data to obtain a target policy model includes the following steps:

[0026] Construct dynamic multi-objective reward functions;

[0027] Constructing an objective function of a maximum entropy reinforcement learning algorithm according to the dynamic multi-objective reward function and the entropy coefficient;

[0028] Based on the empirical data, the initial strategy model is trained through the objective function to obtain the target strategy model.

[0029] In some embodiments, the objective function of the maximum entropy reinforcement learning algorithm is constructed according to the dynamic multi-objective reward function and the entropy coefficient, and the formula used includes:

[0030]

[0031] Where J(π) represents the objective function, which is the performance indicator of strategy π; t represents the time step, t=0,1,…,T; s t Represents the state; a t Represents action; r(s t , a t ) represents the state s t Take action a t The instant reward when , that is, the dynamic multi-objective reward function; α(a t ) represents the state s t The entropy coefficient is adjusted according to the dynamic complexity; Represents the strategy π in state s t Conditional entropy under ; Represents the state s under strategy π t and action a t The expected value of the joint distribution of

[0032] To achieve the above object, another aspect of an embodiment of the present invention proposes a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning, the system comprising:

[0033] The first module is used to generate an environmental state vector according to first environmental data of the production line;

[0034] The second module is used to generate an initial production task according to the environmental state vector;

[0035] The third module is used to perform task scheduling operations on the initial production tasks and generate path planning;

[0036] The fourth module is used to store the environmental state vector, the initial production task and the path planning to obtain experience data;

[0037] The fifth module is used to train the initial strategy model through the maximum entropy reinforcement learning algorithm according to the empirical data to obtain the target strategy model;

[0038] The sixth module is used to obtain the target scheduling strategy for multi-agent collaboration based on the target strategy model.

[0039] In some embodiments, a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning further includes:

[0040] The seventh module is used to collect the second environment data of the production line, adjust the initial scheduling strategy of the target strategy model, and obtain an intermediate scheduling strategy;

[0041] An eighth module is used to train the target strategy model according to the intermediate scheduling strategy and the second environment data through a hybrid cloud-edge collaborative computing architecture to generate the target scheduling strategy;

[0042] The ninth module is used to execute the target production task according to the target scheduling strategy and obtain feedback information.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning as described above.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning as described above.

[0045] To achieve the above purpose, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning.

[0046] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning, which generates an environment state vector according to the first environment data of the production line; generates an initial production task according to the environment state vector; performs task scheduling operations on the initial production task and generates a path planning; stores the environment state vector, the initial production task and the path planning to obtain experience data; based on the experience data, trains the initial strategy model through the maximum entropy reinforcement learning algorithm to obtain a target strategy model; based on the target strategy model, obtains a target scheduling strategy for multi-agent collaboration, which can improve the efficiency of multi-agent collaboration and thus improve production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a flow chart of a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning provided by an embodiment of the present invention;

[0049] Figure 2 It is one of the optional flow charts of the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning provided in an embodiment of the present invention;

[0050] Figure 3 is an operation flow chart of a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning provided by an embodiment of the present invention;

[0051] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0053] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0054] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0056] Flexible production lines usually require the collaborative work of multiple robots and automated equipment to complete tasks such as material handling, assembly, processing and testing. However, traditional production line scheduling systems mostly rely on preset rules and fixed task allocation strategies. When production demand or environment changes (such as increased production tasks, changes in equipment status, etc.), they cannot make adjustments quickly, which easily leads to a decrease in production efficiency. These systems show low flexibility and adaptability when facing frequently changing production tasks and diverse product types, and it is difficult to meet the requirements of modern flexible manufacturing. However, existing multi-robot scheduling methods often lack a deep understanding of the collaborative relationship between robots. Task conflicts and path interference are prone to occur during the scheduling process, causing robots to be blocked or waiting when performing tasks, increasing the time consumption in the production process. During the collaborative process, information transmission between robots is not smooth and there is a lack of effective communication mechanism, resulting in low resource utilization and low collaborative efficiency of the overall production line. In recent years, scheduling and optimization methods based on reinforcement learning have gradually attracted attention. Reinforcement learning can adaptively learn the best strategy through the interaction between agents and the environment, and shows good adaptability in dynamic and complex environments. However, traditional reinforcement learning methods often find it difficult to find a balance between exploration and utilization in multi-agent scenarios. Excessive exploration will lead to low learning efficiency, while over-exploitation may lead to local optimal solutions and make it impossible to find the global optimal strategy. Especially in a changing production environment, the agent needs to adjust its strategy frequently, but traditional reinforcement learning methods converge slowly and are difficult to quickly adapt to new tasks and environmental changes. In addition, existing scheduling systems are usually relatively static in task allocation strategies and lack the ability to learn online. When the environment changes during the production process (such as production line congestion, equipment failure, etc.), the system cannot learn and adjust its strategy autonomously and can only rely on manual intervention, which affects the continuity and stability of the production line.

[0057] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning, which may include but is not limited to steps S100 to S600:

[0058] Step S100, generating an environment state vector according to first environment data of the production line;

[0059] Step S200, generating an initial production task according to the environmental state vector;

[0060] Step S300, performing task scheduling operations on the initial production task and generating a path plan;

[0061] Step S400, storing the environmental state vector, the initial production task and the path planning to obtain experience data;

[0062] Step S500, training the initial strategy model through the maximum entropy reinforcement learning algorithm according to the empirical data to obtain the target strategy model;

[0063] Step S600, obtaining a target scheduling strategy for multi-agent collaboration according to the target strategy model.

[0064] In step S100 of some embodiments, the production line environmental data may include materials, locations, equipment status, etc., but is not limited thereto. By collecting the environmental data of the production line, a multi-dimensional environmental state can be formed, and then the environmental trend can be analyzed and predicted to help the intelligent agent make adjustments in advance to the changes on the production line. Exemplarily, through a variety of sensors such as laser radar, camera, force sensor, etc., the first environmental data of the production line is collected in real time to form a multi-dimensional environmental state, and the corresponding environmental state vector is output. The environmental state vector can be used to monitor the status of the production line in real time and used in subsequent decision-making processes to provide a basis for subsequent decisions. A new context recognition function can also be added to help the intelligent agent make adjustments in advance to the changes on the production line by analyzing and predicting environmental trends, such as material shortage warnings or equipment status change predictions, thereby improving the foresight of production decisions.

[0065] In step S200 of some embodiments, the optimal task execution path and action of the agent are selected according to the collected first environmental data and the generated environmental state vector, and the initial production task is generated. The optimal task execution path and action of the agent, the initial production task and other data selected initially lay a good foundation for the subsequent generation of relevant experience data and the training of the strategy model, thereby generating a target strategy model. The strategy learned according to the target strategy model can be further used in multi-agent decision-making, that is, the agent can select the optimal task execution path and action according to the environmental data, the environmental state vector, and the learned strategy.

[0066] In step S300 of some embodiments, the adaptive task load prediction function can be used to predict future task load fluctuations, thereby making advance task allocations. Optionally, in the actual production process, the production status is monitored in real time and dynamically adjusted according to actual conditions. If the actual load is higher than the predicted value, the system will immediately increase resource allocation; if the actual load is lower than the predicted value, the system will appropriately reduce resource allocation to avoid waste of resources. In addition, combined with the multi-level graph neural network (GNN) communication mechanism, local and global information sharing between intelligent agents is achieved. And through the path interaction avoidance mechanism, the intelligent agent can avoid through negotiation when there is a path conflict, which reduces waiting time and optimizes the overall operation efficiency of the production line.

[0067] In some embodiments, step S300 may include but is not limited to steps S310 to S350:

[0068] Step S310, performing time series analysis according to the first environmental data and the initial production task to obtain predicted load fluctuation data;

[0069] Step S320, constructing a pre-allocation mechanism for the initial production task;

[0070] Step S330, dividing the multi-agent communication mechanism into a local communication mechanism and a global broadcast mechanism through a multi-level graph neural network communication mechanism;

[0071] Step S340, scheduling the initial production task according to the predicted load fluctuation data, the pre-allocation mechanism, the local communication mechanism, and the global broadcast mechanism;

[0072] Step S350, generating the path planning through the path interaction avoidance mechanism and the multi-level graph neural network communication mechanism.

[0073] In step S310 of some embodiments, before task scheduling, time series analysis is performed based on the first environment data and the initial production task to predict load fluctuation data. This analysis helps to identify peak and valley periods in the production process, thereby allowing the system to pre-adjust resource allocation to cope with upcoming load changes.

[0074] In step S320 of some embodiments, constructing a pre-allocation mechanism for the initial production task is part of the task scheduling. The mechanism pre-allocates appropriate resources for the task according to the characteristics of the production task and the capacity of the production line. This pre-allocation helps to reduce waiting time and resource conflicts in the production process.

[0075] In step S330 of some embodiments, in a multi-agent system, the communication mechanism is key. Through a multi-level graph neural network, communication can be divided into a local communication mechanism and a global broadcast mechanism. Local communication helps to respond quickly to local changes, while global broadcast ensures that all agents can obtain key information. The use of a multi-level graph neural network communication mechanism helps to coordinate the operation of the entire production line.

[0076] In step S340 of some embodiments, the initial production task can be scheduled by combining the predicted load fluctuation data, pre-allocation mechanism, local communication and global broadcast. In the process of task scheduling, the production time can be minimized, resource waste can be reduced and product quality can be improved through optimization algorithms, and the efficient execution of production tasks can be ensured through genetic algorithms, reinforcement learning and the like.

[0077] In step S350 of some embodiments, after the task scheduling, the system generates a path plan to guide the movement of production resources (such as robots, transport vehicles, etc.) on the production line. Path planning needs to consider avoiding collisions, optimizing path length, and adapting to the dynamically changing production environment. The path interaction avoidance mechanism ensures that production resources can move safely and efficiently on the production line.

[0078] In step S400 of some embodiments, the environment state vector, the initial production task, and the path planning are stored in a shared experience pool. The experience data obtained by the agent during the task execution process, which may include but is not limited to state, action, reward, and next state, are also stored in the shared experience pool. The experience data generated in the shared experience pool lays a good foundation for subsequent multi-agent learning. Optionally, through the priority learning mechanism, it is ensured that the data of high-priority tasks are fully learned, and the incremental learning sharing mechanism can also enable the agent to selectively learn incremental data from the strategy changes of other agents, which significantly improves the learning efficiency and system response speed.

[0079] After step S400 in some embodiments, the following may also be included: constructing feature weights of multiple agents; constructing a strategy parameter loss function through KL divergence and the feature weights; wherein the strategy parameter loss function is used to weight the process of learning the experience data.

[0080] In some embodiments, the task execution experience of all agents, including data such as state, action, reward, next state, etc., is stored in a shared experience pool. The data in the experience pool is used to update the strategy during the reinforcement learning training process. For agents with different functions in the production line (such as handling robots, assembly robots, etc.), an adaptive strategy sharing mechanism between heterogeneous multi-agents is proposed. Exemplarily, by introducing the heterogeneity characteristics of agents in the shared experience pool, the learning process of different agents is personalized weighted, that is, the feature weights of multiple agents are added to the shared experience pool, and the agents can learn more diverse strategies, while adjusting the personalization of the strategy through KL divergence. This mechanism not only enables agents to accelerate learning by using the experience of other agents, but also enables each agent to optimize its own task through personalized adjustment, thereby improving the collaborative efficiency and adaptability of the entire production system. When sharing experience, agents can adjust the learning rate according to their own capabilities, so that agents with different functions can learn collaboratively while maintaining the optimal strategy for individual task execution. Among them, by introducing the heterogeneity characteristics of agents in the experience pool, the steps of personalized weighting of the learning process of different agents are shown in the following strategy parameter loss function:

[0081]

[0082] In the formula, Represents the strategy parameter loss function; Representative in the empirical data set Next state t The expected value of the distribution of It is used to store data samples such as historical states, actions, rewards, etc. of the agent during training. When updating the strategy, experience can be sampled from the experience data set for training; i represents the i-th agent, i = 1, 2, ..., N; Represents the strategy function of the ith agent; Represents the i-th agent in state s t The strategy under i Determine in state s t The probability distribution of the action selected when Q φ (s t , ·) represents the state based on s t The value function of the action is used to evaluate the expected return of the agent after taking a specific action in the current state, where the value function is represented by the parameter φ; V ψ (s t ) represents the state based on s t The value function is used to evaluate the t The expected return does not depend on the specific action, where the value function is represented by the parameter ψ; α represents the entropy coefficient; λ i Represents the personalization coefficient adjusted according to the agent's capabilities.

[0083] In some embodiments, step S500 may include but is not limited to steps S510 to S530:

[0084] Step S510, constructing a dynamic multi-objective reward function;

[0085] Step S520, constructing an objective function of a maximum entropy reinforcement learning algorithm according to the dynamic multi-objective reward function and the entropy coefficient;

[0086] Step S530: Based on the empirical data, the initial strategy model is trained by the objective function to obtain the target strategy model.

[0087] In step S510 of some embodiments, in order to balance multiple objectives (such as production efficiency, energy consumption, task priority, etc.) in task execution, a dynamic multi-objective reward function based on environmental perception is designed. The agent can dynamically adjust the weights of different optimization objectives through real-time collected environmental information, and the expression of the dynamic multi-objective reward function is:

[0088]

[0089] In the formula, r(s t , a t ) represents the state s t Take action a t The instant reward at the time is the dynamic multi-objective reward function; i represents the i-th agent, i = 1, 2, ..., N; ω i (s t ) represents the state s t The relevant dynamic weight coefficient is used to control the importance that the agent attaches to different rewards; r i (s t , a t ) represents the i-th agent in state s t Take action a t The instant reward at the time , that is, the dynamic multi-objective reward function of the ith agent.

[0090] By adjusting and state s t The dynamic weight coefficients related to production efficiency can be increased when the task on the production line is urgent, so that the agent can automatically adapt to the changes in the optimization target in different production scenarios. The dynamic weight coefficients may include but are not limited to weights related to production efficiency, energy consumption, task priority, etc. For example, when the task on the production line is urgent, the weight ω related to production efficiency can be increased. energy , in order to speed up the task execution; in the energy consumption control stage, the weights related to energy consumption are increased energy This dynamic adjustment mechanism enables the intelligent agent to automatically adapt to changes in optimization objectives in different production scenarios, significantly improving the flexibility and adaptability of production scheduling.

[0091] In step S520 of some embodiments, the agent uses the maximum entropy reinforcement learning (SoftActor-Critic, SAC) algorithm to optimize the strategy during the decision-making process. The goal of the agent is to achieve the purpose of balancing exploration and utilization by maximizing the combination of rewards and entropy values ​​of strategies. Based on the dynamic multi-objective reward function and the entropy coefficient, the objective function of the maximum entropy reinforcement learning algorithm can be constructed. The formula used includes:

[0092]

[0093] Where J(π) represents the objective function, which is the performance indicator of strategy π; t represents the time step, t = 0, 1, ..., T; s t Represents the state; a t Represents action; r(s t , a t ) represents the state s t Take action a t The instant reward at the time of , that is, the dynamic multi-objective reward function; α(st ) represents the state s t The entropy coefficient is adjusted according to the dynamic complexity, not a fixed value; Represents the strategy π in state s t Conditional entropy under ; Represents the state s under strategy π t and action a t The expected value of the joint distribution of .

[0094] In some embodiments, when the agent faces complex tasks or emergencies, α(s t ) will increase, encouraging higher exploration; when the task is relatively simple and the environment is stable, α(s t ) will decrease, allowing the agent to focus more on using the existing optimal strategy. This mechanism allows the agent to flexibly adjust the balance between exploration and utilization and adapt to changes in the production environment more quickly, which is especially suitable for frequently changing task allocation scenarios in flexible production lines. The agent can maintain a higher level of exploration in complex environments, and tends to use the learned optimal strategy in stable environments, thereby improving decision-making efficiency.

[0095] In step S530 of some embodiments, based on the experience data, the initial policy model is trained by the objective function, and the agent uses these experience data to update its policy model. In the maximum entropy reinforcement learning, the policy gradient method is usually used to update the policy parameters to maximize the objective function. During the training process, the agent will constantly try new actions to explore the environment and find better strategies. At the same time, the agent will use the knowledge it has learned to select actions that can bring higher returns. In this way, the agent can find a balance between exploration and utilization, so as to effectively learn in a complex environment. In addition, through the maximum entropy reinforcement learning algorithm, the concept of entropy is introduced in the policy optimization process, which helps the agent to maintain a certain randomness in the face of uncertainty, thereby avoiding falling into a local optimal solution. In addition, by dynamically adjusting the entropy coefficient, the agent can flexibly adjust its exploration strategy according to changes in the environment, which plays an important role in handling dynamic and complex tasks.

[0096] In step S600 of some embodiments, after the maximum entropy reinforcement learning algorithm training is completed, each agent will have an optimized strategy model. These strategy models integrate the actions that the agent should take in various states to maximize the cumulative reward and entropy value. Then, through the target strategy model, a target scheduling strategy for multi-agent collaboration can be generated.

[0097] In some embodiments, Figure 2 As shown, the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning can also include steps S700 to S900:

[0098] Step S700, collecting the second environment data of the production line, adjusting the initial scheduling strategy of the target strategy model, and obtaining an intermediate scheduling strategy;

[0099] Step S800: training the target policy model according to the intermediate scheduling policy and the second environment data through a hybrid cloud-edge collaborative computing architecture to generate the target scheduling policy;

[0100] Step S900: execute the target production task according to the target scheduling strategy and obtain feedback information.

[0101] In step S700 of some embodiments, the initial scheduling strategy of the target strategy model is optimized and adjusted online through the second environmental data of the production line collected in real time. Optionally, the online optimization and adjustment can be that the intelligent agent uses the real-time feedback data to quickly absorb new information and update the strategy through an incremental learning mechanism. The multi-level feedback mechanism enables the intelligent agent to make adjustments in the short term, and at the same time optimizes the behavior based on long-term performance, thereby achieving continuous strategy improvement and adaptability improvement, thereby obtaining an intermediate scheduling strategy after online optimization and adjustment.

[0102] In step S800 of some embodiments, the hybrid cloud-edge collaborative computing architecture is an architecture that combines cloud computing with edge computing, aiming to improve computing efficiency, reduce latency, enhance data processing capabilities, and optimize resource utilization. In this architecture, the cloud and edge terminals each assume different roles and tasks, and collaborate with each other to achieve optimal performance of the entire system. Optionally, complex policy training and global data analysis are performed on the target policy model in the cloud, while the real-time decision-making and environmental perception tasks of the intelligent agent are processed at the edge node to generate a target scheduling strategy. Through the hybrid cloud-edge collaborative computing architecture, while ensuring high computing density, low-latency real-time response can be achieved, allowing the intelligent agent to perform better in data-intensive tasks and rapid response tasks.

[0103] In step S900 of some embodiments, by analyzing the intermediate data generated during the execution of each link and task on the production line (such as scheduling strategies, production tasks, environmental data, feedback information, etc., but not limited to this), the system will automatically generate optimization suggestions (such as adjusting the delivery frequency of a certain type of material or optimizing a certain task path) and present them to the operator through the human-computer interaction interface. In addition, the operator can interact with the system through natural language instructions, adjust the task scheduling parameters in real time, and further improve the efficiency of human-computer collaboration.

[0104] like Figure 3As shown, an embodiment of the present invention further provides a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning, which can implement the above-mentioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning, and the system includes:

[0105] The first module (multi-agent decision-making module) is used to generate an environment state vector according to the first environment data of the production line;

[0106] The second module (multi-agent decision-making module) is used to generate an initial production task according to the environmental state vector;

[0107] The third module (task scheduling and path planning module) is used to perform task scheduling operations on the initial production tasks and generate path planning;

[0108] The fourth module (shared experience pool module) is used to store the environment state vector, the initial production task and the path planning to obtain experience data;

[0109] The fifth module (reinforcement learning training module) is used to train the initial strategy model through the maximum entropy reinforcement learning algorithm according to the empirical data to obtain the target strategy model;

[0110] The sixth module (target scheduling strategy generation module) is used to obtain the target scheduling strategy for multi-agent collaboration based on the target strategy model.

[0111] like Figure 3 As shown, a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning provided by an embodiment of the present invention may also include:

[0112] The seventh module (online optimization and feedback adjustment module) is used to collect the second environmental data of the production line, adjust the initial scheduling strategy of the target strategy model, and obtain an intermediate scheduling strategy;

[0113] An eighth module (hybrid cloud-edge collaborative computing architecture module) is used to train the target strategy model according to the intermediate scheduling strategy and the second environment data through the hybrid cloud-edge collaborative computing architecture to generate the target scheduling strategy;

[0114] The ninth module (intelligent production suggestion and human-computer interaction module) is used to execute the target production task according to the target scheduling strategy and obtain feedback information.

[0115] In some embodiments, a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning includes:

[0116] The multimodal perception module collects environmental data of the production line in real time through various sensors such as lidar, camera, force sensor, etc., including material location, equipment status, location of other robots and task progress, and performs fusion processing to generate state vectors to assist the agent in task selection and path planning. At the same time, it combines the situation recognition function of the situation recognition model to predict and analyze the changing trend of the production environment. Among them, the situation recognition model is combined with a machine learning algorithm, which can make predictions based on historical data and current environmental change patterns, and improve the agent's ability to respond to abnormal situations.

[0117] The multi-agent decision-making module is used to select the optimal task execution path and action for each robot agent based on the strategy generated by the reinforcement learning model and the environmental status obtained by the multimodal perception module. It can also dynamically adjust the balance of exploration and utilization of the agent in different production environments through an adaptive entropy weight adjustment mechanism, and absorb the incremental update data of other agents through an incremental learning sharing mechanism to improve decision-making efficiency.

[0118] The task scheduling and path planning module optimizes the task allocation of each agent based on the Graph Neural Networks (GNNs) model, combines the adaptive task load prediction and pre-allocation mechanism for dynamic task scheduling, and combines the multi-level Graph Neural Network (GNN) communication mechanism to achieve local and global information sharing between agents. In addition, the path interaction avoidance mechanism is applied in path planning to avoid path conflicts between multiple agents and improve overall production efficiency.

[0119] In some embodiments, the adaptive task load prediction and pre-allocation mechanism includes: introducing an adaptive task load prediction module in task scheduling. By performing time series analysis on historical production data and current task status, the agent can estimate task load fluctuations in the future and adjust the task allocation strategy in advance. In this way, the agent can prepare in advance during task peak periods and perform path optimization and maintenance operations during low load periods, thereby achieving dynamic balance of production line load.

[0120] In some embodiments, the multi-level graph neural network communication mechanism includes: using a multi-level GNN communication model to divide the communication of the agent into two parts: local communication and global broadcast. Local communication is used for the agent to share the status and task progress with other nearby agents, while global broadcast synchronizes key information (such as material shortages, equipment maintenance) to all agents when major changes occur in the production line. The multi-level communication mechanism improves the information perception ability of the agent in different environments and ensures the rapid response of the system in changing environments.

[0121] In some embodiments, the path interaction avoidance mechanism includes: introducing the path interaction avoidance mechanism in the path planning process, when the agent detects that a path conflict is about to occur with other agents, the agent can conduct dynamic avoidance negotiation through a shared communication protocol. By evaluating the priority of the path, the agent can independently decide whether to adjust its own speed or change the path direction, thereby reducing the waiting time caused by the path conflict.

[0122] The shared experience pool module is used to store the experience data of the agent during task execution, including state, action, reward and next state. It performs weighted processing on the data of high-priority tasks through the priority learning mechanism and supports the incremental learning sharing mechanism, which enables the agent to selectively learn the incremental strategy updates of other agents, significantly improving the learning efficiency and system response speed.

[0123] The reinforcement learning training module performs centralized training based on the maximum entropy reinforcement learning (Soft Actor-Critic, SAC) algorithm. It trains the optimal strategies of each agent under different tasks and environmental conditions through the cloud computing platform, and deploys the trained model to the edge computing node to achieve distributed real-time decision-making and execution, as well as low-latency decision-making of the agent. Optionally, a strategy adaptive evolution mechanism can also be deployed. The deployed agent can not only perform adaptive scheduling according to the actual production situation, but also gradually optimize its strategy parameters through the strategy evolution algorithm. After the production line has been running for a long time, the agent will evaluate its strategy and generate a new strategy through evolutionary operations such as strategy crossover and mutation based on the evaluation results. The strategy evolution mechanism enables the agent to gradually optimize its task execution strategy during operation and achieve self-optimization.

[0124] The online optimization and feedback adjustment module is used to adjust the agent strategy online according to the real-time collected data during the production process, accelerate the strategy update through the incremental learning sharing mechanism, and optimize the short-term and long-term behavior performance based on the multi-level feedback mechanism to improve the adaptability of the system. Furthermore, by introducing a multi-level feedback mechanism, the agent can not only make short-term adjustments based on the current state, but also make behavioral adjustments based on long-term task execution performance (such as adjusting the priority or action mode for a certain type of task). This multi-level feedback mechanism makes the agent's behavior more refined, so that the task execution efficiency can be gradually optimized in the long run.

[0125] In some embodiments, based on traditional online learning, an incremental learning sharing mechanism between agents is introduced, so that the policy changes updated by each agent during the production process will be synchronized through the shared experience pool, and other agents can selectively learn these incremental updates without having to completely retrain. This approach greatly shortens the policy adjustment time and enables the multi-agent system to learn the latest task execution methods more quickly.

[0126] In some embodiments, a multi-dimensional environmental perception mechanism can also be introduced into the online learning module to not only capture the real-time status data of the production line, but also to perform contextual recognition of potential trends in the environment. For example, when an increase in the material consumption rate on the production line is detected, the agent can issue an early warning and adjust the strategy to avoid shutdowns caused by material shortages.

[0127] The hybrid cloud-edge collaborative computing architecture module supports the system to perform complex strategy training and global data analysis in the cloud, while processing the real-time decision-making and environmental perception tasks of the intelligent agent at the edge node. Through the hybrid cloud-edge collaborative computing architecture, the system achieves low-latency real-time response while ensuring high computing density, making the intelligent agent perform better in data-intensive tasks and fast-response tasks.

[0128] The intelligent production suggestion and human-machine interaction module analyzes the bottleneck links and task execution data on the production line. The system automatically generates optimization suggestions (such as adjusting the delivery frequency of a certain type of material or optimizing a task path) and displays them to the operator through the human-machine interaction interface. It supports natural language interaction. Operators can interact with the system through voice or text commands, adjust scheduling parameters in real time, and further improve the efficiency of human-machine collaboration. This human-machine collaboration method improves the flexibility of the system at different production stages and enables operators to easily optimize and adjust the production process.

[0129] In order to verify the application effect of the multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning in a flexible production line, an experimental environment was built. The experimental environment simulates a multi-robot collaborative production workshop, which includes three types of agents: automated handling robots, assembly robots, and inspection robots, totaling 10 units, each type has different tasks and functions. The task of these agents is to collaboratively complete the assembly, inspection, and distribution of multiple products. The experimental workshop is equipped with a variety of sensors, including lidar, high-definition cameras, pressure sensors, and force sensors, which are used to collect environmental data on the production line in real time. Complex material flow routes and a variety of equipment (such as assembly lines, inspection tables, assembly workbenches, etc.) are set up in the experimental site to simulate the interaction process between different equipment in a real production line.

[0130] In order to achieve real-time communication and task coordination between agents, an edge computing node was built in the experimental environment to process real-time decision-making and data interaction of multiple agents. At the same time, the computing platform connected to the cloud is used to train and optimize complex strategy models. The core algorithm and task scheduling strategy of the system are supported by the SAC reinforcement learning model and graph neural network (GNN) to ensure that each agent can effectively perform task allocation and path optimization in the changing production tasks.

[0131] The goal of the experimental environment is to simulate the performance of the intelligent scheduling system on multiple dimensions such as task execution efficiency, energy consumption optimization, and task response speed of the flexible production line under different production loads, sudden task changes, equipment failures, etc., so as to verify the reliability and superiority of the system.

[0132] The following are the specific steps of the experiment:

[0133] Step 1: Environment initialization

[0134] 1.1 Start the experimental environment, including edge computing nodes and cloud computing platforms, and ensure that the network connection between the two is stable.

[0135] 1.2 Start various sensors in the production workshop, including lidar, high-definition cameras, force sensors, etc., to ensure that real-time data can be transmitted to edge nodes and used by intelligent agents for decision-making.

[0136] 1.3 Initialize the position and initial state of various intelligent agents (handling robots, assembly robots, and inspection robots), and connect the intelligent agents to edge computing nodes to achieve real-time data transmission and task scheduling.

[0137] Step 2: Reinforcement Learning Model Training and Deployment

[0138] 2.1 Use the cloud computing platform to conduct centralized training of the multi-agent strategy model. The training data includes historical production data, task execution data in the simulation environment, etc. The SAC (Soft Actor-Critic) algorithm is used in the training process, combined with a multi-objective reward function to optimize the task execution strategy of each agent.

[0139] 2.2 Download the trained reinforcement learning model to the edge computing node and deploy it to each intelligent agent, so that it can make real-time decisions based on the environmental status during the actual production process.

[0140] 2.3 Start the shared experience pool module. When the intelligent agent performs tasks in actual production, it will upload the experience data (such as status, action, reward, etc.) to the experience pool for subsequent online strategy optimization.

[0141] Step 3: Task Scheduling and Execution

[0142] 3.1 At the beginning of the experiment, a batch of production tasks are randomly generated, including material handling, product assembly, product testing, etc., and the tasks require each intelligent agent to complete them collaboratively.

[0143] 3.2 The multi-agent decision-making module assigns specific tasks to each agent based on real-time perception data and the strategy generated by the SAC model. The task scheduling and path planning module optimizes the path selection of the agent through the graph neural network (GNN) model to avoid path conflicts between agents.

[0144] 3.3 Each agent operates according to the assigned task. For example, the handling robot is responsible for transporting materials from the warehouse to the assembly area, the assembly robot completes the assembly of parts, and the inspection robot performs quality inspection after the product is assembled.

[0145] 3.4 The online optimization module continuously monitors the state changes and task completion status of each agent during task execution. When changes in the environment are detected (such as increased task requirements, equipment failure, etc.), the tasks and paths of the agents are adjusted in real time.

[0146] Step 4: Exception handling and strategy adjustment

[0147] 4.1 Artificially set up some abnormal situations in the experiment, such as adding sudden tasks, equipment downtime due to temporary failures, and partial paths being occupied, to test the system's response speed and adjustment ability under abnormal circumstances.

[0148] 4.2 The agent adjusts its strategy when the environment changes based on the online learning mechanism, quickly acquires new experience from exception handling through incremental learning, and uploads it to the shared experience pool.

[0149] 4.3 After receiving abnormal feedback, the system dynamically adjusts the task allocation plan and uses the adaptive entropy adjustment mechanism to enable the intelligent agent to explore and find adaptive strategies more quickly when facing new and complex environments.

[0150] Step 5: Data Collection and Analysis

[0151] 5.1 Record the task completion time, energy consumption, number of task allocation adjustments, number of path planning adjustments, production efficiency and other data of each intelligent agent during the experiment.

[0152] 5.2 Perform real-time data analysis on the edge computing node, monitor the task execution efficiency and path planning effect of each intelligent agent, and generate real-time data reports.

[0153] 5.3 After the experiment, all data will be synchronized to the cloud platform for further statistical analysis of various data, including the average task completion time, energy consumption changes, collaboration efficiency between agents, and response time under different abnormal situations.

[0154] Step 6: Experimental results evaluation

[0155] 6.1 Compare the scheduling efficiency and overall operation stability of the production line of the intelligent scheduling system under different loads (high load, low load) and different abnormal situations (sudden increase in tasks, equipment failure).

[0156] 6.2 Analyze the difference between the strategy optimization speed of the intelligent agent after using the incremental learning mechanism and traditional online learning, and verify the improvement of incremental learning on rapid adaptation ability.

[0157] 6.3 By adjusting the weights of the multi-objective reward function, the balance point between energy consumption and production efficiency is analyzed, and the system's adaptability to various optimization objectives under different production requirements is evaluated.

[0158] Step 7: Experimental summary and optimization suggestions

[0159] 7.1 Based on the experimental results, evaluate the actual performance of the multi-agent system in the production line and summarize the advantages and disadvantages of the system in terms of task allocation, path planning and online learning.

[0160] 7.2 In response to the problems found in the experiment (such as slow strategy adjustment in certain specific situations, communication delays between agents, etc.), further improvement measures and optimization directions are proposed.

[0161] 7.3 Based on the intelligent production suggestions in the experiment and the optimization plan generated by the human-computer interaction module, suggestions are made for the adjustment of the actual production line, including optimizing the production process, improving equipment utilization, etc.

[0162] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0164] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] refer to Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0166] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0167] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning in an embodiment of the present invention;

[0168] Input / output interface 1003, used to implement information input and output;

[0169] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0170] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0171] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0172] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning.

[0173] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning.

[0175] In summary, a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning in an embodiment of the present invention has the following advantages:

[0176] 1. The embodiment of the present invention improves the task allocation and path optimization capabilities of multi-robot systems in dynamic production environments through the high exploratory nature of maximum entropy reinforcement learning, multi-agent shared experience pool, hierarchical collaboration strategy, and adaptive entropy weight adjustment mechanism. The system combines multimodal perception and online learning to achieve efficient collaboration and real-time optimization between agents in the production process, providing an intelligent and adaptable scheduling solution for flexible production lines.

[0177] 2. The embodiment of the present invention designs a scheduling algorithm based on maximum entropy reinforcement learning, which enables the intelligent agent to quickly explore and find appropriate task allocation and path planning strategies in a highly uncertain and dynamically changing production environment, thereby improving the adaptability of the production line. Through the adaptive entropy weight adjustment mechanism, the intelligent agent automatically balances exploration and utilization at different production stages, improving the strategy optimization speed and decision-making flexibility.

[0178] 3. The embodiment of the present invention realizes effective information sharing and collaborative decision-making of multiple robots during task execution through the shared experience pool and hierarchical collaborative strategy of multiple agents, reduces resource conflicts and path interference between robots, and optimizes the order of task execution. A multi-agent communication mechanism based on graph neural network is designed so that the agent can fully consider the status and task situation of other robots around it when making decisions, thereby improving the overall collaborative efficiency of the multi-robot system.

[0179] 4. The embodiment of the present invention combines a multimodal perception module and an online learning mechanism to enable the intelligent agent to perceive changes in the production environment in real time and dynamically adjust the scheduling strategy based on new data to achieve self-learning and continuous optimization in the production process. An online feedback mechanism between intelligent agents is designed so that the system can quickly adjust its strategy when production anomalies occur (such as equipment failure, material shortage, etc.) to ensure efficient operation and stability of the production line.

[0180] 5. The embodiment of the present invention uses the high exploratory nature of maximum entropy reinforcement learning to enable the intelligent agent to quickly adapt to environmental changes in a variety of production scenarios, avoiding the problem of overall efficiency decline caused by falling into local optimality too early. The introduction of a priority training mechanism among multiple agents enables the intelligent agent to find the optimal strategy more quickly under high-priority tasks, thereby improving the system's response speed to key tasks.

[0181] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0182] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0184] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0185] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0186] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0187] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0188] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0189] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A multi-agent collaborative scheduling method based on maximum entropy reinforcement learning, characterized in that: The following steps are involved: generating an environmental state vector according to first environmental data of the production line; generating an initial production task according to the environmental state vector; Performing task scheduling operations on the initial production tasks and generating path planning; The environmental state vector, the initial production task and the path planning are stored to obtain experience data; According to the empirical data, the initial strategy model is trained by a maximum entropy reinforcement learning algorithm to obtain a target strategy model; According to the target strategy model, a target scheduling strategy for multi-agent collaboration is obtained.

2. According to the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning in claim 1, it is characterized in that: The following steps are also included: Collecting second environmental data of the production line, adjusting the initial scheduling strategy of the target strategy model, and obtaining an intermediate scheduling strategy; Through a hybrid cloud-edge collaborative computing architecture, the target strategy model is trained according to the intermediate scheduling strategy and the second environment data to generate the target scheduling strategy; According to the target scheduling strategy, the target production task is executed to obtain feedback information.

3. According to the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning in claim 1, it is characterized in that: The step of performing task scheduling operation on the initial production task and generating path planning comprises the following steps: Performing time series analysis according to the first environmental data and the initial production task to obtain predicted load fluctuation data; Constructing a pre-allocation mechanism for the initial production tasks; Through the multi-level graph neural network communication mechanism, the communication mechanism of multi-agents is divided into local communication mechanism and global broadcast mechanism; Performing task scheduling on the initial production task according to the predicted load fluctuation data, the pre-allocation mechanism, the local communication mechanism, and the global broadcast mechanism; The path planning is generated through the path interaction avoidance mechanism and the multi-level graph neural network communication mechanism.

4. According to the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning in claim 1, it is characterized in that: After the step of storing the environmental state vector, the initial production task and the path planning to obtain the experience data, the following steps are also included: Construct feature weights of multi-agents; Constructing a strategy parameter loss function through KL divergence and the feature weights; The strategy parameter loss function is used to weight the process of learning the experience data.

5. According to a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning according to claim 1, it is characterized in that: The method of training the initial strategy model by using the maximum entropy reinforcement learning algorithm based on the empirical data to obtain the target strategy model includes the following steps: Construct dynamic multi-objective reward functions; Constructing an objective function of a maximum entropy reinforcement learning algorithm according to the dynamic multi-objective reward function and the entropy coefficient; Based on the empirical data, the initial strategy model is trained through the objective function to obtain the target strategy model.

6. A multi-agent collaborative scheduling method based on maximum entropy reinforcement learning according to claim 5, characterized in that: The objective function of the maximum entropy reinforcement learning algorithm is constructed according to the dynamic multi-objective reward function and the entropy coefficient, and the formula used includes: Where J(π) represents the objective function, which is the performance indicator of strategy π; t represents the time step, t=0,1,…,T; s t Represents the state; a t Represents action; r(s t ,a t ) represents the state s t Take Action t The instant reward at the time of , that is, the dynamic multi-objective reward function; α(s t ) represents the state s t The entropy coefficient is adjusted according to the dynamic complexity; Represents the strategy π in state s t Conditional entropy under ; Represents the state s under strategy π t and action a t The expected value of the joint distribution of .

7. A multi-agent collaborative scheduling system based on maximum entropy reinforcement learning, characterized by ,,include: The first module is used to generate an environment state vector according to first environment data of the production line; The second module is used to generate an initial production task according to the environmental state vector; The third module is used to perform task scheduling operations on the initial production tasks and generate path planning; The fourth module is used to store the environmental state vector, the initial production task and the path planning to obtain experience data; The fifth module is used to train the initial strategy model through the maximum entropy reinforcement learning algorithm according to the empirical data to obtain the target strategy model; The sixth module is used to obtain the target scheduling strategy for multi-agent collaboration based on the target strategy model.

8. The multi-agent collaborative scheduling system based on maximum entropy reinforcement learning according to claim 7 is characterized in that: Also includes: The seventh module is used to collect the second environment data of the production line, adjust the initial scheduling strategy of the target strategy model, and obtain an intermediate scheduling strategy; An eighth module is used to train the target strategy model according to the intermediate scheduling strategy and the second environment data through a hybrid cloud-edge collaborative computing architecture to generate the target scheduling strategy; The ninth module is used to execute the target production task according to the target scheduling strategy and obtain feedback information.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images