Multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning

Abstract

The present invention discloses a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning. The method comprises: generating an environmental state vector based on first environmental data of a production line; generating an initial production task based on the environmental state vector; performing a task scheduling operation on the initial production task and generating a path plan; storing the environmental state vector, the initial production task, and the path plan to obtain empirical data; training an initial policy model using a maximum entropy reinforcement learning algorithm based on the empirical data to obtain a target policy model; and obtaining a target scheduling policy for multi-agent collaboration based on the target policy model. The present invention can improve the efficiency of multi-agent collaboration, thereby improving production efficiency, and can be widely applied to the field of agent scheduling technology.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agent scheduling technology, and in particular to a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning. Background Art

[0002] With the development of industry, intelligent manufacturing technology has been widely used in various production scenarios. As an important component of intelligent manufacturing, flexible production lines can achieve rapid adjustment of production processes and diversified production of product types according to changes in production needs. Flexible production lines usually require the collaborative work of multiple robots and automated equipment to complete tasks such as material handling, assembly, processing and testing. However, traditional production line scheduling systems usually rely on fixed rules or preset paths for task allocation and planning. When faced with changes in production needs or uncertainties in the production line environment, such methods often appear to be unresponsive and difficult to effectively adapt to changes in the production site. In addition, traditional methods lack an in-depth understanding of the collaborative relationship between multiple robots, which can easily lead to resource conflicts and path interference between robots, thereby reducing production efficiency. Summary of the Invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning, in order to solve at least one of the existing technical problems. The present invention can improve the efficiency of multi-agent collaboration to improve production efficiency.

[0004] To achieve the above objectives, an embodiment of the present invention provides a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning, which includes the following steps:

[0005] generating an environmental state vector according to first environmental data of the production line;

[0006] generating an initial production task according to the environmental state vector;

[0007] Performing task scheduling operations on the initial production tasks and generating path planning;

[0008] The environmental state vector, the initial production task, and the path planning are stored to obtain experience data;

[0009] Based on the empirical data, the initial strategy model is trained using a maximum entropy reinforcement learning algorithm to obtain a target strategy model;

[0010] According to the target strategy model, a target scheduling strategy for multi-agent collaboration is obtained.

[0011] In some embodiments, a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning further includes the following steps:

[0012] collecting second environmental data of the production line, adjusting the initial scheduling strategy of the target strategy model, and obtaining an intermediate scheduling strategy;

[0013] Through a hybrid cloud-edge collaborative computing architecture, the target policy model is trained according to the intermediate scheduling policy and the second environment data to generate the target scheduling policy;

[0014] According to the target scheduling strategy, the target production task is executed to obtain feedback information.

[0015] In some embodiments, performing a task scheduling operation on the initial production task and generating a path plan includes the following steps:

[0016] Performing a time series analysis based on the first environmental data and the initial production task to obtain predicted load fluctuation data;

[0017] Establishing a pre-allocation mechanism for the initial production tasks;

[0018] Through the multi-level graph neural network communication mechanism, the multi-agent communication mechanism is divided into local communication mechanism and global broadcast mechanism;

[0019] Performing task scheduling on the initial production task according to the predicted load fluctuation data, the pre-allocation mechanism, the local communication mechanism, and the global broadcast mechanism;

[0020] The path planning is generated through the path interaction avoidance mechanism and the multi-level graph neural network communication mechanism.

[0021] In some embodiments, after the step of storing the environmental state vector, the initial production task, and the path planning to obtain experience data, the following steps are further included:

[0022] Construct feature weights of multiple agents;

[0023] Constructing a strategy parameter loss function through KL divergence and the feature weights;

[0024] The strategy parameter loss function is used to weight the process of learning the experience data.

[0025] In some embodiments, training the initial policy model using a maximum entropy reinforcement learning algorithm based on the empirical data to obtain a target policy model includes the following steps:

[0026] Construct dynamic multi-objective reward function;

[0027] Constructing an objective function of a maximum entropy reinforcement learning algorithm based on the dynamic multi-objective reward function and the entropy coefficient;

[0028] Based on the empirical data, the initial strategy model is trained using the objective function to obtain the target strategy model.

[0029] In some embodiments, the objective function of the maximum entropy reinforcement learning algorithm is constructed based on the dynamic multi-objective reward function and the entropy coefficient, and the formula used includes:

[0030]

[0031] Where J(π) represents the objective function, which is the performance indicator of the strategy π; t represents the time step, t=0,1,…,T; s t Represents the state; a t Represents action; r(s t , a t ) represents the state s t Take action a t The instant reward when , that is, the dynamic multi-objective reward function; α(a t ) represents the state s t The entropy coefficient is adjusted according to the dynamic complexity; Represents the strategy π in state s t Conditional entropy under ; Represents the state s under strategy π t and action a t The expected value of the joint distribution of

[0032] To achieve the above objectives, another aspect of an embodiment of the present invention provides a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning, the system comprising:

[0033] A first module is used to generate an environmental state vector according to first environmental data of the production line;

[0034] A second module is used to generate an initial production task according to the environmental state vector;

[0035] The third module is used to perform task scheduling operations on the initial production tasks and generate path planning;

[0036] A fourth module is configured to store the environmental state vector, the initial production task, and the path planning to obtain empirical data;

[0037] A fifth module is used to train the initial policy model using a maximum entropy reinforcement learning algorithm based on the empirical data to obtain a target policy model;

[0038] The sixth module is used to obtain the target scheduling strategy for multi-agent collaboration based on the target strategy model.

[0039] In some embodiments, a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning further includes:

[0040] A seventh module is configured to collect second environmental data of the production line, adjust the initial scheduling strategy of the target strategy model, and obtain an intermediate scheduling strategy;

[0041] An eighth module is configured to train the target policy model according to the intermediate scheduling policy and the second environment data through a hybrid cloud-edge collaborative computing architecture to generate the target scheduling policy;

[0042] The ninth module is used to execute the target production task according to the target scheduling strategy and obtain feedback information.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning as described above.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning as described above.

[0045] To achieve the above objectives, another aspect of an embodiment of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning.

[0046] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning, which generates an environmental state vector according to the first environmental data of the production line; generates an initial production task according to the environmental state vector; performs task scheduling operations on the initial production task and generates path planning; stores the environmental state vector, the initial production task and the path planning to obtain empirical data; based on the empirical data, trains the initial strategy model through the maximum entropy reinforcement learning algorithm to obtain a target strategy model; based on the target strategy model, obtains a target scheduling strategy for multi-agent collaboration, which can improve the efficiency of multi-agent collaboration and thus improve production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 This is a flow chart of a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning provided by an embodiment of the present invention;

[0049] Figure 2 This is one of the optional flow charts of the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning provided by an embodiment of the present invention;

[0050] Figure 3 This is an operation flow chart of a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning provided by an embodiment of the present invention;

[0051] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0053] It should be noted that although the functional modules are divided in the system schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned figures may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination".

[0054] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include at least one, two or more, multiple, two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0056] Flexible production lines often require the coordinated operation of multiple robots and automated equipment to complete tasks such as material handling, assembly, processing, and inspection. However, traditional production line scheduling systems often rely on pre-set rules and fixed task allocation strategies. These systems are unable to quickly adapt to changes in production demand or the environment (such as an increase in production tasks or changes in equipment status), which can easily lead to decreased production efficiency. These systems exhibit low flexibility and adaptability when faced with frequently changing production tasks and diverse product offerings, making them unable to meet the requirements of modern flexible manufacturing. Existing multi-robot scheduling methods often lack a deep understanding of the collaborative relationships between robots. Task conflicts and path interference are prone to occur during the scheduling process, causing robots to be blocked or waiting while executing tasks, increasing production time. During the collaborative process, information flow between robots is poor, and there is a lack of effective communication mechanisms, resulting in low resource utilization and low overall production line coordination efficiency. In recent years, scheduling and optimization methods based on reinforcement learning have gained increasing attention. Reinforcement learning can adaptively learn optimal policies through the interaction between intelligent agents and the environment, demonstrating good adaptability in dynamic and complex environments. However, traditional reinforcement learning methods often struggle to strike a balance between exploration and exploitation in multi-agent scenarios. Excessive exploration leads to inefficient learning, while overexploitation can lead to local optima, preventing the global optimal strategy from being found. Especially in volatile production environments, agents need to frequently adjust their strategies, but traditional reinforcement learning methods converge slowly and struggle to adapt quickly to new tasks and environmental changes. Furthermore, existing scheduling systems often have static task allocation strategies and lack the ability to learn online. When the production environment changes (such as line congestion or equipment failure), the system is unable to autonomously learn and adjust its strategy, relying instead on manual intervention, which impacts the continuity and stability of the production line.

[0057] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning, which may include but is not limited to steps S100 to S600:

[0058] Step S100, generating an environmental state vector according to first environmental data of the production line;

[0059] Step S200, generating an initial production task according to the environmental state vector;

[0060] Step S300, performing task scheduling operations on the initial production task and generating a path plan;

[0061] Step S400, storing the environmental state vector, the initial production task, and the path planning to obtain experience data;

[0062] Step S500: training the initial policy model using a maximum entropy reinforcement learning algorithm based on the empirical data to obtain a target policy model;

[0063] Step S600: Obtain a target scheduling strategy for multi-agent collaboration based on the target strategy model.

[0064] In step S100 of some embodiments, the production line environmental data may include materials, locations, equipment status, etc., but is not limited thereto. By collecting the environmental data of the production line, a multi-dimensional environmental state can be formed, and then the environmental trends can be analyzed and predicted to help the intelligent agent make adjustments in advance to the changes on the production line. Exemplarily, the first environmental data of the production line is collected in real time through various sensors such as lidar, cameras, force sensors, etc. to form a multi-dimensional environmental state and output a corresponding environmental state vector. The environmental state vector can be used to monitor the status of the production line in real time and to be used in subsequent decision-making processes to provide a basis for subsequent decisions. A new context recognition function can also be added to help the intelligent agent make adjustments in advance to the changes on the production line by analyzing and predicting environmental trends, such as material shortage warnings or equipment status change predictions, thereby improving the foresight of production decisions.

[0065] In step S200 of some embodiments, based on the collected first environmental data and the generated environmental state vector, an optimal task execution path and action for the agent are selected to generate an initial production task. This initially selected optimal task execution path and action for the agent, along with the initial production task, lays the foundation for the subsequent generation of relevant empirical data and training of the policy model, thereby generating a target policy model. The policy learned from the target policy model can be further utilized in multi-agent decision-making, i.e., the agent can select the optimal task execution path and action based on the environmental data, environmental state vector, and the learned policy.

[0066] In step S300 of some embodiments, the adaptive task load prediction function can be used to predict future task load fluctuations, thereby making advance task allocations. Optionally, in the actual production process, the production status is monitored in real time and dynamically adjusted according to actual conditions. If the actual load is higher than the predicted value, the system will immediately increase resource allocation; if the actual load is lower than the predicted value, the system will appropriately reduce resource allocation to avoid wasting resources. In addition, combined with a multi-level graph neural network (GNN) communication mechanism, local and global information sharing between intelligent agents is achieved. And through the path interaction avoidance mechanism, the intelligent agents can avoid through negotiation when there is a path conflict, reducing waiting time and optimizing the overall operating efficiency of the production line.

[0067] In some embodiments, step S300 may include but is not limited to steps S310 to S350:

[0068] Step S310: performing time series analysis based on the first environmental data and the initial production task to obtain predicted load fluctuation data;

[0069] Step S320, constructing a pre-allocation mechanism for the initial production task;

[0070] Step S330, dividing the multi-agent communication mechanism into a local communication mechanism and a global broadcast mechanism through a multi-level graph neural network communication mechanism;

[0071] Step S340: scheduling the initial production task according to the predicted load fluctuation data, the pre-allocation mechanism, the local communication mechanism, and the global broadcast mechanism;

[0072] Step S350: Generate the path plan through the path interaction avoidance mechanism and the multi-level graph neural network communication mechanism.

[0073] In some embodiments, in step S310, before task scheduling, a time series analysis is performed based on the first environment data and the initial production task to predict load fluctuation data. This analysis helps identify peak and trough periods in the production process, allowing the system to pre-adjust resource allocation to cope with upcoming load changes.

[0074] In some embodiments, in step S320, establishing a pre-allocation mechanism for initial production tasks is part of task scheduling. This mechanism pre-allocates appropriate resources for the tasks based on the characteristics of the production tasks and the capacity of the production line. This pre-allocation helps reduce waiting time and resource conflicts during the production process.

[0075] In step S330 of some embodiments, communication mechanisms are crucial in multi-agent systems. Using a multi-layer graph neural network, communication can be divided into local communication mechanisms and global broadcast mechanisms. Local communication facilitates rapid response to local changes, while global broadcast ensures that all agents receive critical information. Leveraging multi-layer graph neural network communication mechanisms helps coordinate operations across the entire production line.

[0076] In some embodiments, step S340 can be used to schedule initial production tasks by combining predicted load fluctuation data, pre-allocation mechanisms, local communication, and global broadcasting. During task scheduling, optimization algorithms can be used to minimize production time, reduce resource waste, and improve product quality. Genetic algorithms, reinforcement learning, and other methods can also be used to ensure efficient execution of production tasks.

[0077] In some embodiments, in step S350, after task scheduling, the system generates a path plan to guide the movement of production resources (such as robots and transport vehicles) along the production line. Path planning must consider collision avoidance, path length optimization, and adaptability to the dynamically changing production environment. A path interaction avoidance mechanism ensures that production resources can move safely and efficiently along the production line.

[0078] In step S400 of some embodiments, the environment state vector, the initial production task, and the path planning are stored in a shared experience pool. The experience data obtained by the agent during task execution, which may include but is not limited to state, action, reward, and next state, is also stored in the shared experience pool. The experience data generated in the shared experience pool lays a good foundation for subsequent multi-agent learning. Optionally, a priority learning mechanism is used to ensure that the data of high-priority tasks is fully learned. An incremental learning sharing mechanism can also be used to enable the agent to selectively learn incremental data from the strategy changes of other agents, significantly improving learning efficiency and system response speed.

[0079] After step S400 in some embodiments, the following may also be included: constructing feature weights of multiple agents; constructing a strategy parameter loss function through KL divergence and the feature weights; wherein the strategy parameter loss function is used to weight the process of learning the experience data.

[0080] In some embodiments, the task execution experience of all agents, including data such as state, action, reward, and next state, is stored in a shared experience pool. The data in the experience pool is used to update policies during reinforcement learning training. For agents with different functions in a production line (such as handling robots, assembly robots, etc.), an adaptive policy sharing mechanism between heterogeneous multi-agents is proposed. For example, by introducing agent heterogeneity characteristics into the shared experience pool, the learning process of different agents is personalized and weighted. That is, by adding the feature weights of multiple agents to the shared experience pool, the agents can learn more diverse strategies and adjust the personalization of the strategies through KL divergence. This mechanism not only enables agents to accelerate learning by leveraging the experience of other agents, but also enables each agent to optimize its own task through personalized adjustments, thereby improving the collaborative efficiency and adaptability of the entire production system. When sharing experience, agents can adjust their learning rates according to their own capabilities, allowing agents with different functions to learn collaboratively while maintaining the optimal strategy for individual task execution. The steps of personalized weighting of the learning process of different agents by introducing agent heterogeneity characteristics into the experience pool are shown in the following policy parameter loss function:

[0081]

[0082] Where, Represents the strategy parameter loss function; Representative in the empirical dataset Next state s t The expected value of the distribution of , where the empirical data set It is used to store data samples such as the historical state, action, and reward of the agent during training. When updating the strategy, it can sample experience from this experience dataset for training. i represents the i-th agent, i = 1, 2, ..., N; Represents the strategy function of the i-th agent; Represents the i-th agent in state s t The strategy under the parameter θ i Determine in state s t The probability distribution of the action selected when Q φ (s t , ·) represents the state based on s t The value function of the action is used to evaluate the expected reward of the agent after taking a specific action in the current state, where the value function is represented by the parameter φ; V ψ (s t ) represents the state based on s t The value function is used to evaluate the t The expected return does not depend on the specific action, where the value function is represented by the parameter ψ; α represents the entropy coefficient; λ i Represents the personalization coefficient adjusted according to the agent's capabilities.

[0083] In some embodiments, step S500 may include but is not limited to steps S510 to S530:

[0084] Step S510, constructing a dynamic multi-objective reward function;

[0085] Step S520, constructing an objective function of a maximum entropy reinforcement learning algorithm based on the dynamic multi-objective reward function and the entropy coefficient;

[0086] Step S530 : Based on the empirical data, the initial strategy model is trained by the objective function to obtain the target strategy model.

[0087] In step S510 of some embodiments, a dynamic multi-objective reward function based on environmental perception is designed to balance multiple objectives (such as production efficiency, energy consumption, task priority, etc.) during task execution. The agent can dynamically adjust the weights of different optimization objectives based on real-time environmental information collected. The expression of the dynamic multi-objective reward function is:

[0088]

[0089] In the formula, r(s t , a t ) represents the state s t Take action a t The instant reward at the time of , that is, the dynamic multi-objective reward function; i represents the i-th agent, i = 1, 2, ..., N; ω i (s t ) represents the state s t The relevant dynamic weight coefficient is used to control the importance that the agent attaches to different rewards; r i (s t , a t ) represents the i-th agent in state s t Take action a t The instant reward at time , that is, the dynamic multi-objective reward function of the i-th agent.

[0090] By adjusting the state t The dynamic weight coefficients related to production efficiency enable the agent to automatically adapt to changes in optimization objectives in different production scenarios. The dynamic weight coefficients may include but are not limited to weights related to production efficiency, energy consumption, task priority, etc. For example, when the task on the production line is urgent, the weight ω related to production efficiency can be increased. energy , in order to speed up the task execution; in the energy consumption control stage, the weight related to energy consumption is increased ω energy This dynamic adjustment mechanism enables the intelligent agent to automatically adapt to changes in optimization objectives in different production scenarios, significantly improving the flexibility and adaptability of production scheduling.

[0091] In step S520 of some embodiments, the agent uses a maximum entropy reinforcement learning (Soft Actor-Critic, SAC) algorithm to optimize its strategy during the decision-making process. The agent's goal is to achieve a balance between exploration and exploitation by maximizing the combination of reward and strategy entropy. Based on the dynamic multi-objective reward function and the entropy coefficient, the objective function of the maximum entropy reinforcement learning algorithm can be constructed. The formula used includes:

[0092]

[0093] Where J(π) represents the objective function, which is the performance indicator of the strategy π; t represents the time step, t = 0, 1, ..., T; s t Represents the state; a t Represents action; r(s t , a t ) represents the state s t Take action a t The instant reward when , that is, the dynamic multi-objective reward function; α(st ) represents the state s t The entropy coefficient is adjusted according to the dynamic complexity, not a fixed value; Represents the strategy π in state s t Conditional entropy under ; Represents the state s under strategy π t and action a t The expected value of the joint distribution of .

[0094] In some embodiments, when the agent faces complex tasks or emergencies, α(s t ) will increase, encouraging higher exploration; when the task is relatively simple and the environment is stable, α(s t ) will decrease, allowing the agent to focus more on leveraging its existing optimal strategy. This mechanism allows the agent to flexibly adjust the balance between exploration and exploitation, more quickly adapting to changes in the production environment. This is particularly suitable for scenarios with frequently changing task allocations in flexible production lines. The agent can maintain a higher level of exploration in complex environments, while tending to utilize the learned optimal strategy in stable environments, thereby improving decision-making efficiency.

[0095] In step S530 of some embodiments, the initial policy model is trained based on empirical data using an objective function. The agent uses this empirical data to update its policy model. In maximum entropy reinforcement learning, a policy gradient method is typically used to update policy parameters to maximize the objective function. During training, the agent continuously tries new actions to explore the environment and find better strategies. At the same time, the agent uses the knowledge it has learned to select actions that can bring higher rewards. In this way, the agent can find a balance between exploration and exploitation, thereby effectively learning in complex environments. In addition, the maximum entropy reinforcement learning algorithm introduces the concept of entropy into the policy optimization process, which helps the agent maintain a certain degree of randomness when facing uncertainty, thereby avoiding falling into local optimal solutions. In addition, by dynamically adjusting the entropy coefficient, the agent can flexibly adjust its exploration strategy according to changes in the environment, which plays an important role in handling dynamic and complex tasks.

[0096] In step S600 of some embodiments, after the maximum entropy reinforcement learning algorithm is trained, each agent will have an optimized policy model. These policy models integrate the actions that the agent should take in various states to maximize the cumulative reward and entropy value. Using the target policy model, a target scheduling strategy for multi-agent collaboration can be generated.

[0097] In some embodiments, as Figure 2 As shown, the multi-agent collaborative scheduling method based on maximum entropy reinforcement learning can also include steps S700 to S900:

[0098] Step S700: collecting second environmental data of the production line, adjusting the initial scheduling strategy of the target strategy model, and obtaining an intermediate scheduling strategy;

[0099] Step S800: Training the target policy model according to the intermediate scheduling policy and the second environment data through a hybrid cloud-edge collaborative computing architecture to generate the target scheduling policy;

[0100] Step S900: Execute the target production task according to the target scheduling strategy and obtain feedback information.

[0101] In step S700 of some embodiments, the initial scheduling strategy of the target strategy model is optimized and adjusted online using the second environmental data of the production line collected in real time. Optionally, the online optimization and adjustment can be performed by the intelligent agent using real-time feedback data to rapidly absorb new information and update the strategy through an incremental learning mechanism. The multi-level feedback mechanism enables the intelligent agent to make adjustments in the short term while optimizing its behavior based on long-term performance, thereby achieving continuous strategy improvement and increased adaptability, thereby obtaining an intermediate scheduling strategy after online optimization and adjustment.

[0102] In step S800 of some embodiments, the hybrid cloud-edge collaborative computing architecture is an architecture that combines cloud computing with edge computing, aiming to improve computing efficiency, reduce latency, enhance data processing capabilities and optimize resource utilization. In this architecture, the cloud and edge each assume different roles and tasks, and collaborate with each other to achieve optimal performance of the entire system. Optionally, complex policy training and global data analysis are performed on the target policy model in the cloud, while the real-time decision-making and environmental perception tasks of the intelligent agent are processed at the edge node to generate a target scheduling strategy. Through the hybrid cloud-edge collaborative computing architecture, while ensuring high computing density, low-latency real-time response can be achieved, making the intelligent agent perform better in data-intensive tasks and fast response tasks.

[0103] In some embodiments, in step S900, by analyzing the intermediate data generated during the execution of various links and tasks on the production line (such as scheduling strategies, production tasks, environmental data, feedback information, etc., but not limited to these), the system automatically generates optimization suggestions (such as adjusting the delivery frequency of a certain type of material or optimizing a task path) and presents them to the operator through a human-computer interaction interface. In addition, the operator can interact with the system through natural language commands and adjust task scheduling parameters in real time, further improving the efficiency of human-computer collaboration.

[0104] like Figure 3As shown, an embodiment of the present invention further provides a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning, which can implement the above-mentioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning. The system includes:

[0105] The first module (multi-agent decision module) is used to generate an environment state vector based on the first environment data of the production line;

[0106] The second module (multi-agent decision module) is used to generate an initial production task according to the environmental state vector;

[0107] The third module (task scheduling and path planning module) is used to perform task scheduling operations on the initial production tasks and generate path planning;

[0108] The fourth module (shared experience pool module) is used to store the environmental state vector, the initial production task and the path planning to obtain experience data;

[0109] The fifth module (reinforcement learning training module) is used to train the initial policy model through the maximum entropy reinforcement learning algorithm based on the empirical data to obtain the target policy model;

[0110] The sixth module (target scheduling strategy generation module) is used to obtain the target scheduling strategy for multi-agent collaboration based on the target strategy model.

[0111] like Figure 3 As shown, the multi-agent collaborative scheduling system based on maximum entropy reinforcement learning provided by the embodiment of the present invention may also include:

[0112] The seventh module (online optimization and feedback adjustment module) is used to collect the second environmental data of the production line, adjust the initial scheduling strategy of the target strategy model, and obtain an intermediate scheduling strategy;

[0113] An eighth module (hybrid cloud-edge collaborative computing architecture module) is configured to train the target policy model based on the intermediate scheduling policy and the second environment data through the hybrid cloud-edge collaborative computing architecture to generate the target scheduling policy;

[0114] The ninth module (intelligent production suggestion and human-computer interaction module) is used to execute the target production task according to the target scheduling strategy and obtain feedback information.

[0115] In some embodiments, a multi-agent collaborative scheduling system based on maximum entropy reinforcement learning includes:

[0116] The multimodal perception module uses multiple sensors, such as lidar, cameras, and force sensors, to collect real-time environmental data from the production line, including material location, equipment status, the positions of other robots, and task progress. It then integrates this data into a state vector, which is used to assist the agent in task selection and path planning. This module, combined with the context recognition capabilities of the context recognition model, predicts and analyzes changing trends in the production environment. The context recognition model, combined with machine learning algorithms, can make predictions based on historical data and current environmental change patterns, improving the agent's ability to respond to abnormal situations.

[0117] The multi-agent decision-making module is used to select the optimal task execution path and action for each robotic agent based on the strategy generated by the reinforcement learning model and the environmental status obtained by the multimodal perception module. It can also dynamically adjust the balance between exploration and utilization of the agent in different production environments through an adaptive entropy weight adjustment mechanism. It can also absorb the incremental update data of other agents through an incremental learning sharing mechanism to improve decision-making efficiency.

[0118] The task scheduling and path planning module optimizes task allocation for each agent based on a graph neural network (GNN) model. It combines adaptive task load prediction and pre-allocation mechanisms for dynamic task scheduling. A multi-layered graph neural network (GNN) communication mechanism enables local and global information sharing between agents. Interactive path avoidance mechanisms are also applied in path planning to prevent conflicts between multiple agents and improve overall production efficiency.

[0119] In some embodiments, the adaptive task load prediction and pre-allocation mechanism includes the introduction of an adaptive task load prediction module into task scheduling. By performing time series analysis on historical production data and current task status, the intelligent agent can predict task load fluctuations over a period of time and adjust task allocation strategies in advance. This allows the intelligent agent to prepare for peak task times and perform route optimization and maintenance operations during periods of low load, thereby achieving dynamic balance of production line load.

[0120] In some embodiments, a multi-level graph neural network communication mechanism includes: using a multi-level GNN communication model to divide agent communication into two parts: local communication and global broadcast. Local communication is used for agents to share status and task progress with other nearby agents, while global broadcast synchronizes key information (such as material shortages and equipment maintenance) to all agents when major changes occur on the production line. This multi-level communication mechanism improves the information perception ability of agents in different environments, ensuring the system's rapid response in changing environments.

[0121] In some embodiments, the interactive path avoidance mechanism includes: Incorporating this mechanism into the path planning process, when an agent detects an impending path conflict with another agent, it can dynamically negotiate avoidance via a shared communication protocol. By evaluating the priority of each path, the agent can autonomously decide whether to adjust its speed or change direction, thereby reducing waiting time caused by path conflicts.

[0122] The shared experience pool module is used to store the experience data of the agent during task execution, including state, action, reward and next state. It uses the priority learning mechanism to weight the data of high-priority tasks and supports the incremental learning sharing mechanism, enabling the agent to selectively learn the incremental strategy updates of other agents, significantly improving learning efficiency and system response speed.

[0123] The reinforcement learning training module performs centralized training based on the maximum entropy reinforcement learning (Soft Actor-Critic, SAC) algorithm. It trains the optimal strategy of each agent under different tasks and environmental conditions through a cloud computing platform, and deploys the trained model to the edge computing node to achieve distributed real-time decision-making and execution, as well as low-latency decision-making of the agent. Optionally, a policy adaptive evolution mechanism can also be deployed. The deployed agent can not only perform adaptive scheduling according to actual production conditions, but also gradually optimize its policy parameters through the policy evolution algorithm. After the production line has been running for a long time, the agent will evaluate its strategy and generate a new strategy through evolutionary operations such as strategy crossover and mutation based on the evaluation results. The policy evolution mechanism enables the agent to gradually optimize its task execution strategy during operation and achieve self-optimization.

[0124] The online optimization and feedback adjustment module is used to adjust the agent's strategy online during the production process based on real-time data. This module accelerates strategy updates through an incremental learning sharing mechanism and optimizes both short-term and long-term behavioral performance based on a multi-level feedback mechanism to improve system adaptability. Furthermore, by introducing a multi-level feedback mechanism, the agent can not only make short-term adjustments based on its current state, but also adjust its behavior based on long-term task execution performance (such as adjusting the priority or action method for a certain task type). This multi-level feedback mechanism further refines the agent's behavior, thereby gradually optimizing task execution efficiency over the long term.

[0125] In some implementations, an incremental learning sharing mechanism between agents is introduced, building upon traditional online learning. Policy changes updated by each agent during production are synchronized through a shared experience pool, allowing other agents to selectively learn these incremental updates without requiring complete retraining. This approach significantly reduces policy adjustment time and enables multi-agent systems to more rapidly learn the latest task execution methods.

[0126] In some embodiments, multi-dimensional environmental perception mechanisms can be incorporated into online learning modules to not only capture real-time production line status data but also contextually identify potential trends in the environment. For example, if an agent detects an increase in material consumption on a production line, it can issue an early warning and adjust its strategy to avoid downtime caused by material shortages.

[0127] The hybrid cloud-edge collaborative computing architecture module supports complex policy training and global data analysis in the cloud, while simultaneously processing the agent's real-time decision-making and environmental perception tasks at the edge. This hybrid cloud-edge collaborative computing architecture ensures high computing density while achieving low-latency, real-time response, enabling the agent to excel in data-intensive and fast-response tasks.

[0128] The intelligent production suggestion and human-machine interaction module analyzes bottlenecks and task execution data on the production line. The system automatically generates optimization suggestions (such as adjusting the delivery frequency of a certain type of material or optimizing a specific task path) and presents them to the operator through the human-machine interaction interface. Supporting natural language interaction, operators can interact with the system through voice or text commands, adjusting scheduling parameters in real time, and further improving the efficiency of human-machine collaboration. This human-machine collaboration method increases the system's flexibility at different production stages and enables operators to easily optimize and adjust production processes.

[0129] To verify the effectiveness of a multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning in a flexible production line, an experimental environment was established. This simulated a multi-robot collaborative production workshop, including 10 agents of three types: automated handling robots, assembly robots, and inspection robots. Each type had distinct tasks and functions. These agents were tasked with collaboratively assembling, inspecting, and delivering multiple products. The experimental workshop was equipped with a variety of sensors, including lidar, high-definition cameras, pressure sensors, and force sensors, to collect real-time environmental data from the production line. The experimental site was equipped with complex material flow routes and a variety of equipment (such as assembly lines, inspection tables, and assembly workbenches) to simulate the interactions between different devices in a real production line.

[0130] To enable real-time communication and task coordination among agents, the experimental environment also includes an edge computing node for processing real-time decision-making and data exchange among multiple agents. Furthermore, a cloud-connected computing platform is used to train and optimize complex policy models. The system's core algorithms and task scheduling strategies are supported by the SAC reinforcement learning model and graph neural networks (GNNs), ensuring that each agent can effectively allocate tasks and optimize paths across diverse production tasks.

[0131] The goal of the experimental environment is to simulate the performance of the intelligent scheduling system in multiple dimensions, such as task execution efficiency, energy consumption optimization, and task response speed of the flexible production line under different production loads, sudden task changes, and equipment failures, thereby verifying the reliability and superiority of the system.

[0132] The following are the specific steps of the experiment:

[0133] Step 1: Environment initialization

[0134] 1.1 Start the experimental environment, including the edge computing node and the cloud computing platform, and ensure a stable network connection between the two.

[0135] 1.2 Start various sensors in the production workshop, including lidar, high-definition cameras, force sensors, etc., to ensure that real-time data can be transmitted to edge nodes and used by intelligent agents for decision-making.

[0136] 1.3 Initialize the positions and initial states of various intelligent agents (handling robots, assembly robots, and inspection robots), and connect the intelligent agents to edge computing nodes to achieve real-time data transmission and task scheduling.

[0137] Step 2: Reinforcement Learning Model Training and Deployment

[0138] 2.1 Centralized training of multi-agent strategy models is conducted using a cloud computing platform. Training data includes historical production data and task execution data from simulation environments. The SAC (Soft Actor-Critic) algorithm is used during training, combined with a multi-objective reward function, to optimize each agent's task execution strategy.

[0139] 2.2 Download the trained reinforcement learning model to the edge computing node and deploy it to each intelligent agent, so that it can make real-time decisions based on the environmental status during the actual production process.

[0140] 2.3 Start the shared experience pool module. When the intelligent agent performs tasks in actual production, it will upload experience data (such as status, action, reward, etc.) to the experience pool for subsequent online strategy optimization.

[0141] Step 3: Task Scheduling and Execution

[0142] 3.1 At the beginning of the experiment, a batch of production tasks are randomly generated, including material handling, product assembly, product testing, etc., and the tasks require each intelligent agent to complete them collaboratively.

[0143] 3.2 The multi-agent decision-making module assigns specific tasks to each agent based on real-time perception data and the strategy generated by the SAC model. The task scheduling and path planning module uses a graph neural network (GNN) model to optimize the agent's path selection and avoid path conflicts between agents.

[0144] 3.3 Each agent operates according to the assigned task. For example, the transport robot is responsible for transporting materials from the warehouse to the assembly area, the assembly robot completes the assembly of components, and the inspection robot performs quality inspection after the product is assembled.

[0145] 3.4 The online optimization module continuously monitors the state changes and task completion status of each agent during task execution. When changes in the environment are detected (such as increased task requirements, equipment failure, etc.), the agent's tasks and paths are adjusted in real time.

[0146] Step 4: Exception handling and strategy adjustment

[0147] 4.1 Artificially set up some abnormal situations in the experiment, such as adding sudden tasks, equipment downtime due to temporary failures, and partial path occupation, to test the system's response speed and adjustment capabilities under abnormal circumstances.

[0148] 4.2 The agent adjusts its strategy when the environment changes based on the online learning mechanism, quickly acquires new experience from exception handling through incremental learning, and uploads it to the shared experience pool.

[0149] 4.3 After receiving abnormal feedback, the system dynamically adjusts the task allocation plan and uses the adaptive entropy adjustment mechanism to enable the intelligent agent to explore and find adaptive strategies more quickly when facing new and complex environments.

[0150] Step 5: Data Collection and Analysis

[0151] 5.1 Record the task completion time, energy consumption, number of task allocation adjustments, number of path planning adjustments, production efficiency and other data of each agent during the experiment.

[0152] 5.2 Perform real-time data analysis on edge computing nodes, monitor the task execution efficiency and path planning effect of each intelligent agent, and generate real-time data reports.

[0153] 5.3 After the experiment, all data will be synchronized to the cloud platform for further statistical analysis of various data, including the average task completion time, energy consumption changes, collaboration efficiency between intelligent agents, and response time under different abnormal situations.

[0154] Step 6: Experimental results evaluation

[0155] 6.1 Compare the scheduling efficiency and overall operation stability of the production line of the intelligent scheduling system under different loads (high load, low load) and different abnormal situations (sudden increase in tasks, equipment failure).

[0156] 6.2 Analyze the difference between the strategy optimization speed of the intelligent agent after using the incremental learning mechanism and traditional online learning, and verify the improvement of incremental learning on rapid adaptation ability.

[0157] 6.3 By adjusting the weights of the multi-objective reward function, analyze the balance between energy consumption and production efficiency, and evaluate the system's adaptability to various optimization objectives under different production requirements.

[0158] Step 7: Experimental summary and optimization suggestions

[0159] 7.1 Based on the experimental results, evaluate the actual performance of the multi-agent system in the production line and summarize the advantages and disadvantages of the system in terms of task allocation, path planning, and online learning.

[0160] 7.2 In response to the problems found in the experiment (such as slow strategy adjustment in certain specific situations, communication delays between agents, etc.), further improvement measures and optimization directions are proposed.

[0161] 7.3 Combining the intelligent production suggestions in the experiment with the optimization plan generated by the human-computer interaction module, make suggestions for adjustments to the actual production line, including optimizing the production process and improving equipment utilization.

[0162] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] An embodiment of the present invention further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0164] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] refer to Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0166] The processor 1001 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0167] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute a multi-agent collaborative scheduling method based on maximum entropy reinforcement learning according to an embodiment of the present invention.

[0168] Input / output interface 1003, used to implement information input and output;

[0169] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0170] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0171] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0172] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning.

[0173] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] An embodiment of the present invention further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned multi-agent collaborative scheduling method based on maximum entropy reinforcement learning.

[0175] In summary, the multi-agent collaborative scheduling method and system based on maximum entropy reinforcement learning in the embodiment of the present invention has the following advantages:

[0176] 1. This embodiment of the present invention leverages the high exploratory nature of maximum entropy reinforcement learning, a multi-agent shared experience pool, a hierarchical collaboration strategy, and an adaptive entropy weight adjustment mechanism to enhance the task allocation and path optimization capabilities of multi-robot systems in dynamic production environments. This system, combining multimodal perception and online learning, enables efficient collaboration and real-time optimization among agents during the production process, providing an intelligent and adaptable scheduling solution for flexible production lines.

[0177] 2. This embodiment of the present invention designs a scheduling algorithm based on maximum entropy reinforcement learning, enabling intelligent agents to quickly explore and find appropriate task allocation and path planning strategies in highly uncertain and dynamically changing production environments, thereby improving the adaptability of the production line. Through an adaptive entropy weight adjustment mechanism, the intelligent agent automatically balances exploration and utilization at different production stages, improving strategy optimization speed and decision-making flexibility.

[0178] 3. This embodiment of the present invention utilizes a shared experience pool and hierarchical collaborative strategies across multiple agents to enable effective information sharing and collaborative decision-making among multiple robots during task execution, reducing resource conflicts and path interference between robots and optimizing the order of task execution. A multi-agent communication mechanism based on a graph neural network enables agents to fully consider the status and task status of surrounding robots when making decisions, thereby improving the overall collaborative efficiency of the multi-robot system.

[0179] 4. This embodiment of the present invention combines a multimodal perception module with an online learning mechanism, enabling intelligent agents to perceive changes in the production environment in real time and dynamically adjust scheduling strategies based on new data, achieving self-learning and continuous optimization during the production process. An online feedback mechanism between intelligent agents enables the system to quickly adjust strategies when production anomalies occur (such as equipment failures and material shortages), ensuring efficient and stable operation of the production line.

[0180] 5. This embodiment of the present invention leverages the high exploratory nature of maximum entropy reinforcement learning to enable intelligent agents to rapidly adapt to environmental changes in diverse production scenarios, avoiding the problem of prematurely falling into local optima and resulting in decreased overall efficiency. The introduction of a multi-agent priority training mechanism enables intelligent agents to more quickly find optimal strategies for high-priority tasks, improving the system's responsiveness to critical tasks.

[0181] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0182] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0184] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0185] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0186] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0187] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0188] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0189] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A multi-agent collaborative scheduling method based on maximum entropy reinforcement learning, characterized in that: The following steps are involved: generating an environmental state vector according to first environmental data of the production line; generating an initial production task according to the environmental state vector; Performing task scheduling operations on the initial production tasks and generating path planning; The environmental state vector, the initial production task, and the path planning are stored to obtain experience data; Based on the empirical data, the initial strategy model is trained using a maximum entropy reinforcement learning algorithm to obtain a target strategy model; According to the target strategy model, a target scheduling strategy for multi-agent collaboration is obtained; The performing of task scheduling operations on the initial production tasks and generating path planning includes: Performing a time series analysis based on the first environmental data and the initial production task to obtain predicted load fluctuation data; Establishing a pre-allocation mechanism for the initial production tasks; Through the multi-level graph neural network communication mechanism, the multi-agent communication mechanism is divided into local communication mechanism and global broadcast mechanism; Performing task scheduling on the initial production task according to the predicted load fluctuation data, the pre-allocation mechanism, the local communication mechanism, and the global broadcast mechanism; The path planning is generated through the path interaction avoidance mechanism and the multi-level graph neural network communication mechanism.

2. A multi-agent collaborative scheduling method based on maximum entropy reinforcement learning according to claim 1, characterized in that: The following steps are also included: collecting second environmental data of the production line, adjusting the initial scheduling strategy of the target strategy model, and obtaining an intermediate scheduling strategy; Through a hybrid cloud-edge collaborative computing architecture, the target policy model is trained according to the intermediate scheduling policy and the second environment data to generate the target scheduling policy; According to the target scheduling strategy, the target production task is executed to obtain feedback information.

3. The multi-agent collaborative scheduling method based on maximum entropy reinforcement learning according to claim 1 is characterized in that: After the step of storing the environmental state vector, the initial production task, and the path planning to obtain experience data, the following steps are also included: Construct feature weights of multiple agents; Constructing a strategy parameter loss function through KL divergence and the feature weights; The strategy parameter loss function is used to weight the process of learning the experience data.

4. The multi-agent collaborative scheduling method based on maximum entropy reinforcement learning according to claim 1 is characterized in that: The method of training the initial strategy model by using the maximum entropy reinforcement learning algorithm based on the empirical data to obtain the target strategy model includes the following steps: Construct dynamic multi-objective reward function; Constructing an objective function of a maximum entropy reinforcement learning algorithm based on the dynamic multi-objective reward function and the entropy coefficient; Based on the empirical data, the initial strategy model is trained using the objective function to obtain the target strategy model.

5. A multi-agent collaborative scheduling method based on maximum entropy reinforcement learning according to claim 4, characterized in that: The objective function of the maximum entropy reinforcement learning algorithm is constructed based on the dynamic multi-objective reward function and the entropy coefficient. The formula used includes: in, Represents the objective function, which is the strategy performance indicators; represents the time step, ; Represents the state; Represents action; Represents the state Take action The instant reward when , that is, the dynamic multi-objective reward function; Represents according to status The entropy coefficient is adjusted according to the dynamic complexity; Representative Strategy In state Conditional entropy under ; Representative in Strategy Lower state and actions The expected value of the joint distribution of .

6. A multi-agent collaborative scheduling system based on maximum entropy reinforcement learning, characterized by: include: A first module is used to generate an environmental state vector according to first environmental data of the production line; A second module is used to generate an initial production task according to the environmental state vector; The third module is used to perform task scheduling operations on the initial production tasks and generate path planning; A fourth module is configured to store the environmental state vector, the initial production task, and the path planning to obtain empirical data; A fifth module is used to train the initial policy model using a maximum entropy reinforcement learning algorithm based on the empirical data to obtain a target policy model; The sixth module is used to obtain a target scheduling strategy for multi-agent collaboration based on the target strategy model; The third module is specifically used for: Performing a time series analysis based on the first environmental data and the initial production task to obtain predicted load fluctuation data; Establishing a pre-allocation mechanism for the initial production tasks; Through the multi-level graph neural network communication mechanism, the multi-agent communication mechanism is divided into local communication mechanism and global broadcast mechanism; Performing task scheduling on the initial production task according to the predicted load fluctuation data, the pre-allocation mechanism, the local communication mechanism, and the global broadcast mechanism; The path planning is generated through the path interaction avoidance mechanism and the multi-level graph neural network communication mechanism.

7. A multi-agent collaborative scheduling system based on maximum entropy reinforcement learning according to claim 6, characterized in that: Also includes: A seventh module is configured to collect second environmental data of the production line, adjust the initial scheduling strategy of the target strategy model, and obtain an intermediate scheduling strategy; An eighth module is configured to train the target policy model according to the intermediate scheduling policy and the second environment data through a hybrid cloud-edge collaborative computing architecture to generate the target scheduling policy; The ninth module is used to execute the target production task according to the target scheduling strategy and obtain feedback information.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images