Multi-agent cooperative communication and scheduling method based on deep reinforcement learning

Abstract

The invention relates to the technical field of computers, in particular to a multi-agent cooperative communication and scheduling method based on deep reinforcement learning, and the method comprises the following steps: S1, constructing a multi-agent system architecture; s2, designing a deep reinforcement learning framework; step S3, introducing a GOLSC algorithm; s4, designing a scheduling rule; step S41, static distribution rules; step S42, dynamically scheduling the rule; and S5, intelligent agent collaboration and optimization are carried out. According to the invention, different types of dynamic events are divided into three types, so that the GOLSC algorithm can adopt different communication link construction strategies for different types of dynamic events to generate specific links with pertinence. After a GOLSC algorithm is introduced into a task scheduling model, numerical experiments are carried out on a large number of examples in different production environments, changes of the total delay rate and the total delay time are researched, and the effectiveness, adaptability and superiority of the algorithm model in the environments are further verified.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-agent collaborative communication and scheduling method based on deep reinforcement learning. Background Art

[0002] With the rapid development of Industry 5.0 and the continuous improvement of artificial intelligence applications, intelligent manufacturing has become a core model of modern industrial production. As the core unit of intelligent manufacturing, the smart workshop incorporates a variety of complex manufacturing resources and production activities. It contains a rich variety of agents, each with specific functions and tasks, and collaborates to achieve overall production goals. However, the increasingly complex workshop environment makes effective collaboration among multiple agents a key issue for the efficient operation of smart workshops. Traditional multi-agent job-shop scheduling (JSP) processes lack interaction between agents, making it difficult to adapt to large-scale, dynamically changing manufacturing environments and internal and external dynamic events such as new job arrivals, order cancellations, excessive bandwidth usage, and machine failures. This creates an urgent need for dynamic scheduling methods that can handle uncertain interference and compensate for effective collaboration between agents.

[0003] With the continuous development of reinforcement learning (RL), deep reinforcement learning (DRL), which combines deep learning (DL), has achieved further success in solving single-agent sequential decision-making problems in interactive and complex environments. In many real-world applications, such as robotics and autonomous driving systems, not only a single agent but often a large number of agents participate in the learning task. Such environments naturally lead to the development of DRL. The key challenges are how to establish a scalable and effective learnable solution in non-stationary environments and how to effectively communicate between agents. This is crucial for strengthening inter-agent collaboration and ultimately improving the policy quality of multi-agent deep reinforcement learning (MADRL).

[0004] In summary, existing dynamic job shop scheduling (DJSP) models based on deep reinforcement learning either adopt a single central scheduling agent architecture or a leader-follower multi-agent architecture, resulting in a serious lack of interaction between actual agents and reducing the upper limit of the optimal solution quality. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-agent collaborative communication and scheduling method based on deep reinforcement learning to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-agent collaborative communication and scheduling method based on deep reinforcement learning, comprising the following steps:

[0008] Step S1, build a multi-agent system architecture: pair each processing machine with a machine agent respectively, each machine agent manages one processing machine, is responsible for the task scheduling and decision-making of the machine, and sets up a workpiece management agent to record the status information of all workpieces and perform multi-agent dynamic workshop job scheduling. The status information includes the processing progress of the workpiece, processing technology requirements, required processing machine type, processing time, etc., and monitors the arrival of dynamic events related to the workpiece, including the arrival of new workpieces, abnormalities in workpiece processing, processing machine failures, order changes, etc.

[0009] Step S2: Design a deep reinforcement learning framework: Using the machine agents and the workpiece management agent as agent units, a deep reinforcement learning environment is constructed. The state space includes the state information of all workpieces, the status of each processing machine, the type of dynamic event, and related parameters. The action space contains the actions corresponding to different scheduling rules for unfinished workpieces selected by the machine agents, as well as the workpiece management agent's response actions to dynamic events. The reward function is designed based on indicators such as the total delay rate, average delay time, and production efficiency of the job instance. Positive rewards are given when the total delay rate decreases, the average delay time decreases, and production efficiency increases, and negative rewards are given when the total delay rate decreases, the average delay time decreases, and production efficiency increases.

[0010] Step S3: Introducing the GOLSC algorithm: This algorithm is incorporated into the task scheduling model to construct a GOLSC model. This algorithm is used to study the changes in the total delay rate and average delay time of job instances under different scheduling strategies. Specifically, the GOLSC algorithm analyzes the processing sequence of workpieces, the allocation of processing machines, and the impact of dynamic events to establish a goal-oriented scheduling strategy, providing more effective guidance for deep reinforcement learning.

[0011] Step S4: Design scheduling rules:

[0012] Step S41, Static Allocation Rules: Based on static information such as the workpiece's processing requirements and the type of machine required, pre-set rules for the machine agent to select unfinished workpieces. For example, workpieces with specific processing requirements are preferentially assigned to machine agents with the corresponding processing capabilities and currently available. Workpieces with earlier delivery dates are prioritized by machine agents based on their due date.

[0013] Step S42: Dynamic Scheduling Rules: The rules for selecting unfinished workpieces by machine agents are dynamically adjusted based on real-time monitored dynamic events and the status information of each agent. When a processing machine fails, unfinished workpieces on the failed machine are reallocated to other available machine agents, taking into account their current load and processing capacity. When a new workpiece arrives, the workpiece management agent sends this information to all machine agents, and each machine agent decides whether to select the new workpiece based on its own status and the dynamic scheduling rules.

[0014] Step S5: Agent Collaboration and Optimization: The machine agents select unfinished workpieces based on the designed static allocation and dynamic scheduling rules to schedule processing tasks. During the scheduling process, the workpiece management agent updates workpiece status information in real time and promptly notifies the relevant machine agents of dynamic events. Through deep reinforcement learning, each machine agent continuously learns and optimizes scheduling strategies to improve agent collaboration efficiency and production responsiveness. This addresses the problem of reduced production efficiency in traditional smart manufacturing workshops, which is caused by a lack of targeted response to different types of dynamic events and a lack of interactivity between agents.

[0015] Preferably, the GOLSC model in step S3 is initially established in a simple propagation environment under MPE, including:

[0016] There are n node agents N={N1,N2,...,N n} and m landmarks, the agent must learn to cover all landmarks, where the node agent i’s own position is np i , the location of landmark j is np j , from the perspective of node agent i, the relative positions of other node agents are the set onp = {onp1, onp2, ..., onp n-1};

[0017] The CBRP topology protocol in LSC is used to construct the entire node set V = {v1, v2, ..., v n}, advanced node set V h ={v h1 ,v h2 ,...,v hn} and the low-level node set V l ={v l1 ,v l2 ,...,v ln}, and the edge set E={E1,E2,...,E n(n-1)}, where E k =[s k ,e k ,mes k ,dis k,gw k ];

[0018] where s k ,e k Represents edge E k The starting and ending nodes of mes k Represents the perception message transmitted through this edge, dis k ,gw k Represents edge E k The physical distance and the generated specific link weight;

[0019] A specific link is defined as a path, and the weight of node agent i is w i At time t, the dynamic event label is flag t Bandwidth occupancy is represented by b t,i , the number of errors is expressed as wrong t,i , the general reward under the LSC module is r, and the specific reward under the GO module is r g , the final reward is R = r + r g .

[0020] Preferably, the multi-agent dynamic job shop scheduling in step S1 includes:

[0021] n consecutive workpieces J={J1,J2,...,J n};

[0022] m machines M={M1,M2,...,M m};

[0023] The information recorded in the job management agent includes:

[0024] Homework J i The arrival time and due date are A i and D i ;

[0025] Homework J i Composed of operations, CO i Indicates the operation sequence number that the job is currently processing;

[0026] O i,j (j=1,2,...,n i ) represents homework J i The jth operation of

[0027] Each operation O i,j Can be used on any available machine M k (M k ∈M i,j , ) for processing;

[0028] The operation O i,j On machine M k The processing time on i,j,k ;

[0029] Assignment J i The start and end time set of wt i For homework J i delay time.

[0030] Preferably, the self-information recorded by each machine agent includes: SM k ={sm k,1 ,sm k,2 ,...}, EM k ={em k,1 ,em k,2 ,...} are respectively machine M k The start and end time set, T k ={T k,1 ,T k,2 ,...} is the set of working time periods of the machine;

[0031] Among them, the machine malfunctions randomly, FM k ={fm k,1 ,fm k,2 ,...} represents machine M k The set of failure times, where fm k,r Represents machine M k The rth failure time fm k,r =[sf k,r ,ef k,r ];

[0032] where sf k,r and ef k,r Represents machine M k The start time of the rth fault and the end time of repair;

[0033] Correspondingly, machine M k The repair time is rt k , at time t, U t ={u t,1 ,u t,2 ,...,u t,m} represents the utilization set of all machines;

[0034] B t ={b t,1 ,b t,2 ,...,b t,m} represents the bandwidth usage of all machines, wrong t,k Represents machine M k Number of failures;

[0035] IB i Represents the set to be selected, if IB i =k represents job J i Has been machine M k To be selected, if IB i =-1 represents job J i Not selected by any machine;

[0036] UCt represents the set of unfinished operations at time t, RT i Representative assignment J i The remaining operation processing time, three targets, ATWT stands for average delay time, and TWR stands for total delay rate.

[0037] Preferably, DFJSP should satisfy the following predefined constraints:

[0038] 1) A machine can only process one operation of a job at a time;

[0039] 2) All operations of the same job should be processed in a fixed order;

[0040] 3) Transportation time and setup time are negligible;

[0041] 4) Different operations may take different processing times on different machines;

[0042] 4) All machines are available from the beginning, and each job is available only when it arrives;

[0043] 5) Once a machine fails, it should be repaired immediately, and if any jobs are transferred to other machines due to the failure, the processing time needs to be recalculated.

[0044] Preferably, the multi-agent dynamic job shop scheduling in step S1 is further optimized to include two objectives, as follows:

[0045] The total delay rate (TWR) is defined as formula (4), and the total delay time (ATWT) is defined as formula (5);

[0046]

[0047] The specific constraints are as follows:

[0048]

[0049]

[0050] Among them, equations (6) and (7) define DFJSP as an integrated optimization problem, equation (8) defines that each operation should only be assigned to one available machine, equation (9) defines that the job can only be processed after it arrives, equation (10) defines that the end time of the previous operation is less than the start time of the next operation, that is, the processing operations need to be performed in a fixed order, equation (11) defines the capacity constraint, equation (12) defines that each job can only be processed by one machine at the same time, and equation (13) defines that all jobs have been processed.

[0051] Preferably, the step S42, dynamic scheduling rules, further includes:

[0052] When a scheduling task begins, the machine agent can interact and collaborate with the job management agent and share real-time updates and feedback. The machine agent will select a job based on its own local observations. When processing a job, the agent can still mark the next job it wants to perform through the waiting set.

[0053] After the operation is completed, the machine becomes idle and prepares for the next marked job. If the marked job is not processed because the machine is busy, and another idle machine also marks the job, it will be processed by the idle machine. During this process, the agent constantly updates its own state.

[0054] If the operation is the last operation of the job, it means that the job has been completed;

[0055] The job management agent will obtain the job completion rate, deadline, number of operations, and pending set status of all jobs and share it with the machine agent. Through this information, the machine agent and the job management agent will cooperate to complete all jobs while pursuing the same goal.

[0056] Preferably, the machine agents share a Q network with identical parameters and an experience pool, Buffer, to store the agent's state information and actions. This structure facilitates centralized optimization and decentralized execution, while meeting the convenience requirements of decentralized agents and utilizing deep reinforcement learning to centrally optimize network parameters.

[0057] The machine agent M k The observed characteristics of are defined as:

[0058] Machine Agent M k Last working time:

[0059] ls k =max(SM k ); (17)

[0060] Machine Agent M k Last released time:

[0061] lr k =max(EM k ); (18)

[0062] Machine Agent M k Utilization at time t: (19)

[0064] Machine Agent M k Busy status:

[0065]

[0066] Machine Agent M k Bandwidth usage:

[0067]

[0068] Where n≤4Blog2K / log(2πeσ 2 ) represents the maximum number of symbols that can be accommodated in the communication channel per second, N is the number of agents, L is the message length, J represents the communication between the current agent and J agents, F is the sampling frequency set to F = 45 Hz, K = 2, σ 2 =0.151, B = 2750 bit / s, NJLF represents the actual total communication symbols of the agent;

[0069] Among them, wrong k represents the number of failures of machine k;

[0070] CO i Represents the job J that the machine is processing i CO i operations;

[0071] The machine is processing job J i Completion rate:

[0072]

[0073] D i Represents the job J that the machine is processing i the expiry date;

[0074] IB i Indicates that the job being processed is selected i The machine number of the next operation.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] This paper introduces goal-oriented communication (GO) into LSC by combining deep Q-learning (DQN) with the Dijkstra algorithm, which makes up for the current defect of the lack of autonomous communication awareness of agents in the field of multi-agent structured communication. It also preliminarily verifies the effectiveness of our GOLSC algorithm in a simple propagation environment of multi-agent particles (MPE).

[0077] In the proposed real-time task scheduling model, each processing machine is paired with an intelligent agent to record machine status information, including bandwidth usage and number of errors. By carefully designing a workpiece management agent to record the status information of all workpieces and monitor the arrival of dynamic events related to the workpieces, a reward structure based on the change in total twitch time (TWT) at different time steps is designed to calculate the rewards corresponding to different higher goals.

[0078] This paper categorizes different types of dynamic events into three types, enabling the GOLSC algorithm to employ different communication link construction strategies for each type of dynamic event, generating targeted, specific links. After incorporating the GOLSC algorithm into the task scheduling model, numerical experiments were conducted on a large number of examples in various production environments, examining the overall delay rate and overall delay time. This further validated the effectiveness, adaptability, and superiority of the proposed algorithm model in these environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a structural framework diagram of the GOLSC model of the present invention;

[0080] Figure 2 It is the structural framework diagram of the DFJSP model of the present invention;

[0081] Figure 3 This is a linear comparison diagram of the total delay rate of the communication-free scheduling model of the present invention under a static environment with different numbers of machines and initial job numbers;

[0082] Figure 4 This is a comparison chart of the simultaneous training of multiple models within 3000 sets in the present invention;

[0083] Figure 5 This is a comparison chart of the total delay rate (TWR) and reward changes of different algorithms under the same environment of the present invention;

[0084] Figure 6 This is a comparison chart of the average delay time (ATWT) of different groups of algorithms under different job numbers n in a representative dynamic environment of the present invention. DETAILED DESCRIPTION

[0085] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0086] Example 1:

[0087] (1) Multi-agent structured communication

[0088] The GOLSC model of the present invention is initially established in a simple propagation environment under MPE: n node agents N = {N1, N2, ..., N n} and m landmarks. The agent must learn to cover all landmarks. The position of the node agent i itself is np i , the position of landmark j is np j , the relative positions of other node agents from the perspective of node agent i are the set onp = {onp1, onp2, ..., onp n-1}. The CBRP topology protocol in LSC is used to construct the complete node set V = {v1, v2, ..., v n}, advanced node set V h ={v h1 ,v h2 ,...,v hn} and the low-level node set V l ={v l1 ,v l2 ,...,v ln}, and the edge set E={E1,E2,...,E n(n-1)}, where E k =[s k ,e k ,mes k ,dis k ,gw k ], where s k ,e k Represents edge E k The starting and ending nodes of mes k Represents the perception message transmitted through this edge, dis k ,gw k Represents edge E k The physical distance and the generated specific link weight. The specific link is defined as path. The weight of node agent i is w i At time t, the dynamic event label is flag t Bandwidth occupancy is represented by b t,i, the number of errors is expressed as wrong t,i The general reward under the LSC module is r, and the specific reward under the GO module is r g , the final reward is R = r + r g In preliminary experiments, we take maximizing the reward R as the objective to verify the effectiveness of the communication model.

[0089] (2) Multi-agent dynamic job shop scheduling

[0090] The present invention studies multi-agent dynamic job scheduling in a discrete environment, where the job is a flexible job, so we will regard it as dynamic flexible job shop scheduling (DFJSP), which is defined as follows. n consecutive arriving workpieces J = {J1, J2, ..., J n}. m machines M={M1,M2,...,M m}.

[0091] The information recorded in the job management agent includes: i The arrival time and due date are A i and D i Homework J i It consists of operations, CO i Indicates the operation number that the job is currently processing. i,j (j=1,2,...,n i ) represents homework J i The jth operation of each operation O i,j Can be used on any available machine M k (M k ∈M i,j , ) is processed. i,j On machine M k The processing time on i,j,k .

[0092] Assignment J i The start time and end time of wt i For homework J i delay time.

[0093] The self-information recorded by each machine agent includes: SM k ={sm k,1 ,sm k,2 ,...}, EM k ={em k,1 ,em k,2 ,...} are respectively machine M k The start and end time set, T k={T k,1 ,T k,2 ,...} is the set of machine working time periods. Among them, the machine fails randomly, FM k ={fm k,1 ,fm k,2 ,...} represents machine M k The set of failure times, where fm k,r Represents machine M k The rth failure time fm k,r =[sf k,r ,ef k,r ], where sf k,r and ef k,r Represents machine M k The start time of the rth fault and the end time of repair. k The repair time is rt k At time t, U t ={u t,1 ,u t,2 ,...,u t,m} represents the utilization rate of all machines, B t ={b t,1 ,b t,2 ,...,b t,m} represents the bandwidth usage of all machines, wrong t,k Represents machine M k Number of failures. i Represents the set to be selected, if IB i =k represents job J i Has been machine M k To be selected, if IB i =-1 represents job J i Not selected by any machine. UC t Represents the set of unfinished operations at time t. RT i Representative assignment J i The remaining operation processing time. Three objectives, ATWT stands for average delay time, TWR stands for total delay rate, among which DFJSP should meet the following predefined constraints:

[0094] 1) A machine can only process one operation of a job at a time.

[0095] 2) Operations of the same job should be processed in a fixed order.

[0096] 3)Transportation time and setup time are negligible.

[0097] 4) Different operations may take different processing times on different machines.

[0098] 4) All machines are available from the beginning, and each job is available only when it arrives.

[0099] 5) If a machine fails, it should be repaired immediately. If jobs are transferred to other machines due to the failure, the processing time needs to be recalculated.

[0100] The two optimization goals of the present invention are: the total delay rate (TWR) is defined as formula (4), and the total delay time (ATWT) is defined as formula (5).

[0101]

[0102] The specific constraints are as follows:

[0103]

[0104] Equations (6) and (7) define DFJSP as an integrated optimization problem. Equation (8) defines that each operation should only be assigned to one available machine. Equation (9) defines that a job can only be processed after it arrives. Equation (10) defines that the end time of the previous operation is less than the start time of the next operation, that is, the processing operations need to be performed in a fixed order. Equation (11) defines the capacity constraint. Equation (12) defines that each job can only be processed by one machine at a time. Equation (13) defines that all jobs have been processed. In general, these constraints ensure the feasibility of the derived solution.

[0105] Example 2:

[0106] We will detail the proposed goal-oriented GOLSC model and validate it in a preliminary MPE environment, based on the definition of extended observations, its integration with Dijkstra's algorithm, and the definition of a reward function. We will then introduce our real-time scheduling model, including the agent architecture and local observation definitions, the proposed allocation and scheduling rules, the proposed agent network structure, and the definition of a reward function. Finally, we will present the training and implementation architecture of GOLSC in a scheduling environment.

[0107] (1) Goal-oriented GOLSC model

[0108] This research is based on a simple MPE communication environment, where agents aim to cover all landmarks while avoiding collisions. The LSC model achieves inter-agent communication by building an agent topology. Our goal is to achieve agent personalization by building specific links based on the existing topology. This process is divided into two parts:

[0109] 1) According to different dynamic event labels flag at time t t, learn to generate specific weights for all edges through the DQN algorithm. 2) Combined with the Dijkstra algorithm to obtain different flags t The total specific weight and the shortest specific link.

[0110] In the first part, we regard the process as a multi-agent partially observable Markov decision process (POMDP) ​​and use the existing multi-agent topology to achieve further centralized optimization and decentralized decision-making. The framework is as follows Figure 1 .

[0111] Specifically, in the second part, each time step agent randomly selects non-adjacent target agents and dynamic events, where the dynamic events are divided into three categories according to the construction goals, namely the shortest physical distance requirement, the minimum DQN specific weight generator (GWG) The network is used to generate a specific weight gw of the existing link. Among them, the specific local observation used by the node agent i to generate a specific weight is defined as:

[0112]

[0113] Among them, flag t Specifically defined as:

[0114]

[0115] Then, the shortest specific link that meets the dynamic event requirements is obtained through a specific routing protocol based on Dijkstra (Goal Oriented). Otherwise, direct communication is performed. Specifically, it is Algorithm 1:

[0116] Algorithm 1. Dijkstra-based shortest path algorithm in the Goal Oriented protocol.

[0117] Input: all node sets V, edge sets E, flag t , starting node start_node, target node goal_node

[0118] Output: specific path path, total path weight total_weight, reward

[0119]

[0120] In terms of rewards, we will t And conditional judgment is performed on the generated specific link path to build a reward structure, which is specifically Algorithm 2:

[0121] Algorithm 2. Reward function get_reward() in Goal Oriented protocol

[0122] Input: All node sets V, edge sets E, dynamic event type labels flag t , specific link path;

[0123] Output: Reward

[0124] ① Initialize total_distance, total_bandwidth_rate, and total_wrong_times to represent the total physical distance, total bandwidth usage, and total number of errors respectively;

[0125] ② Traverse the path and edge set E, obtain the physical distance of each edge, the bandwidth usage and error count of each node contained therein, and accumulate them into total_distance, total_bandwidth_rate, and total_wrong_times;

[0126] ③Calculate the link path’s avg_distance, avg_bandwidth_rate, and avg_wrong_times

[0127] ④ According to the distribution of avg_distance in the interval (-∞,1], (1,3], (3,+∞), give rewards d_r of 5, 0, and -5 respectively;

[0128] ⑤ According to the distribution of avg_bandwidth_rate in the interval (-∞,0.2], (0.2,0.5], (0.5,+∞), give rewards b_r of 5, 0, and -5 respectively;

[0129] ⑥Give rewards w_r of 5, 0, or -5 based on the distribution of avg_wrong_times in the interval or value 0, 1, (1, +∞) respectively;

[0130] ⑦In different flags t Give d_r, b_r, w_r different weights, and finally calculate the weighted reward and

[0131] The loss function in the specific weight generator is updated by formula (16):

[0132]

[0133] In the formula in Represents the specific reward of agent i.

[0134] To address the uncertainty caused by the randomness of dynamic events during each training session: 1) The GO module and LSC module have different experience pools, potentially preventing synchronization. Therefore, we set up separate experience pools (LSC Buffer and GO Buffer) for the LSC and GO modules in the experience replay area. 2) Because the observations used to generate their own weights differ from the specific observation dimensions used to generate links, we also adjusted the buffer capacity. 3) Regarding update frequency, we only update the GO module when specific communications occur, i.e., when dynamic events arrive, generating specific links and corresponding rewards. These constraints ensure the rigor and reliability of our proposed algorithm.

[0135] (2) Real-time scheduling model

[0136] This part of the study also constructs the problem as a POMDP. First, our multi-agent-based scheduling framework includes multiple machine agents and a centralized job management agent, and describes the integration and interaction between multiple machine agents and between machine agents and job management agents. Figure 2 As shown, the job management agent stores the status information of all jobs, the machine agents have been created at the beginning, and each machine is paired with a machine agent.

[0137] When a scheduling task begins, the machine agent selects a job based on its local observations, as it can interact and collaborate with the job management agent, sharing real-time updates and feedback. While processing a job, the agent can also mark the next job it wants to perform using the pending set. After the operation is completed, the machine becomes idle, preparing for the next marked job. If a marked job is delayed due to a busy machine and begins processing, and another idle machine also marks the same job, it will be processed by the idle machine, with the agent constantly updating its status during the process. If the operation is the last in the job, the job is considered completed. Simultaneously, the job management agent obtains the completion rate, deadline, number of operations, and pending set status of all jobs and shares this information with the machine agent. Using this information, the machine agent and the job management agent collaborate to complete all jobs, pursuing the same goal.

[0138] Decentralized machine agents share a Q-network with identical parameters and an experience pool (Buffer) for storing agent state information and actions. This structure facilitates centralized optimization and decentralized execution, while simultaneously meeting the convenience requirements of decentralized agents and leveraging deep reinforcement learning to centrally optimize network parameters.

[0139] The machine agent M k The observed characteristics of are defined as:

[0140] 1) Machine Agent M k Last working time:

[0141] ls k =max(SM k ); (17)

[0142] 2) Machine Agent M k Last released time:

[0143] lr k =max(EM k ); (18)

[0144] 3) Machine Agent M k Utilization at time t:

[0145]

[0146] 4) Machine Agent M k Busy status:

[0147]

[0148] 5) Machine Agent M k Bandwidth usage:

[0149]

[0150] (According to Hu Guangzheng et al., through bandwidth-limited gated communication, we know that n≤4Blog2K / log(2πeσ 2 ) represents the maximum number of symbols that can be accommodated in the communication channel per second, N is the number of agents, L is the message length, J represents the communication between the current agent and J agents, and F is the sampling frequency. Here we set it to F = 45 Hz, K = 2, σ 2 =0.151, B = 2750 bit / s, NJLF represents the actual total communication symbols of the agent. In this invention, we regard the ratio of the two as the bandwidth occupation of the agent)

[0151] 6) wrong k represents the number of failures of machine k;

[0152] 7) CO i Represents the job J that the machine is processing i CO i operations;

[0153] 8) The machine is processing job J i Completion rate:

[0154]

[0155] 9)D i Represents the job J that the machine is processing i the expiry date;

[0156] 10)IB i Indicates that the job being processed is selected i The machine number of the next operation.

[0157] As shown in Table 1, the action space specified for the machine agent is discrete and contains 10 dimensions. This action represents the machine agent selecting a job to process from among the unfinished jobs, based on its priority under a certain scheduling rule. For example, if the SPT rule requires selecting the job with the shortest total processing time, the agent will evaluate the total processing time of all unfinished jobs and select the job with the smallest value.

[0158] Table 1 Heuristic scheduling rules for machine allocation

[0159]

[0160] To enable agents to select jobs that match their own status, we designed allocation and scheduling rules for both static and dynamic environments to guide the agent's job selection. In static environments, the agent will additionally determine its own busy and faulty status before assigning jobs, as shown in Algorithm 3.

[0161] Algorithm 3. Agent allocation and scheduling rules in static environment

[0162]

[0163]

[0164] Traditional DFJSP workshops have difficulty handling different types of dynamic events, such as order cancellations, high bandwidth usage, or machine failures, in a targeted manner. However, under our dynamic scheduling method, machine agents adopt different scheduling rules when facing different dynamic events, as shown in Algorithm 4:

[0165] Algorithm 4. Agent allocation and scheduling rules in dynamic environments

[0166]

[0167]

[0168]

[0169] In summary, when training the task scheduling model, we can add GOLSC communication as a collaboration method between intelligent agents to study the performance of this communication model in the task scheduling environment, as shown in Algorithm 5.

[0170] Algorithm 5. GOLSC-based training algorithm

[0171]

[0172]

[0173]

[0174] The reward obtained by the machine agent k is defined as the mean of the total delay time difference at each time step, as shown in formula (23):

[0175]

[0176] Example 3:

[0177] (1) Communication model experiment

[0178] We first verify the effectiveness of GOLSC in the MPE multi-particle simple propagation environment. In this environment, we expand the observation of each agent i to a specific local observation Specifically, it is formula (14).

[0179] like Figure 3 As shown in the figure. We set the number of agents N = 10 and the number of landmarks M = 10, and train our LSC and GOLSC models simultaneously in 3000 episodes. We can see that the rewards obtained by our proposed GOLSC model under the defined reward structure and specific local observations are higher than those obtained by the LSC model, and converge after 500 episodes. This means that the agent can learn to generate different flags in this environment according to the reward structure. t The corresponding specific link is given a reasonable specific reward, and the specific reward structure is shown in Algorithm 2.

[0180] (2) Scheduling model experiment

[0181] Next, we present the design of numerical experiments.

[0182] The experiments use different configurations to simulate different manufacturing environments. The main purpose is to study the effectiveness and adaptability of the proposed scheduling model with GOLSC cooperative communication. Table 2 provides the details of the training and testing, where job J i The maturity date of is defined in formula (24).

[0183]

[0184] Table 2 Numerical experiment details

[0185]

[0186] (3) Job scheduling in a static environment

[0187] In order to verify the effectiveness of the GOLSC model, we initially conducted numerical experiments in a static environment. First, we divided the model into three different groups according to different df. Then, we conducted static experiments in the same group with the same df to study the changes in the total delay rate under different numbers of machines and different numbers of jobs. Figure 4 As shown in the figure, we can see that under the same delay factor df, as the number of machine agents increases, our total delay rate (TWR) continues to decrease, and when df = 3.0, it exceeds 70% of the experimental groups with a total delay rate of TWR = 0.

[0188] (4) Job scheduling in a dynamic environment

[0189] To further demonstrate the effectiveness of the GOLSC algorithm in a dynamic scheduling environment, we conducted comparative experiments. Dynamic numerical experiments were conducted under different df,n,m environments, i.e., different delay factors, initial job numbers, and machine numbers. In this dynamic environment, dynamic events may occur at any time while the machine agent is processing jobs. For convenience, we separately show the total delay rate and reward comparison of different algorithms under the environment of df = 1.5, n = 40, and m = 20. Figure 5 As shown in the figure, under the same dynamic environment, our GOLSC algorithm outperforms all other algorithms in terms of both total delay rate and reward, including the Base baseline model with no communication, ATOC tree-structured communication, CommNet star-structured communication, and LSC communication. As the number of machine agents increases, our GOLSC model achieves the fastest gradient decrease in production delay rate and achieves the optimal production delay rate and maximum reward. Specific training-related hyperparameters and average delay time (ATWT) are shown in Table 4.

[0190] Table 3 Some hyperparameters used for training in the GOLSC algorithm

[0191]

[0192]

[0193] In summary, we can conclude that in a scheduling environment where the scale of intelligent agents continues to increase, the GOLSC algorithm we proposed has better adaptability and flexibility than other similar communication algorithms, which fully proves that the dynamic task scheduling of intelligent agents with GOLSC cooperative communication has more efficient scheduling efficiency than traditional task scheduling without communication and task scheduling with similar communication strategies.

[0194] Table 4 Average total delay time obtained by different communication models under different cases

[0195]

[0196]

[0197]

[0198] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent collaborative communication and scheduling method based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1: Build a multi-agent system architecture: Pair each processing machine with a machine agent. Each machine agent manages one processing machine and is responsible for scheduling and making decisions about the machine's tasks. A workpiece management agent is also set up to record the status information of all workpieces and perform multi-agent dynamic shop floor job scheduling. Step S2: Design a deep reinforcement learning framework: Use each machine agent and workpiece management agent as agent units to build a deep reinforcement learning environment; Step S3: Introducing the GOLSC algorithm: Adding the GOLSC algorithm to the task scheduling model to construct a GOLSC model. This algorithm is used to study the changes in the total delay rate and average delay time of job instances under different scheduling strategies. Step S4: Design scheduling rules: Step S41, static allocation rules: pre-set the rules for the machine agent to select unfinished workpieces based on static information such as the processing requirements of the workpiece and the type of processing machine required; Step S42, dynamic scheduling rules: combining the real-time monitored dynamic events and the status information of each agent, dynamically adjusting the rules for the machine agent to select unfinished workpieces; Step S5, agent collaboration and optimization: The machine agent selects unfinished workpieces according to the designed static allocation rules and dynamic scheduling rules to schedule the processing tasks. During the scheduling process, the workpiece management agent updates the status information of the workpiece in real time and notifies the relevant machine agents of dynamic events in a timely manner.

2. A multi-agent collaborative communication and scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The GOLSC model in step S3 is initially established in a simple propagation environment under MPE, including: n node agents and m landmarks, the agent must learn to cover all landmarks, where the node agent Your own position is , Landmark The location is , from the node agent Let's look at the relative positions of other node agents as a set ; Construct all node sets through the CBRP topology protocol in LSC , advanced node set and low-level node sets , and edge sets ,in ; in Represents edges The starting and ending nodes of represents the perception message transmitted through this edge, Represents edges The physical distance and the specific link weight generated; A specific link is defined as , node agent The weight of itself is At time t, the dynamic event label is Bandwidth usage is expressed as , the number of errors is expressed as , the general reward under the LSC module is r, and the specific reward under the GO module is , the final reward is .

3. The multi-agent collaborative communication and scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The multi-agent dynamic job shop scheduling in step S1 includes: Workpieces arriving continuously ; Machines ; The information recorded in the job management agent includes: Operation The arrival time and due date are and ; Operation It consists of operations, Indicates the operation sequence number that the job is currently processing; Representative work No. operations; Every operation Can be used on any available machine Processing on The operation In the machine The processing time on ; , , respectively, for the assignment The start and end time sets, For homework delay time.

4. The multi-agent collaborative communication and scheduling method based on deep reinforcement learning according to claim 3 is characterized in that: The self-information recorded by each machine agent includes: , Machine The start and end time sets, A collection of working time periods for the machine; The machine malfunctioned randomly. Indicates the machine The set of failure times, where Representative Machine No. Second failure time ; in and Represents the machine No. The start time of the fault and the end time of repair; Accordingly, the machine The repair time is , at time t, Represents the utilization set of all machines; Represents the bandwidth usage of all machines. Representative Machine Number of failures; Represents the set to be selected, if Representative work Already been machine To be selected, if Representative work Not selected by any machine; Indicates The set of unfinished operations at the moment, Representative work The remaining operation processing time, three goals, represents the average delay time, Represents the total delay rate.

5. The multi-agent collaborative communication and scheduling method based on deep reinforcement learning according to claim 4 is characterized in that: DFJSP should meet the following predefined constraints: 1) A machine can only process one operation of a job at a time; 2) All operations of the same job should be processed in a fixed order; 3) Transportation time and setup time are negligible; 4) Different operations may take different processing times on different machines; 5) All machines are available from the beginning, and each job is available only when it arrives; 6) Once a machine fails, it should be repaired immediately, and if any jobs are transferred to other machines due to the failure, the processing time needs to be recalculated.

6. A multi-agent collaborative communication and scheduling method based on deep reinforcement learning according to claim 5, characterized in that: The multi-agent dynamic job shop scheduling in step S1 is further optimized, including two objectives, as follows: The total delay rate (TWR) is defined as formula (4), and the total delay time (ATWT) is defined as formula (5); ， (4) , (5) The specific constraints are as follows: ， (6) ， (7) ， (8) ， (9) ， (10) ， (11) ， (12) ， (13) Among them, equations (6) and (7) define DFJSP as an integrated optimization problem, equation (8) defines that each operation should only be assigned to one available machine, equation (9) defines that the job can only be processed after it arrives, equation (10) defines that the end time of the previous operation is less than the start time of the next operation, that is, the processing operations need to be performed in a fixed order, equation (11) defines the capacity constraint, equation (12) defines that each job can only be processed by one machine at the same time, and equation (13) defines that all jobs have been processed.

7. The multi-agent collaborative communication and scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The step S42, dynamic scheduling rules, further includes: When a scheduling task begins, the machine agent can interact and collaborate with the job management agent and share real-time updates and feedback. The machine agent will select a job based on its own local observations. When processing a job, the agent can still mark the next job it wants to perform through the waiting set. After the operation is completed, the machine becomes idle and prepares for the next marked job. If the marked job is not processed because the machine is busy, and another idle machine also marks the job, it will be processed by the idle machine. During this process, the agent constantly updates its own state. If the operation is the last operation of the job, it means that the job has been completed; The job management agent will obtain the job completion rate, deadline, number of operations, and pending set status of all jobs and share it with the machine agent. Through this information, the machine agent and the job management agent will cooperate to complete all jobs while pursuing the same goal.

8. The multi-agent collaborative communication and scheduling method based on deep reinforcement learning according to claim 7 is characterized in that: The machine agents share a Q network with identical parameters and an experience pool, Buffer, to store the agent's state information and actions. This structure facilitates centralized optimization and decentralized execution, while meeting the convenience requirements of decentralized agents and leveraging deep reinforcement learning to centrally optimize network parameters. Machine Agent The observed characteristics of are defined as: Machine Agent Last working time: ， (17) Machine Agent Last released time: ， (18) Machine Agent Utilization at time t: ， (19) Machine Agent Busy status: ， (20) Machine Agent Bandwidth usage: ， (21) in, Represents the maximum number of symbols that can be accommodated in the communication channel per second, is the number of agents, where is the message length, Represents the current agent and Agent communication, The sampling frequency is set to ,NJLF represents the actual total communication symbol of the agent; in, represents the number of failures of machine k; Represents the job being processed by the machine No. operations; The machine is processing the job Completion rate: ， (22) Represents the job being processed by the machine the expiry date; Indicates that the job being processed is waiting to be selected The machine number of the next operation.

Images