A novel dynamic scheduling method for flexible manufacturing system based on multi-agent reinforcement learning

Abstract

The application discloses a novel flexible manufacturing system dynamic scheduling method based on multi-agent reinforcement learning, relates to the technical field of intelligent manufacturing, and comprises the following steps: constructing a cross-domain knowledge graph driven state space, fusing process knowledge and equipment capacity to perform semantic embedding; establishing an adaptive evolution intelligent agent collaborative network of a space-time heterogeneous graph, and dynamically adjusting a topological structure according to task dependence and equipment coupling; adopting a progressive collaborative value decomposition network to optimize a strategy and accurately realize credit distribution; and constructing a multi-scale closed-loop compensation decision mechanism to form deep coupling and closed-loop feedback at the equipment layer, the production line layer and the system layer, so that the scheduling efficiency and robustness of the flexible manufacturing system are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, and in particular to a novel dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning. Background Technology

[0002] Flexible manufacturing systems are the core carriers for realizing intelligent manufacturing, characterized by diverse product types, frequent batch changes, and reconfigurable equipment. Traditional scheduling methods for flexible manufacturing systems mainly employ strategies such as mathematical programming, heuristic algorithms, and rule-based scheduling. While these methods perform well in handling scheduling problems under deterministic environments, they often struggle to respond quickly and adapt adaptively to dynamic disturbances and uncertainties. In recent years, artificial intelligence technologies, particularly reinforcement learning, have demonstrated enormous potential in the field of dynamic scheduling. Multi-agent reinforcement learning, with its distributed decision-making and collaborative optimization capabilities, has become an effective approach to solving complex scheduling problems in flexible manufacturing systems.

[0003] Existing multi-agent reinforcement learning scheduling methods still have many shortcomings. Chinese invention patent CN120103803A discloses a flexible production line adaptive scheduling control method integrating multi-agent reinforcement learning. This method constructs a state space and action space, maps equipment to agents, establishes a local communication network, and uses an experience replay pool to update the state-action value function. However, this method has the following problems: First, the state space construction lacks deep integration of process knowledge, relying only on simple mapping based on original feature vectors, failing to effectively utilize the rich process knowledge and equipment capability information in the manufacturing system, resulting in insufficient state representation. Second, the agent communication network uses a static or semi-static topology, unable to adjust the communication mode in real time according to dynamic task changes and equipment status, leading to low inter-agent collaboration efficiency. Third, the value function update uses traditional centralized or simple distributed methods, failing to effectively handle the credit allocation problem among multiple agents, making it difficult to accurately evaluate the individual contribution and collaborative gain of each agent. Fourth, the compensation scheduling mechanism lacks multi-level closed-loop coordination, making compensation decisions only at a single level, failing to form deep coupling and closed-loop feedback between the equipment layer, production line layer, and system layer, resulting in insufficient disturbance recovery capability. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a novel dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning. This method solves the technical problems of insufficient state representation, low agent collaboration efficiency, inaccurate credit allocation, and weak disturbance recovery capability in the prior art, and enables intelligent adaptive dynamic scheduling of flexible manufacturing systems.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0006] This invention provides a novel dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning, comprising:

[0007] Acquire raw target data from the flexible manufacturing system, and construct a cross-domain knowledge graph-driven state space based on the raw target data;

[0008] The manufacturing units in the equipment are mapped as intelligent agents, and the intelligent agents are used as nodes to construct an intelligent agent cooperative network with a spatiotemporal heterogeneous graph topology.

[0009] The global situational representation of the agents is obtained from the agent cooperative network, and combined with the local observation information of the agents, the agents make decisions to generate scheduling actions.

[0010] The scheduling action is executed, and the action reward is calculated based on the preset global value function to construct the corresponding experience sample. The experience sample includes the current state, the scheduling action, the action reward, the next state, and the completion flag. The current state and the next state are the state representation vector obtained from the state space and the global situation representation obtained from the agent cooperative network.

[0011] A priority experience replay mechanism is adopted to perform weighted sampling of high-value experience samples, and the policy parameters of the agent are updated using a progressive target network.

[0012] Optionally, the original target data includes equipment operating status, process constraints, and task requirement information;

[0013] The state space constructed based on the original target data and driven by the cross-domain knowledge graph includes:

[0014] Map the equipment operating status, process constraints, and task requirements information in the flexible manufacturing system into an initial feature vector;

[0015] The initial feature vector is semantically embedded based on the process knowledge base and equipment capability map. A state representation vector that integrates process knowledge is generated through a heterogeneous relation propagation algorithm. A multi-level state space is constructed based on the state representation vector.

[0016] Optionally, the step of semantically embedding the initial feature vector based on the process knowledge base and equipment capability map, and generating a state representation vector that integrates process knowledge through a heterogeneous relation propagation algorithm, includes:

[0017] Extract the process relationship types between processes from the process knowledge base and construct process relationship triples: [process A, process relationship type, process B];

[0018] Extract the capability attributes of each device and the compatibility relationships between devices from the device capability map, and construct device relationship triples: [device a (capability attribute), compatibility relationship, device b (capability attribute)]; the capability attributes include processing capability and accuracy level, and the processing capability includes the types of processes that the device can perform;

[0019] Combine the process relationship triples and the equipment relationship triples into knowledge triples, and construct a set of knowledge triples;

[0020] A knowledge graph embedding algorithm is used to embed entities and relations into the knowledge triple set. A graph convolutional network is used to perform multi-hop neighborhood aggregation on the embedding vectors to obtain entity representation vectors that integrate semantic information.

[0021] The initial feature vector and the entity representation vector are aligned across modalities. The semantic similarity between the initial feature vector and the entity representation vector is calculated through an attention mechanism. The initial feature vector is then weighted and fused based on the semantic similarity to generate a state representation vector that integrates process knowledge.

[0022] Optionally, the agent cooperative network that uses the agents as nodes to construct a spatiotemporally heterogeneous graph topology further includes:

[0023] The dynamic weights between agents are calculated based on task dependency and device coupling, and the communication edges of the spatiotemporal heterogeneous graph topology are adaptively adjusted according to the dynamic weights.

[0024] The node features and edge features in the spatiotemporal heterogeneous graph topology are subjected to bidirectional self-attention encoding to generate a global situational representation of the intelligent agent that integrates spatiotemporal information.

[0025] Optionally, the calculation of dynamic weights between agents based on task dependency and device coupling includes:

[0026] Obtain the task information currently being executed by each agent, and determine the task dependency between agents based on the sequential relationship of the procedures in the task information: if there is a direct procedure dependency between two agents, the task dependency is set to a preset high weight value; if there is an indirect procedure dependency between two agents, the task dependency is set to a preset medium weight value; if there is no procedure dependency between two agents, the task dependency is set to a preset low weight value.

[0027] Obtain the physical layout information and resource sharing information of the devices to which each intelligent agent belongs; determine the physical distance between devices based on the physical layout information; determine the preliminary device coupling degree based on the principle that the closer the devices are, the higher their coupling degree; determine the weighting factor of the device coupling degree based on the resource sharing information; and multiply the preliminary device coupling degree by the weighting factor to obtain the final device coupling degree.

[0028] The task dependency and the device coupling are nonlinearly fused to obtain the dynamic weights between agents;

[0029] The step of adaptively adjusting the communication edges of the spatiotemporal heterogeneous graph topology based on the dynamic weights includes:

[0030] Construct an adjacency matrix based on the dynamic weights: Set a dynamic weight threshold. When the dynamic weight is greater than the dynamic weight threshold, establish a communication edge between the nodes of the corresponding agent. When the dynamic weight is less than or equal to the dynamic weight threshold, delete the communication edge between the nodes of the corresponding agent.

[0031] Optionally, the bidirectional self-attention encoding of node features and edge features in the spatiotemporal heterogeneous graph topology includes:

[0032] The node features of the agent are updated according to the task execution progress. The node features include device state vector, task queue vector and performance index vector. The node features are temporally encoded to capture the temporal evolution of the node features.

[0033] The edge features are updated based on the material flow and resource competition relationships between intelligent agents. The edge features include communication delay, cooperation strength and conflict probability. The edge features are spatially encoded to capture the spatial interaction patterns between intelligent agents.

[0034] The global situational representation of the intelligent agent that generates fused spatiotemporal information includes:

[0035] The updated node features and edge features are input into the spatiotemporal graph neural network, and information aggregation between agents is realized through the message passing mechanism to generate a global situational representation of the agents that integrates spatiotemporal information.

[0036] Optionally, the global value function is the expected cumulative reward of the joint action of all agents in the current state, and its value is the sum of the local value functions of each agent plus the cooperative value function between agents.

[0037] The local value function is calculated through the independent value network of each agent. The input of the value network is the local observation information of the agent, and the output is the estimated value of the agent's scheduling actions.

[0038] The collaborative value function is calculated through a hybrid network. The input of the hybrid network consists of the local value function outputs of all agents and the global situation representation. A hybrid weight is generated through a super network. The hybrid weight is used to weight and combine the local value functions of each agent. A nonlinear collaborative term is introduced to capture the collaborative effect between agents and generate a collaborative value function between agents.

[0039] Optionally, the weighted sampling of high-value experience samples using the priority experience replay mechanism includes:

[0040] Store the experience samples generated by the interaction of the intelligent agent into the experience replay pool;

[0041] A priority weight is calculated for each experience sample. The priority weight is determined based on the absolute value of the time difference error. Experience samples with larger time difference errors have higher priority weights.

[0042] During training, empirical samples are weighted according to the priority weights, and the sampling probability is proportional to the priority weights.

[0043] The strategy parameters for updating the agent using a progressive target network include:

[0044] The policy update gradient is calculated by time difference error, and importance sampling weights are introduced to correct the policy update gradient. The importance sampling weights are inversely proportional to the sampling probability.

[0045] Optionally, it also includes continuous monitoring of the flexible manufacturing system and identification of disturbance events;

[0046] When a disturbance event is detected, a local adjustment strategy is generated at the equipment level, a global rescheduling scheme is generated at the production line level, and a resource reconfiguration instruction is generated at the system level. The local adjustment strategy, the global rescheduling scheme, and the resource reconfiguration instruction generate a comprehensive compensation decision through a multi-scale closed-loop fusion mechanism. After feasibility verification, the comprehensive compensation decision works in conjunction with the normal scheduling strategy to form a complete adaptive scheduling instruction sequence and is executed.

[0047] Optionally, the continuous monitoring of the flexible manufacturing system and identification of disturbance events includes:

[0048] Real-time sensor data from each device in the flexible manufacturing system is collected, and the real-time sensor data is preprocessed and feature extracted to generate the current status of the device.

[0049] Based on long short-term memory networks, the device state under normal operation mode is modeled in a time series, the state evolution law under normal operation mode is learned, and the deviation between the current state of the device and the state under normal operation mode is calculated.

[0050] When the deviation exceeds a preset threshold, a disturbance event detection process is triggered. The disturbance event type is identified by multi-classifier fusion. The disturbance event types include equipment failure, material shortage, process abnormality, and order change.

[0051] Based on the type and scope of the disturbance event, the level at which compensation decisions need to be made is determined, including the equipment level, production line level, and system level.

[0052] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0053] This invention provides a novel dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning. 1) By constructing a cross-domain knowledge graph-driven state space, the process knowledge base and equipment capability graph are deeply integrated into the state representation, enabling agents to make decisions based on rich semantic information. Compared to methods based solely on original features, the semantic richness of the state representation is significantly improved, and the accuracy of scheduling decisions is significantly enhanced. 2) A spatiotemporal heterogeneous graph adaptive evolution mechanism is adopted to adjust the communication topology between agents in real time according to task dependence and equipment coupling. Compared to static communication networks, communication overhead is significantly reduced, collaborative decision-making efficiency is significantly improved, and it can better adapt to the dynamic changes of the manufacturing system. 3) A progressive collaborative value decomposition network is introduced to decouple the global value function into local value functions and collaborative value functions, accurately evaluating the individual contribution and collaborative gain of each agent. This solves the credit allocation problem in multi-agent systems, significantly improves policy convergence speed, and ultimately significantly improves scheduling performance. 4) A multi-scale closed-loop compensation decision-making mechanism is constructed, forming deep coupling and closed-loop feedback at the equipment, production line, and system levels. When disturbances occur, it can quickly generate multi-level coordinated comprehensive compensation decisions, significantly shortening the disturbance recovery time and significantly enhancing the robustness and flexibility of the manufacturing system. In summary, this invention can realize intelligent adaptive dynamic scheduling of flexible manufacturing systems. Attached Figure Description

[0054] Figure 1 This is a flowchart of a dynamic scheduling method for a flexible manufacturing system based on multi-agent reinforcement learning, provided in an embodiment of the present invention.

[0055] Figure 2 This is a flowchart of constructing a cross-domain knowledge graph-driven state space provided in an embodiment of the present invention;

[0056] Figure 3 This is a flowchart illustrating the generation of a complete adaptive scheduling instruction sequence provided in this embodiment of the invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment of the invention provides a novel dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning, comprising the following steps:

[0060] Step S1: Obtain the original target data in the flexible manufacturing system and construct a cross-domain knowledge graph-driven state space based on the original target data.

[0061] like Figure 2 As shown, in this specific embodiment, the original target data includes equipment operating status, process constraint relationships, and task requirement information;

[0062] The equipment operating status includes the real-time working status, current task information, and performance parameters of each machining center, assembly station, and testing equipment. In this embodiment, the operating parameters of the equipment are collected through the Industrial Internet of Things (IIoT), including key indicators such as spindle speed, feed rate, cutting force, and temperature, with a sampling frequency of 1Hz to 10Hz.

[0063] Process constraints include the sequential dependencies, parallel executable relationships, and resource mutual exclusion relationships among the processes in the product process route. In this embodiment, process route data is extracted from the manufacturing execution system to establish a process dependency graph.

[0064] The task requirements information includes the specifications, quantity, delivery date, and priority of the products to be processed. In this embodiment, order information is obtained from the enterprise resource planning system and synchronized in real time.

[0065] The original target data is then processed to extract features and mapped into initial feature vectors. For equipment operating status, an equipment status vector is constructed, typically with 50 to 100 dimensions, including features such as equipment type code, current task identifier, remaining processing time, equipment health, and energy consumption level. For process constraints, a process feature vector is constructed, typically with 30 to 60 dimensions, including features such as process type, required equipment capacity, processing accuracy requirements, and estimated processing time. For task requirement information, a task feature vector is constructed, typically with 20 to 40 dimensions, including features such as product complexity, priority weight, and deadline margin. These feature vectors are then concatenated according to a unified dimension to form an initial feature vector, typically with a total dimension of 100 to 200.

[0066] Semantic embedding is performed on the initial feature vector based on the process knowledge base and equipment capability graph. The process knowledge base stores knowledge such as historical processing experience, process specifications, quality standards, and optimization strategies. In this embodiment, the process knowledge base is stored using a relational database or graph database, containing process entities, equipment entities, material entities, and the relationships between them. The process relationship types between processes are extracted from the process knowledge base to construct process relationship triples: [process A, process relationship type, process B]; the process relationship types are: pre- and post-process dependencies, parallel executable relationships, and resource mutual exclusion relationships.

[0067] The equipment capability map describes the processing capability spectrum, accuracy level, types of materials that can be processed, and compatibility relationships between equipment. The processing capabilities of each piece of equipment are extracted from the capability map, including the set of executable process types, the processing accuracy range of each process, and production efficiency parameters. Accuracy level information is extracted, typically categorized into high, medium, and normal accuracy levels, and quantified numerically. Compatibility relationships are extracted, describing which equipment can serve as backups for each other and which equipment needs to work collaboratively. The capability attributes of each piece of equipment and the compatibility relationships between them are extracted from the capability map, constructing equipment relationship triples: [equipment a (capability attribute), compatibility relationship, equipment b (capability attribute)]; capability attributes include processing capability and accuracy level, with processing capability including the types of processes that the equipment can execute.

[0068] The process relation triples and equipment relation triples are combined into knowledge triples, and a set of knowledge triples is constructed. A knowledge graph embedding algorithm is used to embed entities and relations into the knowledge triple set. A graph convolutional network is then used to perform multi-hop neighborhood aggregation on the embedding vectors to obtain entity representation vectors that integrate semantic information. In this embodiment, the knowledge graph embedding algorithm uses either the TransE algorithm or the RotatE algorithm for embedding learning. The TransE algorithm models relations as translation operations in the entity embedding space. For a triple (head entity, relation, tail entity), the sum of the head entity vector and the relation vector is approximately equal to the tail entity vector. The embedding vectors of entities and relations are learned by minimizing this translation error. The RotatE algorithm models relations as rotation operations in complex space, which can better express complex relation patterns such as symmetric, antisymmetric, and transitive relationships. The embedding vector dimension is preferably 64 to 128 dimensions.

[0069] Graph convolution operations update the representation of the current node by aggregating the feature information of neighboring nodes. In this embodiment, 2 to 3 graph convolutional layers are used. Each graph convolutional layer updates the node representation in the following way: for node i, all its neighboring nodes j are traversed, the feature vectors of neighboring nodes j are transformed with the relation vectors of the connecting edges, then a weighted sum is performed, and finally the updated node representation is obtained through a nonlinear activation function. The activation function is preferably ReLU or LeakyReLU. Multi-hop neighborhood aggregation enables nodes to perceive a wider range of semantic information, capturing indirect dependencies between processes and collaborative capabilities between devices.

[0070] The initial feature vector and the entity representation vector are aligned across modalities. The semantic similarity between the initial feature vector and the entity representation vector is calculated through an attention mechanism. The initial feature vector is then weighted and fused based on the semantic similarity to generate a state representation vector that incorporates process knowledge.

[0071] The purpose of cross-modal alignment is to fuse initial feature vectors based on the original target data with entity representation vectors based on knowledge graphs, so that the final state representation contains both real-time dynamic information and static knowledge information. Semantic similarity between the initial feature vector and the entity representation vector is calculated using an attention mechanism. Specifically, for each feature element in the initial feature vector, its similarity score with all relevant entity representation vectors is calculated. Similarity calculation employs a dot product or bilinear transformation method. In this implementation, a scaled dot product attention mechanism is used; the similarity score is obtained by dividing the dot product of the feature vector and the entity representation vector by the square root of the dimension, and then normalized to attention weights using a softmax function.

[0072] The entity representation vectors are weighted and summed based on attention weights to obtain a knowledge-enhanced vector for the current features. This knowledge-enhanced vector is then concatenated or additively fused with the initial feature vector to generate a state representation vector that incorporates process knowledge. In this implementation, a gated fusion mechanism is employed, dynamically controlling the fusion ratio of the initial feature vector and the knowledge-enhanced vector through learnable gating parameters. This allows the model to adaptively decide whether to rely on the original features or the knowledge information based on the current state.

[0073] Finally, a multi-level state space is constructed based on the state representation vectors. This multi-level state space includes the equipment-level state space, the production line-level state space, and the system-level state space. The equipment-level state space describes the local state of each device, typically with 50 to 100 dimensions. The production line-level state space describes the overall state of the production line or workstation, obtained by aggregating the state representation vectors of multiple devices, typically with 100 to 200 dimensions. The system-level state space describes the global state of the entire manufacturing system, obtained by aggregating the state representation vectors of all production lines and fusing system-level information (such as overall task progress, resource availability, bottleneck status, etc.), typically with 200 to 500 dimensions. This multi-level state space provides corresponding state representations for decisions at different levels, supporting multi-scale scheduling decisions.

[0074] Step S1 uses deep fusion of cross-domain knowledge graphs to explicitly embed process knowledge and equipment capabilities into the state representation. Compared with methods that rely solely on raw sensor data, this provides richer semantic information and more accurate state descriptions, providing strong knowledge support for subsequent agent decision-making.

[0075] Step S2: Map the manufacturing units in the equipment to intelligent agents, and use the intelligent agents as nodes to build an intelligent agent cooperative network with a spatiotemporal heterogeneous graph topology.

[0076] A manufacturing unit includes machining centers, assembly workstations, testing equipment, and material handling equipment. Each manufacturing unit is abstracted as an intelligent agent, possessing perception, decision-making, and execution capabilities. The agent's perception capability is reflected in its ability to acquire its own state, task information, and shared information from neighboring intelligent agents. The agent's decision-making capability is reflected in its ability to select and schedule actions based on the current state and policy network. The agent's execution capability is reflected in its ability to control its subordinate manufacturing unit to perform corresponding processing, assembly, or material handling operations.

[0077] Spatiotemporal heterogeneous graphs are a graph representation method capable of simultaneously modeling spatial structural relationships and temporal evolution patterns. In spatiotemporal heterogeneous graphs, nodes represent agents, and edges represent communication links and collaborative relationships between agents. Heterogeneity is reflected in the different type attributes of nodes and edges. Node types are categorized based on the function of the manufacturing unit, such as processing, assembly, inspection, and transportation. Edge types are categorized based on the relationships between agents, such as process-dependent edges, resource-sharing edges, physical proximity edges, and information-interaction edges.

[0078] The construction process of the spatiotemporal heterogeneous graph is as follows: Based on the physical layout and logical relationships of the manufacturing system, the node set and edge set of the graph are initialized. For any two agents, if there is a process dependency relationship between them (i.e., the output of one agent is the input of another agent), a process dependency edge is established between them. If they share certain resources (such as a shared material buffer area or a shared handling robot), a resource sharing edge is established. If they are physically adjacent (the distance is less than a preset threshold, set to 5m to 15m in this embodiment), a physical proximity edge is established. Information interaction edges are added or deleted as needed during subsequent dynamic adjustments.

[0079] Bidirectional self-attention encoding is performed on the node and edge features in the spatiotemporal heterogeneous graph topology. Node features are provided by the state representation vector obtained in step S1, and further expanded with real-time information such as the current working state of the device, task queue length, and performance indicators. The node feature dimension is preferably 64 to 128 dimensions. Edge features describe the interaction characteristics between agents, including communication latency, cooperation strength, and conflict probability. Communication latency represents the time required for information to be transmitted from one agent to another, typically tens to hundreds of milliseconds. Cooperation strength represents the degree of efficiency improvement when two agents work together, with a value ranging from 0 to 1. Conflict probability represents the likelihood of conflict between two agents when competing for resources, with a value ranging from 0 to 1. The edge feature dimension is preferably 16 to 32 dimensions.

[0080] A bidirectional self-attention encoding mechanism is used to capture the interdependencies between nodes. For each node, its attention weights to all neighboring nodes are calculated. The attention weights are obtained through the similarity of node features. In this embodiment, a multi-head attention mechanism is employed, projecting node features onto multiple subspaces, calculating attention weights independently in each subspace, and then concatenating the results of the multi-head attention. The number of multi-head attention mechanisms is preferably 4 to 8. When calculating attention weights, the influence of edge features is also considered, and edge features are fused with node features. Specifically, a gating mechanism is used to modulate edge features into the attention calculation between nodes, so that different types of edges have different effects on the attention weights.

[0081] Bidirectional self-attention encoding updates not only node features but also edge features. Node feature updates are achieved by aggregating weighted information from neighboring nodes, while edge feature updates are achieved by aggregating information from connected nodes. This bidirectional encoding mechanism allows node and edge representations to mutually reinforce each other, resulting in richer graph representations.

[0082] Dynamic weights between agents are calculated based on task dependency and device coupling, and communication edges in the spatiotemporal heterogeneous graph topology are adaptively adjusted according to these dynamic weights; specifically including:

[0083] Obtain information about the tasks currently being performed by each agent. This information includes the task type, processing stage, and remaining time. The task type specifies the current process category, such as roughing, finishing, assembly, or inspection. The processing stage indicates the task's progress, such as just started, in progress, or nearing completion. Remaining time indicates the time needed to complete the task.

[0084] Determine the task dependency between agents based on the sequential relationship of procedures in the task information:

[0085] If there is a direct process dependency between two agents (such as the output workpiece of agent x being directly passed to agent y for the next process), the task dependency is set to a preset high weight value, preferably 0.8 to 1.0.

[0086] If there is an indirect process dependency relationship between two agents (such as the output workpiece of agent x being passed to agent y after passing through other processes), the task dependency degree takes a preset medium weight value, preferably 0.4 to 0.8.

[0087] If there is no process dependency between the two agents, the task dependency is set to a preset low weight value, preferably 0.1 to 0.2.

[0088] The system acquires physical layout information and resource sharing information for the devices belonging to each intelligent agent. Physical layout information includes the three-dimensional coordinates of the devices within the workshop. Resource sharing information includes shared material buffer areas, shared material handling systems, and shared tool magazines.

[0089] The physical distance between devices is determined based on physical layout information. Based on the principle that closer devices have higher coupling, a preliminary device coupling degree is determined. In this embodiment, an exponential decay function is used to model the impact of distance on coupling degree. The basic value of device coupling degree is calculated as exp(-distance / λ), where distance is the distance between devices and λ is the decay coefficient.

[0090] The weighting factor for equipment coupling is determined based on resource sharing information. The initial equipment coupling is multiplied by the weighting factor to obtain the final equipment coupling. In this embodiment, for equipment sharing a material buffer area, the equipment coupling is multiplied by a weighting factor of 1.3 to 1.5. For equipment sharing a transportation system, the equipment coupling is multiplied by a weighting factor of 1.2 to 1.4. For equipment sharing a tool magazine or other auxiliary resources, the equipment coupling is multiplied by a weighting factor of 1.1 to 1.3.

[0091] The dynamic weights between agents are obtained by nonlinearly fusing task dependency and device coupling. The nonlinear fusion employs a weighted geometric average or a weighted harmonic average to capture the interaction effects of task dependency and device coupling. In this embodiment, the dynamic weights are calculated as follows: Here, α and β are fusion parameters, with α preferably set to 0.6 and β to 0.4, indicating that the impact of task dependency is slightly higher than that of equipment coupling. `task_dependency` and `equipment_coupling` represent task dependency and equipment coupling, respectively. Dynamic weights are updated in real time according to manufacturing progress and resource status. When a task is completed, a new task arrives, or the equipment status changes, the dynamic weights between relevant agents are recalculated.

[0092] An adjacency matrix is ​​constructed based on dynamic weights, where each element represents the connection strength between agents. A dynamic weight threshold is set, preferably between 0.3 and 0.5.

[0093] When the dynamic weight is greater than the dynamic weight threshold, a communication edge is established between the nodes of the corresponding agent (or the weight of the existing communication edge is increased). When the dynamic weight is less than or equal to the dynamic weight threshold, the communication edge between the nodes of the corresponding agent is deleted (or the weight of the existing communication edge is reduced).

[0094] This adaptive adjustment mechanism enables the communication topology between agents to change dynamically according to current task requirements and resource status, avoiding unnecessary communication overhead and improving collaboration efficiency.

[0095] Node features are updated based on task execution progress. Node features include device state vectors, task queue vectors, and performance indicator vectors. The device state vector records information such as the device's operating status, health, and energy consumption. The task queue vector records information such as the number of tasks to be processed, priority distribution, and estimated processing time. The performance indicator vector records indicators such as the device's real-time productivity, quality pass rate, and failure rate. Node features are temporally encoded to capture the temporal evolution of node states. In this embodiment, a Long Short-Term Memory (LSTM) network or a gated recurrent unit (GRU) is used to perform temporal modeling of the node feature sequence, learning the dynamic patterns of state changes and predicting future state trends.

[0096] Edge features are updated based on material flow and resource competition relationships. Edge features include communication latency, cooperation strength, and conflict probability. Communication latency is dynamically adjusted based on network load and agent processing capacity. Cooperation strength is updated based on historical cooperation results; if two agents have cooperated well in the past, the cooperation strength increases, and vice versa. Conflict probability is updated based on historical resource competition records; if two agents frequently engage in resource conflicts, the conflict probability increases. Spatial encoding is performed on edge features to capture spatial interaction patterns between agents. In this embodiment, a graph convolutional network is used to encode edge features, updating them by aggregating information from connected nodes, enabling the edge features to reflect the relative positional relationships and interaction frequencies between agents.

[0097] The updated node and edge features are input into the spatiotemporal graph neural network, and information aggregation between agents is achieved through a message passing mechanism. The spatiotemporal graph neural network consists of alternating stacked spatial graph convolutional layers and temporal convolutional layers. The spatial graph convolutional layers are responsible for aggregating the spatial information of neighboring nodes, while the temporal convolutional layers are responsible for capturing the patterns of node features changing over time. In this embodiment, 2 to 4 spatial-temporal convolutional blocks are used. In the spatial graph convolutional layers, a graph attention network mechanism is employed to weight and aggregate neighbor information according to the attention weights between nodes. In the temporal convolutional layers, one-dimensional convolution or gated convolution is used to extract temporal features.

[0098] The updated node and edge features are input into the spatiotemporal graph neural network. Information aggregation among agents is achieved through a message passing mechanism, generating a global situational representation of the agents that integrates spatiotemporal information. This global situational representation integrates the local states of each agent, the interaction relationships between agents, and the spatiotemporal evolution patterns of the entire system, providing comprehensive information support for collaborative decision-making among agents.

[0099] Step S2 uses a spatiotemporal heterogeneous graph to model the agent cooperative network, which can simultaneously capture spatial structural relationships and temporal evolution patterns, providing richer cooperative information. Through dynamic weight calculation based on task dependence and device coupling, adaptive adjustment of the network topology is achieved, enabling flexible changes in communication modes between agents according to actual needs, significantly improving cooperative efficiency.

[0100] Step S3: Obtain the global situational representation of the agent from the agent cooperative network, and combine it with the agent's local observation information to generate scheduling actions.

[0101] Local observations include the agent's own state and task information. Global situational information includes the overall system load, bottleneck locations, and resource availability. The agent's policy network takes local observations and global situational representations as input, outputs the probability distribution or Q-value of each candidate action, and selects actions to execute based on an exploration policy (such as an ε-greedy policy or a Boltzmann policy).

[0102] Step S4: Execute the scheduling action and calculate the action reward based on the preset global value function to construct the corresponding experience sample. The experience sample includes the current state, scheduling action, action reward, next state, and completion flag. The current state and next state are the state representation vector obtained from the state space and the global situation representation obtained from the agent cooperative network.

[0103] Actions are the scheduling actions chosen by the agent, such as which task to select, which device to assign it to, and when to start execution. Rewards are the reward signals calculated above. The next state is the new state the system transitions to after executing an action. The completion flag is a Boolean value indicating whether the round has ended (e.g., all tasks completed or the maximum time step reached). The experience replay pool uses a fixed-size circular buffer, preferably with a capacity of 10,000 to 100,000 samples.

[0104] Reward signal design is central to reinforcement learning. In flexible manufacturing system scheduling, the reward signal comprehensively considers multiple performance metrics. In this embodiment, the reward signal consists of the following parts:

[0105] Task completion rewards: A positive reward is given when a task is completed. The reward value is related to the task priority and the timeliness of completion.

[0106] Equipment utilization rate bonuses are awarded based on the percentage of effective working time of the equipment, encouraging increased equipment utilization.

[0107] Energy consumption penalties, which provide negative rewards based on energy consumption, encourage energy-saving scheduling.

[0108] Delay penalty: If the task is completed beyond the deadline, a significant negative reward will be given.

[0109] Waiting time penalty: A slight negative reward is given based on the waiting time of the task in the queue. Conflict penalty: A negative reward is given if resource conflicts or deadlocks occur.

[0110] The total reward is the weighted sum of the above components, with the weights adjusted according to the scheduling objectives.

[0111] In this embodiment, the global value function is the expected cumulative reward of the joint action of all agents in the current state, and its value is the sum of the local value functions of each agent plus the cooperative value function between agents.

[0112] Specifically, the global value function The decomposition form is:

[0113]

[0114] in, For the global value function, This is the global state. This is the joint action vector for all agents. For the number of agents, For the first Local value function of an agent For the first Local observation information of an agent For the first The actions of an intelligent agent The value function is the collaborative value function. Value function decomposition is a key technique for solving the multi-agent credit allocation problem.

[0115] The local value function is calculated through independent value networks for each agent. The input to the value network is the agent's local observation information, and the output is the estimated value of the agent's scheduled actions. The local value network adopts a multilayer perceptron structure, containing 2 to 3 hidden layers, with 64 to 256 neurons in each layer, and ReLU is used as the activation function. In this embodiment, a dueling architecture is used in the last layer to separate the value stream and the advantage stream. The value stream estimates the state value, and the advantage stream estimates the relative advantage of each action. Then, they are combined to obtain the action value.

[0116] The collaborative value function is calculated through a hybrid network. The input of the hybrid network consists of the local value function outputs of all agents and the global situation representation. A hybrid weight is generated through a super network. The hybrid weight is used to weight and combine the local value functions of each agent. A nonlinear collaborative term is introduced to capture the collaborative effect between agents and generate the collaborative value function between agents.

[0117] The input to the hybrid network consists of the local value functions of all agents and the global situational representation, with the output being the collaborative value. The design of the hybrid network is crucial and must satisfy the monotonicity constraint, meaning the global value function must monotonically increase with respect to each local value function. This ensures that, during distributed execution, each agent independently choosing the optimal action is equivalent to jointly choosing the globally optimal action.

[0118] Hybrid weights are generated through a supernetwork, with the global situational awareness as input and the weight parameters of the hybrid network as output. In this embodiment, the hybrid weights generated by the supernetwork are ensured to be non-negative and satisfy monotonicity constraints through an absolute value activation function (such as absolute or softplus). The hybrid weights are a weighted combination of the local value functions of each agent.

[0119] Furthermore, a nonlinear cooperative term is introduced to capture the cooperative effects between agents. This nonlinear cooperative term is modeled using a neural network, with inputs being local value vectors and the global state, and outputting a correction term for the cooperative value. In this embodiment, the cooperative term network contains one or two hidden layers, using tanh or sigmoid activation functions. The nonlinear cooperative term can express complex interaction patterns between agents, such as mutual promotion and mutual inhibition.

[0120] The final form of the collaborative value function is:

[0121]

[0122] in, and For the hypernet based on the global state The generated mixed weight matrix and bias vector, The activation function is a nonlinear activation function for the hybrid network, preferably ELU or ReLU. This is a nonlinear cooperative term network. The formula shows that the cooperative value function consists of two parts: the first part is a weighted linear combination of local values, with weights generated by the supernetwork; the second part is a nonlinear cooperative term that captures higher-order interactions between agents.

[0123] Step S5: Use a priority experience replay mechanism to perform weighted sampling of high-value experience samples, and use a progressive target network to update the agent's policy parameters.

[0124] Target networks are a common technique in reinforcement learning used to provide stable training objectives. Progressive target networks are an improvement on traditional target networks, where parameters are moved at a slow rate to the current value network parameters, rather than through periodic hard updates. In this implementation, the target network parameters... The update method is as follows:

[0125]

[0126] In the formula, For the current value network parameters, For the update rate, a value between 0.001 and 0.01 is preferred. Smaller values ​​are also preferable. This value makes the target network updates smoother and helps with training stability.

[0127] The gradient is updated using a strategy calculated through temporal difference error. Temporal difference error reflects the gap between the current value estimate and the target value. For the transition sample... , These are the current state, the scheduled action, the action reward, and the next state, respectively. The time difference error is defined as follows: :

[0128]

[0129] In the formula, The discount factor is preferably 0.95 to 0.99. The maximum value for the next state is estimated by the target network. For the next scheduling action, The current value is estimated by the current network. Policy update objective. To minimize the square of the time difference error:

[0130]

[0131] in, For experience replay pool, For expectations;

[0132] Update network parameters using gradient descent algorithm:

[0133]

[0134] in, The learning rate is preferably between 0.0001 and 0.001. This is the descent gradient.

[0135] A priority experience replay mechanism is employed to perform weighted sampling of high-value experience samples. Priority experience replay prioritizes samples based on their learning value, thus improving sample utilization efficiency and learning speed.

[0136] A priority weight is calculated for each empirical sample. The priority weight is determined based on the absolute value of the temporal difference error. Samples with a large temporal difference error indicate that the current network's estimation of the sample's value is biased, thus they have high learning value and should be sampled first. In this implementation, the priority weight... The calculation is as follows:

[0137]

[0138] In the formula, This represents the absolute value of the time difference error. Use a small constant (preferably 0.01) to avoid a priority of 0. The priority index parameter (preferably 0.6 to 0.8) controls the degree of prioritization. Corresponding to uniform sampling, This corresponds to fully prioritized sampling.

[0139] During training, empirical samples are weighted according to priority weights. Sampling probability. for:

[0140]

[0141] in, For the first The priority weight of each sample, The sum of priority weights for all samples, where k is the number of samples.

[0142] Importance sampling weights are introduced to correct gradient updates. Since the sampling distribution deviates from the empirical true distribution, it introduces bias, which needs to be corrected using importance sampling weights. Importance sampling weights The calculation is as follows:

[0143]

[0144] in, For the size of the experience replay pool, The importance sampling index (preferably 0.4 to 1.0) is used, taking a smaller value in the early stages of training and gradually increasing to 1.0 as training progresses. During gradient updates, the loss function is multiplied by the importance sampling weight:

[0145]

[0146] Prioritizing experience replay allows high-value samples to be used more frequently for training, accelerating policy convergence. This is the set of samples for weighted sampling.

[0147] Steps S4 and S5, through a progressive collaborative value decomposition network, decouple the global value function into local and collaborative value functions, achieving accurate credit allocation and solving the problem of difficulty in evaluating the contributions of each agent in a multi-agent system. Combined with a priority experience replay mechanism, this further improves learning efficiency and policy quality.

[0148] like Figure 3As shown, based on the above-mentioned dynamic scheduling of flexible manufacturing systems through multi-agent systems, the dynamic scheduling method also includes continuous monitoring of the flexible manufacturing system and identification of disturbance events.

[0149] When a disturbance event is detected, a local adjustment strategy is generated at the equipment level, a global rescheduling scheme is generated at the production line level, and a resource reconfiguration instruction is generated at the system level. The local adjustment strategy, the global rescheduling scheme, and the resource reconfiguration instruction generate a comprehensive compensation decision through a multi-scale closed-loop fusion mechanism. After the comprehensive compensation decision is verified for feasibility, it works in conjunction with the normal scheduling strategy to form a complete adaptive scheduling instruction sequence and is executed.

[0150] Specifically, continuous monitoring of flexible manufacturing systems and identification of disturbance events includes:

[0151] Real-time sensor data from each device in the flexible manufacturing system is collected, preprocessed, and feature extracted to generate the current status of the equipment. This real-time sensor data includes operating parameters, health status, and fault signals. Operating parameters include physical quantities such as spindle speed, feed rate, cutting force, vibration, temperature, and current. The sampling frequency is preferably 1Hz to 10Hz, and can be increased to 100Hz for critical equipment. Health status is assessed by the equipment health management system, which comprehensively judges the equipment's health level based on historical data and current operating parameters. The value ranges from 0 to 1; a health level below 0.7 requires attention. Fault signals come from the equipment's fault diagnosis module; a fault alarm is triggered when an abnormal pattern is detected.

[0152] Real-time sensor data undergoes preprocessing and feature extraction. Preprocessing includes denoising, normalization, and missing value imputation. A sliding window method is used to segment the time series data, with a window size preferably between 30 and 300 seconds. Statistical features are extracted for each time window, including mean, variance, maximum, minimum, peak-to-peak value, skewness, and kurtosis. Time-frequency domain features are extracted by using Fast Fourier Transform or Wavelet Transform to convert the signal from the time domain to the frequency domain, extracting spectral features such as dominant frequency and energy distribution. The original signal and feature vectors are concatenated to form a comprehensive state feature vector, with a preferred dimension of 50 to 200.

[0153] This study uses a Long Short-Term Memory (LSTM) network to perform time-series modeling of equipment states under normal operating conditions. It learns the state evolution patterns under normal operating conditions and calculates the deviation between the current state and the state under normal operating conditions. Historical normal operating data is used as the training set to train the LSTM network model. The network structure consists of 2 to 3 LSTM layers, with 64 to 256 hidden units per layer. The input is the sequence of state features within a time window, and the output is the predicted state for the next time step. The training objective is to minimize the mean squared error between the predicted and actual states.

[0154] Calculate the deviation between the current state and the normal pattern. The deviation is defined as the distance between the current actual state and the normal state predicted by the Long Short-Term Memory (LSTM) network. Preferably, Mahalanobis distance or Euclidean distance is used to calculate the deviation. Mahalanobis distance considers the correlation between features and is defined as:

[0155]

[0156] in, Let this be the current state vector. This is the predicted normal state vector. This is the covariance matrix. The greater the deviation, the further the current state deviates from the normal operating mode, and the higher the probability of an anomaly.

[0157] When the deviation exceeds a preset threshold, a disturbance event detection process is triggered. The preset threshold is determined based on historical data statistics, preferably 3 to 5 times the standard deviation of the normal deviation. After detection is triggered, the disturbance event type is identified through multi-classifier fusion. Multi-classifier fusion combines the advantages of multiple classification algorithms to improve recognition accuracy. Preferably, support vector machines, random forests, and neural network classifiers are used, and the outputs of each classifier are fused through voting or weighted averaging. Disturbance event types include equipment failure, material shortage, process abnormality, and order change. Equipment failure refers to mechanical, electrical, or control system failures that prevent normal operation. Material shortage refers to insufficient inventory of raw materials or work-in-process to meet production needs. Process abnormality refers to quality problems during processing, such as dimensional deviations or surface defects. Order change refers to customers temporarily modifying orders or adding urgent orders.

[0158] Based on the type and scope of the disturbance event, the level at which compensation decisions are required is determined. The scope of the disturbance event is assessed using an impact propagation model. This model analyzes how the disturbance propagates from its origin to other manufacturing units based on process dependencies and resource sharing relationships. Preferably, a Bayesian network or Markov chain model is used for impact propagation modeling. The scope of impact is categorized into three levels: local impact, regional impact, and global impact. Local impact involves only a single device or workstation and is suitable for handling at the device level. Regional impact involves a production line or multiple related workstations and is suitable for handling at the production line level. Global impact involves multiple production lines or the entire manufacturing system and requires handling at the system level.

[0159] Local adjustment strategies are generated at the device level. Based on the state information of local agents, these strategies adjust parameters and optimize operations for affected devices.

[0160] Equipment-level adjustment strategies include process parameter adjustment, machining speed adjustment, and mold change operations. Process parameter adjustment refers to modifying cutting parameters, feed rate, spindle speed, etc., to compensate for equipment performance degradation or adapt to process changes. Preferably, a model-based parameter optimization method is used to calculate the optimal parameter settings based on the current equipment status and task requirements. Machining speed adjustment refers to increasing or decreasing the machining speed while ensuring quality, in order to catch up with or buffer production schedules. Mold change operations refer to replacing machining dies, fixtures, or cutting tools to adapt to different machining tasks or restore machining capacity.

[0161] A global rescheduling scheme is generated at the production line level. Based on the agent collaborative state of the affected area, the global rescheduling scheme adjusts task allocation and execution order. The production line-level rescheduling scheme is determined by solving a local optimization problem. Preferably, the rescheduling problem is modeled as a constraint satisfaction problem or a mixed-integer programming problem, with the objective of minimizing task completion time or maximizing equipment utilization. Constraints include process sequence constraints, equipment capacity constraints, and time window constraints. Heuristic or metaheuristic algorithms are used to solve the optimization problem. Preferably, genetic algorithms, simulated annealing algorithms, or particle swarm optimization algorithms are used for fast solution. The rescheduling scheme determines task redistribution, transferring tasks originally assigned to faulty equipment to standby or idle equipment. It also determines the execution order adjustment, rearranging task priorities and execution order, prioritizing urgent tasks or tasks on the critical path.

[0162] Resource reconfiguration instructions are generated at the system level. These instructions, based on global resource availability and task priority, perform system-level resource scheduling and organizational adjustments. System-level resource reconfiguration instructions include equipment reorganization, personnel allocation, and emergency material procurement. Equipment reorganization refers to temporarily changing the grouping of equipment or production line configuration, combining equipment from different production lines to form new production capacity. Preferably, a graph matching algorithm is used to find the optimal equipment combination scheme based on the equipment capacity map and current task requirements. Personnel allocation refers to adjusting the work arrangements of operators or maintenance personnel, allocating human resources to the most needed positions. Emergency material procurement refers to initiating an emergency procurement process to accelerate material supply when materials are scarce and no inventory is available.

[0163] Establish a multi-scale closed-loop fusion mechanism. This mechanism employs a hierarchical coordination strategy to achieve coordinated decision-making at the equipment, production line, and system levels. Lower-level adjustment strategies are fed back to higher levels regarding their feasibility and expected effects. After a local adjustment strategy at the equipment level is implemented, its effectiveness is evaluated, including whether it successfully alleviated the local problem and its impact on surrounding equipment. The evaluation results are then fed back to the production line level. The production line level's rescheduling plan considers the feedback from the equipment level to determine whether the local adjustment is sufficient and whether further global adjustments are needed. After the production line level plan is implemented, its effects are also fed back to the system level.

[0164] Upper-level solutions impose constraints and optimization goals on lower-level solutions. Based on global resource constraints and strategic objectives, the system layer imposes constraints on the production line layer, such as a production line must complete a specific task within a specific time, or certain resources cannot be overused. The production line layer must adhere to these constraints when formulating rescheduling plans. Similarly, the production line layer imposes constraints and optimization goals on the equipment layer, guiding local adjustments at the equipment level.

[0165] A globally consistent comprehensive compensation decision is generated through multiple rounds of iterative negotiation. Initially, each level independently generates a preliminary plan, followed by information exchange and plan negotiation. If conflicts or inconsistencies are found, each level adjusts its plan and exchanges information again. The iterative negotiation process continues until the plans at each level reach consensus or the maximum number of iterations is reached (preferably 3 to 5). The final comprehensive compensation decision forms a closed loop at the equipment level, production line level, and system level, with decisions at each level supporting and constraining each other, achieving deep coupling.

[0166] Feasibility verification is performed on the comprehensive compensation decision. Constraint checks include equipment capacity constraints, process flow constraints, time window constraints, and resource availability constraints. Equipment capacity constraints check whether the adjusted task allocation exceeds the equipment's processing capacity. Process flow constraints check whether the task execution sequence violates the process specifications. Time window constraints check whether the tasks can be completed within the specified time window. Resource availability constraints check whether the required resources (such as materials, tools, and personnel) are available. If the comprehensive compensation decision meets all constraints, the feasibility verification is passed; otherwise, the plan needs to be adjusted and re-verified.

[0167] If the comprehensive compensation decision passes feasibility verification, a performance evaluation will be conducted. The expected completion time, resource utilization rate, and cost indicators of the comprehensive compensation decision will be calculated. The expected completion time will be estimated using the critical path method or simulation. Resource utilization rate will be calculated as the percentage of effective working time for each piece of equipment and personnel. Cost indicators will include additional costs caused by disturbances, such as overtime pay, emergency procurement costs, and delay penalties. The comprehensive compensation decision will be compared with the normal scheduling strategy to assess the necessity and benefits of adopting the compensation decision.

[0168] Based on the disturbance recovery level and system performance indicators, the fusion weights of the comprehensive compensation decision and the normal scheduling strategy are dynamically calculated. When a disturbance first occurs, the system deviates significantly from its normal state, and the comprehensive compensation decision weight is high, preferably between 0.7 and 0.9, to quickly respond to the disturbance. As compensation measures are implemented and the system gradually returns to stability, the comprehensive compensation decision weight gradually decreases, while the normal scheduling strategy weight gradually increases. Preferably, the fusion weights are updated using an exponential decay method.

[0169]

[0170]

[0171] in, for The weight of compensation decision at any given moment. For normal scheduling strategy weights, The attenuation coefficient is preferably 0.01 to 0.1. This represents the time following the disturbance. Dynamic adjustments to the weights ensure a smooth transition from disturbance response to normal operation.

[0172] The comprehensive compensation decision and normal scheduling strategy are weighted and fused based on fusion weights. The fusion method involves weighted combination of the scheduling instructions output by the two strategies. For discrete scheduling decisions (such as task allocation), probabilistic fusion is used, selecting which strategy to adopt based on the fusion weights as probabilities. For continuous scheduling parameters (such as processing speed and start time), a weighted average is used for fusion. This generates a smoothly transitioning sequence of scheduling instructions, ensuring the manufacturing system smoothly recovers from a disturbance state to normal operation, avoiding drastic adjustments that could lead to new disturbances or resource waste.

[0173] The innovation of this step lies in the construction of a multi-scale closed-loop compensation decision-making mechanism, forming deep coupling and closed-loop feedback at the equipment, production line, and system levels. Decisions at each level are not isolated but coordinated through information exchange, constraint issuance, and effect feedback, forming a complete closed-loop control system. This multi-scale closed-loop mechanism significantly enhances the manufacturing system's ability to cope with disturbances, achieving rapid response and efficient recovery.

[0174] To further enhance the system's adaptability and long-term performance, this invention also includes a knowledge closed-loop feedback and a system autonomous evolution mechanism.

[0175] During the scheduling and execution process, execution feedback information from each agent is collected in real time. This feedback includes actual completion time, quality indicators, and anomaly events. The comparison between actual and estimated completion time reflects the accuracy of the scheduling decision. Quality indicators, such as product pass rate, rework rate, and scrap rate, reflect the impact of scheduling on product quality. Anomaly event records include details of the disturbances that occurred, how they were handled, and their effects, providing empirical data for subsequent optimization.

[0176] The process knowledge base and equipment capability graph are updated based on execution feedback information. The knowledge graph embedding model is optimized using incremental learning methods. Incremental learning allows for continuous learning of new knowledge without forgetting existing knowledge. When new process relationships or equipment capability changes are discovered, they are added to the knowledge graph, and the embedding model is retrained. Preferably, techniques such as elastic weight consolidation or experience replay are used to prevent catastrophic forgetting. Through continuous updates to the knowledge graph, the semantic representation capability of the state space is continuously enhanced.

[0177] The parameters of the progressive collaborative value decomposition network are updated based on execution feedback information. The scheduling strategy is continuously optimized through online learning. Online learning allows the model to continuously receive new data and update parameters during actual operation. Each time a scheduling round is executed, the experience samples generated during execution are added to the experience replay pool, and the policy network is periodically updated in small batches. Preferably, the learning rate for online learning is set to 0.1 to 0.5 times that of offline training to maintain policy stability. Through online learning, the policy network can adapt to slow changes in the manufacturing environment and new task patterns.

[0178] A knowledge-loop feedback mechanism is constructed, propagating new knowledge extracted from execution feedback back to the state space construction process. When certain state features are found to be highly correlated with scheduling performance but underutilized, the feature extraction method is adjusted to enhance their weights. When certain knowledge relationships are found to be frequently used in actual scheduling, their weights in the knowledge graph are increased. When certain collaborative patterns among agents are found to be highly effective, they are solidified as prior knowledge in the collaborative value function. Through this knowledge-loop feedback, the scheduling system achieves a complete closed loop from execution to learning to optimization, possessing autonomous evolution capabilities.

[0179] Experimental results show that, through knowledge loop feedback and continuous online learning, the system's scheduling performance can still improve by 5% to 10% after running 1000 rounds, fully demonstrating its autonomous evolution capability.

[0180] To better illustrate the present invention, a specific embodiment is given below.

[0181] A certain automotive parts manufacturing workshop includes 15 machining centers, 8 assembly workstations, and 5 testing devices, producing various specifications of engine blocks and crankshafts. The workshop employs a flexible manufacturing system, allowing for equipment reconfigurability and dynamic task scheduling. The workshop faces challenges such as a wide variety of orders, small batch sizes, and tight delivery deadlines, while also frequently encountering disruptions such as equipment failures and unstable material supplies.

[0182] The method of this invention is applied for dynamic scheduling. First, a process knowledge base is constructed, recording the machining process routes for the cylinder block and crankshaft, including roughing, semi-finishing, finishing, deburring, cleaning, and inspection, as well as their sequential relationships. Then, an equipment capability map is constructed, marking the types of processes each machining center can perform, the machining accuracy level (e.g., CNC lathes are suitable for roughing rotating parts, while high-precision machining centers are suitable for finishing box-type parts), and the substitution relationships between equipment (e.g., two machine tools of the same model can serve as backups for each other).

[0183] During the state space construction phase, real-time operating data of each device is collected, including parameters such as spindle speed (1500 rpm to 3000 rpm), feed rate (100 mm / min to 800 mm / min), cutting force (200 N to 1500 N), spindle temperature (30℃ to 70℃), and vibration (0.5 mm / s to 3 mm / s). Order information for currently pending cylinder blocks and crankshafts is obtained, including model, quantity (batch size 5 to 50 pieces), delivery time (1 to 5 days), and priority (normal or urgent). This raw data is mapped into an initial feature vector with 150 dimensions.

[0184] Semantic embedding is performed based on a process knowledge base and an equipment capability graph. The process knowledge base contains over 200 process relationship triples, such as (roughing, must_before, semi-finishing) and (deburring, can_parallel, cleaning). The equipment capability graph contains over 100 equipment capability triples, such as (machining center A, can_execute, finishing) and (lathe B, precision_level, normal precision). The TransE algorithm is used for embedding learning, with an embedding dimension of 64. Neighborhood aggregation is performed through a two-layer graph convolutional network to obtain entity representation vectors that fuse semantic information. A scaled dot product attention mechanism is used to align and fuse the initial feature vector with the entity representation vector, generating a 150-dimensional state representation vector. A multi-level state space is constructed, including an equipment layer (15 machining centers + 8 assembly stations + 5 inspection devices = 28 equipment-level states), a production line layer (3 production line-level states), and a system layer (1 system-level state).

[0185] In the construction phase of the intelligent agent collaborative network, 28 manufacturing units are mapped to 28 intelligent agent nodes. A spatiotemporal heterogeneous graph is constructed based on the physical layout and process relationships. The node types include processing (15 nodes), assembly (8 nodes), and inspection (5 nodes). The edge types include process-dependent edges (established according to the process route, approximately 40 edges), resource-sharing edges (such as shared material buffers, approximately 15 edges), and physical proximity edges (established between equipment with a distance of less than 10m, approximately 30 edges).

[0186] Calculate the dynamic weights between agents. For agent pairs executing direct preceding and following processes, the task dependency is set to 0.9. For agent pairs sharing the same material buffer, the device coupling is multiplied by an additional 1.4. The formula `task_dependency` is used. 0.6 ×equipment_coupling 0.4 Calculate dynamic weights. Set the dynamic weight threshold to 0.4. Establish communication edges between agent pairs with dynamic weights greater than 0.4 to form a sparse communication network. The number of communication edges is approximately 50 to 80, which is dynamically adjusted according to the task.

[0187] A bidirectional self-attention encoding method is applied to the spatiotemporal heterogeneous graph, employing a 4-head attention mechanism, with each attention head having a dimension of 32. Information is aggregated through a spatiotemporal graph neural network (containing 3 spatial-temporal convolutional blocks) to generate a global situational representation that integrates spatiotemporal information, with a dimension of 256. Each agent uses the concatenation (320 dimensions) of its local observations (64 dimensions) and global situational representation (256 dimensions) as input to the policy network, outputting the probability distribution of each candidate action.

[0188] During the strategy optimization phase, a comprehensive reward function is designed. The task completion reward is +100 (normal task) or +200 (urgent task). The equipment utilization reward is utilization_rate × 50, where utilization_rate is a value between 0 and 1. The energy consumption penalty is -energy_consumption × 0.01, where energy_consumption is in kilowatt-hours. The delay penalty is -500 × delay_days, where delay_days is the number of days of delay. The waiting time penalty is -wait_time × 0.1, where wait_time is in minutes. The conflict penalty is -100.

[0189] A progressive collaborative value decomposition network is employed. The local value network for each agent is a 3-layer MLP with 128-128-64 hidden layer neurons, using a dueling architecture. The hybrid network generates hybrid weights through a supernetwork, a 2-layer MLP, with the global state (256-dimensional) as input and the hybrid weights (28-dimensional) of the 28 agents as output, ensuring non-negativity through an absolute value activation function. The nonlinear collaborative term network is a 2-layer MLP with 64-32 hidden layer neurons and a tanh activation function.

[0190] Prioritized experience replay is employed, with a replay pool capacity of 50,000 samples. The priority index α_priority is set to 0.7. The importance sampling index β increases linearly from 0.4 to 1.0. The learning rate α is set to 0.0005, and the discount factor γ is set to 0.97. The progressive target network update rate τ is set to 0.005. Policy updates are performed every 10 time steps, with a batch size of 256.

[0191] During the multi-scale closed-loop compensation decision-making phase, equipment status was continuously monitored. At the 200th time step of scheduling execution, the vibration of machining center 3 suddenly jumped from the normal 1.2 mm / s to 4.5 mm / s, and the temperature rose from 45℃ to 82℃. The deviation calculated by the Long Short-Term Memory network was 5.8, exceeding the preset threshold of 3.0, triggering disturbance event detection. Multi-classifier fusion (SVM+RF+NN voting) identified it as an equipment fault, specifically spindle bearing wear, with an estimated repair time of 2 hours. The impact propagation model assessed the impact as regional, involving 3 machines and 5 pending processing tasks on production line 1.

[0192] At the equipment level, a local adjustment strategy is generated: the current task (finish machining of cylinder block A) of machining center 3 is paused, and an emergency maintenance process is initiated, with resumption expected in 2 hours. At the production line level, a global rescheduling plan is generated: the finishing task of cylinder block A is reassigned to the backup machining center 7 (the equipment capability map shows that machining center 7 has the same processing capability), the task priority is adjusted, and the subsequent tasks of the emergency order are brought forward. At the system level, a resource reconfiguration instruction is generated: a maintenance engineer is dispatched to machining center 3 immediately to retrieve a spare bearing from the spare parts warehouse.

[0193] A multi-scale closed-loop fusion mechanism was used for three rounds of iterative negotiation. In the first round, the equipment layer reported that local adjustments were feasible, but required the production line layer to cooperate in task transfer. The production line layer reported that the rescheduling plan was feasible, but the system layer needed to confirm the availability of backup equipment. In the second round, the system layer confirmed that machining center 7 was idle and had matching capacity, approved the rescheduling plan, and issued a constraint: urgent orders for production line 1 must be completed before the original delivery date. The production line layer adjusted task priorities according to the constraint, further optimizing the execution order. In the third round, all levels reached a consensus and generated a comprehensive compensation decision.

[0194] Feasibility verification passed. Performance evaluation showed that after adopting the compensation decision, the expected completion time increased by 0.5 hours (only 20% of the uncompensated case), equipment utilization remained at 82%, and the additional cost was 500 yuan for maintenance. The initial fusion weight was set at 0.85 for compensation decision and 0.15 for normal scheduling. As maintenance was completed and the system recovered, the fusion weight decreased at a rate of λ=0.05. After 40 time steps (approximately 2 hours), the fusion weight was adjusted to 0.30 for compensation decision and 0.70 for normal scheduling, and the system basically returned to normal operation mode.

[0195] In a 100-round test including 10 equipment failures, 8 material shortages, and 15 order changes, the average completion time using the method of this invention was 4.3 days, 23% shorter than the traditional rule-based scheduling method (5.6 days) and 16% shorter than the MARL method without multi-scale closed-loop compensation (5.1 days). Equipment utilization was 85%, 31% higher than the traditional method (65%). The average disturbance recovery time was 1.8 hours, 45% faster than the traditional method (3.3 hours). The on-time order delivery rate was 94%, 16 percentage points higher than the traditional method (78%).

[0196] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0200] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A novel dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning, characterized in that, include: Acquire raw target data from the flexible manufacturing system, and construct a cross-domain knowledge graph-driven state space based on the raw target data; The manufacturing units in the equipment are mapped as intelligent agents, and the intelligent agents are used as nodes to construct an intelligent agent cooperative network with a spatiotemporal heterogeneous graph topology. The global situational representation of the agents is obtained from the agent cooperative network, and combined with the local observation information of the agents, the agents make decisions to generate scheduling actions. The scheduling action is executed, and the action reward is calculated based on the preset global value function to construct the corresponding experience sample. The experience sample includes the current state, the scheduling action, the action reward, the next state, and the completion flag. The current state and the next state are the state representation vector obtained from the state space and the global situation representation obtained from the agent cooperative network. A priority experience replay mechanism is adopted to perform weighted sampling of high-value experience samples, and the policy parameters of the agent are updated using a progressive target network.

2. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 1, characterized in that, The original target data includes equipment operating status, process constraints, and task requirement information; The state space driven by the cross-domain knowledge graph constructed based on the original target data includes: Map the equipment operating status, process constraints, and task requirements information in the flexible manufacturing system into an initial feature vector; The initial feature vector is semantically embedded based on the process knowledge base and equipment capability map. A state representation vector that integrates process knowledge is generated through a heterogeneous relation propagation algorithm. A multi-level state space is constructed based on the state representation vector.

3. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 2, characterized in that, The step of semantically embedding the initial feature vector based on the process knowledge base and equipment capability map, and generating a state representation vector that integrates process knowledge through a heterogeneous relation propagation algorithm includes: Extract the process relationship types between processes from the process knowledge base and construct process relationship triplet: process A, process relationship type, process B; The capability attributes of each device and the compatibility relationships between devices are extracted from the device capability map to construct device relationship triples: device a and capability attributes, compatibility relationships, device b and capability attributes; the capability attributes include processing capability and accuracy level, and the processing capability includes the types of processes that the device can perform; Combine the process relationship triples and the equipment relationship triples into knowledge triples, and construct a set of knowledge triples; A knowledge graph embedding algorithm is used to embed entities and relations into the knowledge triple set. A graph convolutional network is used to perform multi-hop neighborhood aggregation on the embedding vectors to obtain entity representation vectors that integrate semantic information. The initial feature vector and the entity representation vector are aligned across modalities. The semantic similarity between the initial feature vector and the entity representation vector is calculated through an attention mechanism. The initial feature vector is then weighted and fused based on the semantic similarity to generate a state representation vector that integrates process knowledge.

4. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 1, characterized in that, The agent cooperative network that uses the agents as nodes to construct a spatiotemporally heterogeneous graph topology also includes: The dynamic weights between agents are calculated based on task dependency and device coupling, and the communication edges of the spatiotemporal heterogeneous graph topology are adaptively adjusted according to the dynamic weights. The node features and edge features in the spatiotemporal heterogeneous graph topology are subjected to bidirectional self-attention encoding to generate a global situational representation of the intelligent agent that integrates spatiotemporal information.

5. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 4, characterized in that, The calculation of dynamic weights between agents based on task dependency and device coupling includes: Obtain the task information currently being executed by each agent, and determine the task dependency between agents based on the sequential relationship of the procedures in the task information: if there is a direct procedure dependency between two agents, the task dependency is set to a preset high weight value; if there is an indirect procedure dependency between two agents, the task dependency is set to a preset medium weight value; if there is no procedure dependency between two agents, the task dependency is set to a preset low weight value. Obtain the physical layout information and resource sharing information of the devices to which each intelligent agent belongs; determine the physical distance between devices based on the physical layout information; determine the preliminary device coupling degree based on the principle that the closer the devices are, the higher their coupling degree; determine the weighting factor of the device coupling degree based on the resource sharing information; and multiply the preliminary device coupling degree by the weighting factor to obtain the final device coupling degree. The task dependency and the device coupling are nonlinearly fused to obtain the dynamic weights between agents; The step of adaptively adjusting the communication edges of the spatiotemporal heterogeneous graph topology based on the dynamic weights includes: Construct an adjacency matrix based on the dynamic weights: Set a dynamic weight threshold. When the dynamic weight is greater than the dynamic weight threshold, establish a communication edge between the nodes of the corresponding agent. When the dynamic weight is less than or equal to the dynamic weight threshold, delete the communication edge between the nodes of the corresponding agent.

6. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 4, characterized in that, The bidirectional self-attention encoding of node features and edge features in the spatiotemporal heterogeneous graph topology includes: The node features of the agent are updated according to the task execution progress. The node features include device state vector, task queue vector and performance index vector. The node features are temporally encoded to capture the temporal evolution of the node features. The edge features are updated based on the material flow and resource competition relationships between intelligent agents. The edge features include communication delay, cooperation strength and conflict probability. The edge features are spatially encoded to capture the spatial interaction patterns between intelligent agents. The global situational representation of the intelligent agent that generates fused spatiotemporal information includes: The updated node features and edge features are input into the spatiotemporal graph neural network, and information aggregation between agents is realized through the message passing mechanism to generate a global situational representation of the agents that integrates spatiotemporal information.

7. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 1, characterized in that, The global value function is the expected cumulative reward of the joint action of all agents in the current state, and its value is the sum of the local value functions of each agent plus the cooperative value function between agents. The local value function is calculated through the independent value network of each agent. The input of the value network is the local observation information of the agent, and the output is the estimated value of the agent's scheduling action. The collaborative value function is calculated through a hybrid network. The input of the hybrid network consists of the local value function outputs of all agents and the global situation representation. A hybrid weight is generated through a super network. The hybrid weight is used to weight and combine the local value functions of each agent. A nonlinear collaborative term is introduced to capture the collaborative effect between agents and generate a collaborative value function between agents.

8. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 1, characterized in that, The weighted sampling of high-value experience samples using the priority experience replay mechanism includes: Store the experience samples generated by the interaction of the intelligent agent into the experience replay pool; A priority weight is calculated for each experience sample. The priority weight is determined based on the absolute value of the time difference error. Experience samples with larger time difference errors have higher priority weights. During training, empirical samples are weighted according to the priority weights, and the sampling probability is proportional to the priority weights. The strategy parameters for updating the agent using a progressive target network include: The policy update gradient is calculated by time difference error, and importance sampling weights are introduced to correct the policy update gradient. The importance sampling weights are inversely proportional to the sampling probability.

9. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 1, characterized in that, This also includes continuous monitoring of flexible manufacturing systems and identification of disturbance events; When a disturbance event is detected, a local adjustment strategy is generated at the equipment level, a global rescheduling scheme is generated at the production line level, and a resource reconfiguration instruction is generated at the system level. The local adjustment strategy, the global rescheduling scheme, and the resource reconfiguration instruction generate a comprehensive compensation decision through a multi-scale closed-loop fusion mechanism. After feasibility verification, the comprehensive compensation decision works in conjunction with the normal scheduling strategy to form a complete adaptive scheduling instruction sequence and is executed.

10. The dynamic scheduling method for flexible manufacturing systems based on multi-agent reinforcement learning according to claim 9, characterized in that, The continuous monitoring of the flexible manufacturing system and identification of disturbance events includes: Real-time sensor data from each device in the flexible manufacturing system is collected, and the real-time sensor data is preprocessed and feature extracted to generate the current status of the device. Based on long short-term memory networks, the device state under normal operation mode is modeled in a time series, the state evolution law under normal operation mode is learned, and the deviation between the current state of the device and the state under normal operation mode is calculated. When the deviation exceeds a preset threshold, a disturbance event detection process is triggered. The disturbance event type is identified by multi-classifier fusion. The disturbance event types include equipment failure, material shortage, process abnormality, and order change. Based on the type and scope of the disturbance event, the level at which compensation decisions need to be made is determined, including the equipment level, production line level, and system level.

Images