Multi-agent manufacturing system dynamic optimization method based on generative model

Abstract

The application is a multi-agent manufacturing system dynamic optimization method based on a generative model. First, a device agent constructor is constructed by integrating a cloud-side large model and an edge lightweight model. The protocol and process documents of machines and other devices are analyzed to generate device agents containing an adaptation layer, a communication layer, and an intelligent decision layer, realizing the rapid access and unified semantic expression of heterogeneous devices. Second, the relationship Transformer and the gated hybrid expert mechanism are used to model the workshop state, extract the local and global context features of each agent, and form a shared environment semantic representation. Under the centralized training and distributed execution framework, an improved multi-agent PPO algorithm is introduced to jointly optimize the workpiece allocation, station selection, and logistics scheduling strategies based on the context features, realizing adaptive collaborative control under the arrival of new jobs, device failure, and processing time fluctuations. This method can significantly improve resource utilization and production efficiency, and has good engineering application value.

Description

Technical Field

[0001] This application relates to the field of multi-agent manufacturing systems in intelligent manufacturing technology, and in particular to a collaborative decision-making and dynamic optimization method for multi-agent manufacturing systems based on generative models. Background Technology

[0002] With the increasing diversification and personalization of product demands in society, multi-variety, small-batch production has become the mainstream production mode in manufacturing. The production process is characterized by high concurrency, mixed-line operation, and strong uncertainty. Uncertain events in the manufacturing system operating environment have become commonplace, including order adjustments, emergency order insertions, equipment failures, and resource fluctuations, causing significant dynamic and nonlinear changes in production load. Against this backdrop, maintaining stable and efficient system operation under complex disturbances has become a critical problem that modern manufacturing systems urgently need to solve.

[0003] Multi-Agent Manufacturing Systems (MAMS) abstract entities such as machine tools, robots, AGVs, and storage units into intelligent agents with autonomous perception and decision-making capabilities. Relying on local autonomy and group collaboration to achieve task allocation and dynamic scheduling, MAMS is considered an effective organizational model for addressing manufacturing uncertainties. However, significant bottlenecks remain in the engineering implementation of MAMS: on the one hand, many heterogeneous devices in real-world workshops remain in a "dumb terminal" state, lacking directly encapsulated perception and decision-making capabilities. Existing methods for constructing intelligent agents rely on engineers adapting each device individually, resulting in insufficient universality and scalability. On the other hand, existing multi-agent systems lack dynamic collaboration mechanisms based on environmental semantics, leading to rigid collaboration patterns that struggle to adapt to frequent changes and complex interactions in manufacturing scenarios. In recent years, generative artificial intelligence technologies, represented by DeepSeek and ChatGPT, have made breakthroughs in reasoning capabilities, knowledge extraction, and semantic modeling, providing new technical paths for the automatic construction of intelligent agents and the collaborative optimization of multi-agent systems. Generative models can understand equipment protocols, process logic, and collaboration rules across documents, laying the foundation for the rapid encapsulation of heterogeneous devices and the unified expression of knowledge. With the development of domestically produced large-scale models, autonomous, multi-agent manufacturing systems based on generative models are ushering in new development opportunities. Therefore, there is an urgent need for a method that can integrate generative reasoning capabilities with multi-agent collaborative optimization mechanisms to achieve automatic construction of equipment agents and adaptive control in complex production environments. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a dynamic collaborative optimization method for multi-agent manufacturing systems based on generative models. This method effectively solves key technical issues in current manufacturing systems, such as the difficulty in rapidly constructing the capabilities of equipment agents, insufficient decision-making response of multi-agents in dynamic environments, and low cross-equipment collaboration efficiency. It ensures that the manufacturing system can still achieve real-time, adaptive, and near-optimal collaborative optimization decisions under different production environments, such as order fluctuations, process changes, uneven equipment load, order insertion disturbances, and equipment failures.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] A dynamic optimization method for a multi-agent manufacturing system based on a generative model includes the following steps:

[0007] Step 1: Using the fusion generative model device agent builder, analyze the device parameter documents, communication protocol descriptions, register mapping tables, operation manuals, tool compensation rules and process procedures of machine tools, robots, automated guided vehicles (AGVs) and physical equipment in the workshop, and automatically generate the corresponding device agents.

[0008] Step 2: During the operation of the manufacturing system, the operating status, task attributes, logistics information, order insertion and fault information of machine tools, workpieces or processes, AGVs and storage or buffer units are collected in real time to construct the original state features containing multiple types of entities and their attributes.

[0009] A set of relationship matrices is constructed based on the processing feasibility relationship, pre- and post-process constraint relationship, resource competition relationship, material interaction relationship and path conflict relationship between entities. The entity attributes and their relationships are aggregated at multiple levels through the relationship Transformer network.

[0010] A gated hybrid expert attention mechanism is introduced after each layer of the relational Transformer network. Specifically, the semantics of equipment health, tasks and processes, logistics paths, and disturbance risks are mapped to different expert networks. The gated network adaptively fuses the outputs of each expert network according to the current operating state of the manufacturing system to obtain a high-dimensional contextual representation of each entity. Local contextual features corresponding to each equipment agent are generated through output mapping, and all entity representations are weighted and aggregated based on attention to generate global contextual features that reflect the overall system load, resource consumption, and disturbance risk distribution.

[0011] Step 3: Using the local context features and global context features as state inputs, construct a multi-agent proximal policy optimization (PPO) model for each manufacturing equipment agent. Adopt a centralized training and distributed execution architecture to conduct interactive learning in a digital twin workshop or simulation environment.

[0012] The proximal policy optimization (PPO) model is trained by advantage estimation, policy objective pruning, entropy regularization, learning rate decay, and gradient pruning mechanisms to achieve stable convergence and performance improvement of multi-agent policies.

[0013] Step 4: After the near-end policy optimization (PPO) model converges, the policy models of each device agent are deployed on the edge control node or local controller. When the workshop enters any decision moment, the context feature extraction module in Step 2 generates the latest local and global context features in real time. Each device agent can autonomously output control and scheduling decisions based solely on its local state and the converged policy.

[0014] Furthermore, the device intelligence agent adopts a three-layer architecture consisting of an adaptation layer, a communication layer, and an intelligent decision-making layer. The adaptation layer is used for issuing low-level action commands, reading and writing registers, alarm monitoring, executing security logic, and collecting device operating status. The communication layer is responsible for semantic-level message interaction between devices, and the intelligent decision-making layer is responsible for local planning, strategy selection, and online optimization. The device intelligence agent accesses the manufacturing system through a unified semantic interface.

[0015] Furthermore, in step 1, the device intelligent agent builder adopts a cloud-edge hybrid generative model inference system, including:

[0016] The cloud-based full-parameter generative large model is used to perform semantic understanding, mechanism reasoning, and cross-document knowledge extraction on equipment parameter documents, communication protocol descriptions, register mapping tables, operation manuals, tool compensation rules, and process procedures to generate adaptation layer control logic, communication semantic templates, and initial decision rules.

[0017] The edge-lightweight model is deployed on the device control node to perform real-time command parsing, motion control, and safety response.

[0018] Furthermore, the set of relation matrices is as follows: , Indicates time The set of relational matrices of the manufacturing system Indicates the current moment. Indicates a relation type index. This indicates the number of relation types. These relation types cover processing feasibility relationships between workpieces and equipment, pre- and post-process constraints between workpieces, parallel processing and resource competition relationships between equipment, material interaction relationships between AGVs and machine tools or storage units, and path conflict relationships between AGVs.

[0019] Furthermore, in step 2, the initial state vector and the structural position vector are concatenated and linearly mapped to generate the input features of the relational Transformer network.

[0020] ;

[0021] in, Represents entity nodes in a manufacturing system. Indicates at time Lower entity The initial embedding vector at layer 0 of the relational Transformer network Representing entities At any moment The initial state vector, For entities At any moment The structural position vector, For embedding matrix, For bias terms, To hide dimensions, The dimension of the initial state feature vector. For the structural location information vector dimension, Represents the real number field. This is a vector concatenation symbol;

[0022] A relational Transformer is used to perform deep semantic modeling of contextual information. The Transformer network consists of multiple stacked encoding layers. In each coding layer, for entities Generate query vectors:

[0023] ;

[0024] in, For the first The query mapping matrix of the layer, For query vector type, Indicates at time Lower entity The following is a description of the Transformer network after passing through the relationship. The updated feature representation of the layer. Indicates at time Lower entity The following is a description of the Transformer network after passing through the relationship. The query vector generated by the layer;

[0025] For each type of relation For entities under this relationship type Related entities Using relation types The corresponding dedicated key-value mapping matrix generates key vectors and value vectors:

[0026]

[0027] in, The first Layers based on relation types The key and value transformation matrices, Indicates at time Lower entity The following is a description of the Transformer network after passing through the relationship. The updated feature representation of the layer. and They represent the times at time 1 and 2 respectively. Lower entity The following is about relation types In the The layer generates value vectors and key vectors;

[0028] Based on the query vector and key vector Define the relational attention scoring function:

[0029] ;

[0030] in, For relation types The corresponding learnable bias scalar, Indicates in entity time The following is about relation types In the Layers and Entities Relationship attention score To hide the feature dimension, For a moment The following is about relation types ,entity With entity Is there a relationship between them?

[0031] All satisfied The score values ​​of the entity pairs are normalized using softmax to obtain the relation type. Lower entity For each adjacent entity Attention weights:

[0032] ;

[0033] in, Indicates in entity time The following is about relation types In the Layers and Entities Relationship attention score For a moment The following is about relation types , representing entities With entity Is there a relationship between them? It is an exponential function. Representing entities time The following is about relation types In the Layers and Entities The relational attention weight is used to weight and aggregate the value vector based on this weight at time 1. Next, in the Layer-generated entities In relation types The intermediate features below :

[0034] ;

[0035] A relation-level attention fusion mechanism is introduced to adaptively weight intermediate features under different relation types; for entities... Based on the intermediate features of each relation type, the importance weight of the relation is calculated. Then, based on the weight, the intermediate features under each relation type are weighted and aggregated to obtain the entity. Relationship fusion characteristics:

[0036] ;

[0037]

[0038] in, This represents a learnable relational score vector. Indicates at time , No. Layer to Entity The relative importance weights, Representing entities At any moment Relationship Transformer Network Layer fusion characteristics;

[0039] Preset A semantic expert network Each expert corresponds to a different semantic subspace, for entities Relationship fusion characteristics Through the gating network Generate a set of expert weight distributions:

[0040] ;

[0041] in, and , indicating the entity in the current context It should be from the first The amount of information obtained from each expert; combining the outputs of each semantic expert network, the entity is obtained. In the The final representation of the layer:

[0042] ;

[0043] in, It consists of several layers of nonlinear fully connected networks, fusing features. It contains several layers of information, which are integrated layer by layer from multi-source semantic information such as equipment status, task constraints, resource consumption, logistics path and disturbance risk to form high-dimensional features with contextual meaning;

[0044] When the number of layers reaches At that time, for all entities at time step The final semantic representation set ,in It represents the set of all entity nodes in the manufacturing system. Representing entities At any moment Relationship Transformer Network The final semantic representation obtained after layer processing; for the first layer... The entity corresponding to each device intelligent agent By outputting the mapping matrix Calculation time Local context features :

[0045] ;

[0046] Among them, local context features Used as the observation input for the device's agent policy network in step 3;

[0047] Constructing global context features First, we introduce the global attention vector. Calculate importance weights for all entities:

[0048] ;

[0049] in, and , represents an entity For time Contribution in the global context Representing entities At any moment Relationship Transformer Network The final semantic representation obtained after layer processing The global attention vector is represented by a weighted sum of all entity representations to obtain the time step. global context feature vector :

[0050] ;

[0051] Local context features With global context features Together they constitute the manufacturing system at time steps Contextual features are expressed.

[0052] Furthermore, in step 3, the adoption of a centralized training and distributed execution architecture for interactive learning in a digital twin workshop or simulation environment specifically refers to:

[0053] A shared, centralized value network is used to evaluate the global context state. In time Next global state For input, where Given the number of intelligent agents in the equipment, the output is a value estimate of the global state at the current moment, serving as an evaluation result of the global state of the manufacturing system. For value network parameters;

[0054] For time step The value estimation error is calculated using the time-series difference method:

[0055] ;

[0056] in This is a discount factor used to balance current rewards with future rewards. Indicates time Lower timing difference error, For time The global state below; take the total state within a training batch. At each time step, the squared TD errors are averaged to obtain the loss function of the value network. :

[0057] ;

[0058] Through the Update parameters using gradient descent This allows the value network to gradually approximate the true value function;

[0059] Based on the evaluation results of the global state, a generalized advantage estimation (GAE) method is used to construct the advantage function in a line of length [missing information]. In the interaction trajectory, for each time step Define the advantage function :

[0060] ;

[0061] in, The smoothing coefficient is the dominant factor. For time Lower timing difference error, Indicates the time step offset index;

[0062] Standardize the advantage values ​​at all time steps within the current training batch, and record the mean and standard deviation of the advantage for this batch as follows: and Then the normalization advantage for:

[0063] .

[0064] Furthermore, step 4 specifically involves:

[0065] For device intelligent agents Define the old and new strategies in the state. Next action The probability ratio is:

[0066] ;

[0067] in, For intelligent agents In time Local state observation, For intelligent agents In time The action performed For intelligent agents New strategy network parameters, For intelligent agents The old strategy network parameters, For the policy function, Represents intelligent agents In time The ratio of the probability of the new and old strategies on the same state-action pair;

[0068] Construct the objective function of the pruning policy that optimizes PPO using a multi-agent proximal strategy. :

[0069] ;

[0070] in This is a parameter to limit the range of strategy changes. The truncation operator is used to... Limited to Within the interval, This represents the expectation of the state-action-reward samples sampled in the current training batch. It is a minimum value function;

[0071] Add an entropy regularization term to the objective function of the pruning strategy and define the strategy entropy:

[0072] ;

[0073] in, For intelligent agents In time and status The policy entropy below, Indicated by network parameters The policy function represented Indicates the distribution from the current policy Actions obtained from sampling This indicates that the policy distribution is followed. Mathematical expectation of the action samples In the state Select Action The logarithmic probability.

[0074] Construct the total loss function of the Actor network :

[0075] ;

[0076] in These are entropy weighting coefficients;

[0077] Employing a learning rate decay and gradient clipping mechanism, let the first... The learning rate for the next iteration is It adopts an exponential decay form:

[0078] ;

[0079] in, The initial learning rate, To minimize the learning rate, For the first The decay coefficient of the next iteration;

[0080] Backpropagation yields the gradient vector Calculate its L2 norm. ,like Then the gradient is scaled:

[0081] ;

[0082] in, >0 is the gradient norm threshold. This represents the clipped gradient vector.

[0083] Beneficial effects:

[0084] This application achieves a unified fusion of device-level intelligence, semantic-level collaboration, and system-level proactive scheduling in manufacturing systems by integrating generative model-based device agent construction technology, context feature extraction methods using relational Transformers and gating expert mechanisms, and a multi-agent PPO dynamic optimization algorithm. This method can automatically construct device agents for machine tools, robots, AGVs, etc., from different brands and protocols, enabling rapid device access, unified semantic expression, and adaptive control. Simultaneously, it can extract high-dimensional contextual semantics from multi-source heterogeneous data, achieving accurate modeling of processing status, resource competition, logistics paths, and disturbance risks. Furthermore, by introducing a context-driven multi-agent reinforcement learning framework, this application can optimize task allocation and collaborative scheduling in real time under complex dynamic scenarios such as new workpiece insertion, equipment failure, and logistics congestion, effectively reducing work-in-process waiting time, alleviating bottleneck resource occupation, improving equipment utilization, and significantly increasing overall production efficiency. This method has good generalization and scalability, and can be widely applied to intelligent scheduling, device collaboration, and dynamic optimization control in discrete manufacturing workshops, possessing significant engineering application value and promising prospects for widespread adoption. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is an optimized flowchart for this application;

[0087] Figure 2 This is a diagram of the device intelligent agent architecture of this application;

[0088] Figure 3 This is a diagram of the hybrid generative model response service architecture of this application;

[0089] Figure 4 This is a diagram of the crowd intelligence decision-making optimization architecture based on multi-agent contextual collaborative learning proposed in this application. Detailed Implementation

[0090] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] like Figure 1 As shown, this application discloses a dynamic optimization method for a multi-agent manufacturing system based on a generative model, comprising the following steps:

[0092] Step 1: This application first proposes an AgentBuilder, a device intelligence builder that integrates generative models, to automate the generation of intelligent agents for manufacturing equipment of different brands, models, and interface protocols. The device intelligence agent adopts a three-layer architecture consisting of an adaptation layer, a communication layer, and an intelligent decision-making layer, enabling it to directly control physical equipment and perform scheduling decisions and collaborative optimization tasks. The entire construction process is based on a cloud-edge hybrid generative model inference system. The cloud side is a large, fully parameterized generative model responsible for highly complex tasks such as semantic understanding, mechanistic reasoning, and cross-document knowledge extraction; the edge side is a lightweight model deployed on the device control node for real-time control, command parsing, and security response.

[0093] During the construction of the device intelligent agent, the flexible adaptation engine first receives device parameter documents, communication protocol descriptions (such as Modbus, OPC UA, EtherCAT, HNC, etc.), register mapping tables, machine tool operation manuals, tool compensation rules, and process specifications. This text is then input into the cloud-based generative large model, which performs structured parsing to automatically generate adaptation layer control code. The adaptation layer is primarily responsible for issuing underlying action commands, reading and writing device registers, alarm monitoring, executing safety logic, and collecting device operating status. To ensure that the automatically generated adaptation code meets the real-time and safety requirements of the device, this application designs an interface connectivity verification module (verifying whether the generated code is consistent with the device communication protocol), a compilation consistency verification module (ensuring the generated code has no syntax errors and can be compiled correctly), and a semantic consistency verification module (verifying whether the behavior of the generated code conforms to the device manual and process constraints), ensuring that the generated adaptation layer has correctness, real-time performance, and reusability.

[0094] The communication layer is automatically constructed by a semantic negotiation engine. Its core task is to enable semantic-level message interaction between devices, including process completion notifications, task requests, expressions of collaborative intent, conflict detection, and anomaly negotiation. This application uses a generative large model to extract semantic roles, device capability descriptions, task dependencies, and communication behavior templates from inter-device collaboration process documents and MES / ERP system interface documents, and automatically generates cross-device communication protocol parsing modules, collaboration protocol matching modules, conflict detection modules, and semantic response modules. The communication layer can automatically configure network links based on the workshop topology, enabling real-time collaboration and data sharing between machine tools, robots, AGVs, and warehouses.

[0095] The intelligent decision-making layer is built upon a dynamic knowledge base. This dynamic knowledge base comprises an equipment parameter library, a kinematic parameter library, a process constraint library, a strategy experience library, and a semantic knowledge graph, and is continuously updated as processing tasks, equipment status, process changes, and algorithm training progress. This application uses a generative model to perform incremental semantic parsing on newly introduced equipment documents, processing logs, and alarm logs, writing them into the knowledge graph in the form of structured triples, enabling the equipment agent to have real-time knowledge update capabilities. Based on the equipment status collected by the adaptation layer and the collaborative semantic input obtained by the communication layer, the intelligent decision-making layer, combined with the dynamic knowledge base, performs local planning, strategy selection, and online optimization, enabling the equipment to complete tasks such as rule generation, processing action planning, task allocation, and collaborative scheduling.

[0096] Through the above mechanism, this application realizes the fully automated generation of the three-layer structure of the equipment intelligent agent, enabling machine tools, robots, AGVs and warehousing systems of different brands to access the manufacturing system in a consistent manner, and realize unified semantic expression, unified control logic and unified collaboration strategy.

[0097] Step 2: To address the challenges of multi-source heterogeneous, dynamically changing, and semantically densely related environmental information in manufacturing systems, a context feature extraction method based on a hybrid expert attention mechanism of relational Transformer and gating is proposed to construct a unified environmental semantic representation that can be shared by intelligent agents of multiple types of devices.

[0098] At any time step Let the set of entities in the current manufacturing system be denoted as . The total number of entities is For each entity Construct its time step Original state feature vector ,in The dimension of the initial state feature vector. Represents the real number field. When the entity When it is a machine tool, It can include fields such as spindle speed, feed rate, temperature, current, machining queue length, and health status index; when the entity When it is a workpiece or process, It can include fields such as current process number, number of remaining processes, task priority, delivery date deviation, and estimated processing time; when the entity When it is an AGV, It can include fields such as current location coordinates, speed, remaining battery power, path occupancy flag, and current task queue length; when the entity When used as a storage or buffer unit, It can include fields such as inventory quantity, buffer occupancy rate, and material type distribution. Continuous features are normalized, Boolean states are encoded using binary encoding, and time-series sensor signals can have local trend features extracted using sliding windows or lightweight convolution before being concatenated into the vector, ensuring accurate data transmission at each time step. Obtain the entity state set .

[0099] To characterize the structural relationships between entities, this application constructs a relation matrix set. , Indicates time The set of relational matrices of the manufacturing system Indicates the current moment. Indicates a relation type index. This indicates the number of relation types. These types cover various relationships, including workpiece-equipment processing feasibility relationships, workpiece-workpiece pre- and post-process constraints, parallel processing and resource competition relationships between equipment, material interaction relationships between AGVs and machine tools or storage units, and path conflict relationships between AGVs. Relationship Matrix Representing entities With entity In semantic type If there is a business relationship, the value is 0; otherwise, it is 0. To provide a unified topology for subsequent location encoding, this application merges the various relationship matrices into an adjacency matrix according to trainable weights. :

[0100]

[0101] in, This represents the total number of predefined relation types in the manufacturing system. Indicates the first The fusion weights corresponding to the relationship types are determined, and a degree matrix is ​​constructed accordingly. and normalized Laplace matrix :

[0102]

[0103] It is the identity matrix. For Perform eigenvalue decomposition to obtain the matrix

[0104]

[0105] in The eigenvector matrix, Let be the eigenvalue matrix. Take the smallest eigenvalue. A matrix is ​​formed by non-trivial eigenvectors. , its first The row vector is:

[0106]

[0107] For entities At any moment The structural location encoding is used to describe the global location, structural distance, and similarity of an entity within the manufacturing system topology. To integrate the structural location encoding with entity features, this application concatenates the initial state with the structural location vector and generates the input features of the relational Transformer through a linear mapping.

[0108]

[0109] in, Represents entity nodes in a manufacturing system. Indicates at time Lower entity The initial embedding vector at layer 0 of the relational Transformer network Representing entities At any moment The initial state vector, For entities At any moment The structural position vector, For embedding matrix, For bias terms, To hide dimensions, The dimension of the initial state feature vector. For the structural location information vector dimension, Represents the real number field. This refers to vector concatenation symbols. After constructing the entity input vector, this application employs a relational Transformer to perform deep semantic modeling of the contextual information. In the... In the layer, for entities Generate query vectors:

[0110]

[0111] in, For the first The query mapping matrix of the layer, For query vector type, Indicates at time Lower entity The following is a description of the Transformer network after passing through the relationship. The updated feature representation of the layer. Indicates at time Lower entity The following is a description of the Transformer network after passing through the relationship. The query vector generated by the layer. Then, for each relation type... For entities under this relationship type Related entities Using relation types The corresponding dedicated key-value mapping matrix generates key vectors and value vectors:

[0112]

[0113] in, The first Layers based on relation types The key and value transformation matrices, Indicates at time Lower entity The following is a description of the Transformer network after passing through the relationship. The updated feature representation of the layer. and They represent the times at time 1 and 2 respectively. Lower entity The following is about relation types In the The layer generates value vectors and key vectors. Based on these vectors, a relational attention scoring function is defined.

[0114]

[0115] in, For relation types The corresponding learnable bias scalar, Indicates in entity time The following is about relation types In the Layers and Entities Relationship attention score To hide the feature dimension, For a moment The following is about relation types ,entity With entity Does a relationship exist between them? (Find all that satisfy...) The score values ​​of the entity pairs are normalized using softmax to obtain the relation type. Lower entity For each adjacent entity Attention weights

[0116]

[0117] in, Indicates in entity time The following is about relation types In the Layers and Entities Relationship attention score For a moment The following is about relation types ,entity With entity Is there a relationship between them? It is an exponential function. Representing entities time The following is about relation types In the Layers and Entities The relational attention weight is used to weight and aggregate the value vector based on this weight at time 1. Next, in the Layer-generated entities In relation types The intermediate features below :

[0118]

[0119] A relation-level attention fusion mechanism is introduced to adaptively weight intermediate features under different relation types. This is applied to entities... Based on the intermediate features of each relation type, the importance weight of the relation is calculated. Then, based on the weight, the intermediate features under each relation type are weighted and aggregated to obtain the entity. Relationship fusion characteristics:

[0120]

[0121]

[0122] in, This represents a learnable relational score vector. Indicates at time , No. Layer to Entity The relative importance weights, Representing entities At any moment Relationship Transformer Network Layer fusion characteristics.

[0123] To further enhance the selective modeling capability of contextual semantics, this application introduces a gated hybrid expert mechanism after each relational Transformen layer. This mechanism is pre-defined. A semantic expert network Each expert corresponds to a different semantic subspace, such as equipment health semantics, task and process semantics, logistics and path semantics, and disturbance semantics. For entities... Relationship fusion characteristics Through gating network Generate a set of expert weight distributions:

[0124]

[0125] in and , indicates the entity in the current context It should be from the first How much information is obtained from each expert? The entity is obtained by combining the outputs of each semantic expert network. In the The final representation of the layer

[0126]

[0127] in It typically consists of several layers of nonlinear fully connected networks. Entity representation is achieved through the overlay of multi-layer relational Transformers and gated hybrid expert mechanisms. By integrating multi-source semantic information such as equipment status, task constraints, resource usage, logistics routes, and disturbance risks layer by layer, high-dimensional features with rich contextual meaning are formed.

[0128] When the number of layers reaches At that time, this application obtained all entities at the time step The final semantic representation set In order to map contextual information into observational features that the agent can directly use, for an agent with respect to a particular device... Corresponding entity By outputting the mapping matrix Calculate local context features

[0129]

[0130] in This is used as the observation input for the agent's policy network in step 3. Simultaneously, to support centralized value assessment and global collaborative decision-making, this application constructs global context features. Specifically, we first introduce a global attention vector. Calculate importance weights for all entities.

[0131]

[0132] in and , represents an entity For time Contribution in the global context Representing entities At any moment Relationship Transformer Network The final semantic representation obtained after layer processing The global attention vector is represented by a weighted sum of all entity representations to obtain the time step. global context feature vector

[0133]

[0134] Ultimately, local contextual features With global context features Together they constitute the manufacturing system at time steps The contextual features are used as inputs to the multi-agent reinforcement learning policy network and the centralized value network in step 3, providing each device agent with complete environmental context information including device status, task semantics, resource consumption, logistics path and disturbance risk, thereby realizing real-time intelligent decision-making and collaborative control of complex manufacturing systems.

[0135] Step 3: Construct a context-driven dynamic optimization method based on multi-agent PPO, enabling machine tool agents, work-in-process agents, workstation agents, AGV agents, etc., to utilize a unified context state. This method employs strategy learning to achieve cross-equipment collaborative optimization and proactive scheduling control in a workshop environment characterized by frequent dynamic disturbances such as new workpiece arrivals, process completions, order insertions, equipment failures, and logistical congestion. It utilizes a multi-agent reinforcement learning framework with centralized training and distributed execution. During training, it shares a global context for joint optimization, while during execution, it relies solely on its local context for rapid decision-making. At any given time step... For device intelligent agents Its context state is ,in The local context features obtained in step 2 are used to characterize the device states, task queues, neighborhood resources, and local logistics information directly related to the agent. This application uses global context features to characterize system-level resource allocation, bottleneck distribution, path congestion, and disturbance risks. Based on this, this application provides features for each intelligent agent. Design a policy network (Actor) and a value network (Critic). The policy network is based on states. Given the input, output the probability distribution of the candidate action set. ,in For intelligent agents The strategy network parameters, For the policy function, This represents actions sampled from the current strategy distribution; candidate actions can include workpiece selection actions for machine tool-work-in-process agents, parallel workstation allocation actions for work-in-process-station agents, task assignment and path selection actions for AGV agents, and outbound sorting actions for warehouse agents, etc. The roulette wheel method is used to select... Mid-sampling yields the actual actions performed. This ensures that the strategy has sufficient exploratory power in the early stages of training and gradually forms a stable preference in the later stages of training.

[0136] The multi-agent scheduling optimization process in this application aims to minimize the maximum completion time. With maximizing average machine utilization With the core optimization objective as the guiding principle, the following dual-objective optimization model is constructed:

[0137]

[0138]

[0139] in, For the first The completion time of the final process for each workpiece. For workpiece The number of processes included. The number of workpieces. It refers to the number of machines. For process Is it assigned to a machine? 0-1 variables, It is a process In the machine On the processing time, It is a process In the machine On the processing time, It is a function with maximum value. The summation function is used. Based on the optimization objective, a reward function is designed, mainly consisting of two parts: (i) the change in the maximum completion time of the task, denoted as... (ii) Changes in average machine utilization rate, denoted as .in, Describes the maximum deviation in job completion time, while This measures whether the average utilization rate increases or decreases. Under a given scheduling state, the process... The expected start time is defined as follows: if its preceding process Already assigned to the machine Then its start time is ,in For a moment Next process The expected start time of processing, For a moment Next preceding process The expected start time of processing, It is a process In the machine The processing time is calculated based on the average processing time; otherwise, the start time is estimated based on the average processing time. ,in For operation Average processing time across all available machines. For operations not yet scheduled. The lower bound of its completion time Estimated as in For operation The completion time, For the first One machine, To enable processing steps The first term represents the set of machines, and the second term represents the shortest processing time among all the machines that can process the data. To ensure... and The dimensions are consistent, and the completion time is expressed using normalized processing time. Ultimately, the overall reward... The calculation is as follows:

[0140]

[0141]

[0142]

[0143] When all the agents that need to make decisions are at time step After obtaining the individual actions, a joint action vector is formed. ,in The number of intelligent agents involved in control. For the first Each agent's time Selected local actions. For agents that do not need to make decisions at certain times, this application uses hold actions or placeholder actions to maintain the consistency of the joint action vector structure. Joint actions The action set is assigned to the workshop environment for execution, and the environment calculates an immediate reward based on the impact of this action set on system performance. This drives the manufacturing system to the next time step. The reward function is comprehensively constructed based on the shop floor optimization goals, and can simultaneously consider factors such as average workpiece flow time, maximum completion time, average work-in-process waiting time, key equipment utilization rate, AGV operating efficiency, energy consumption level, and delay penalties. Actions that improve overall production efficiency, alleviate bottlenecks, reduce work-in-process inventory, and reduce energy consumption are rewarded more highly, while actions that cause long waiting times, resource conflicts, frequent switching, or increased energy consumption are penalized. After the action is executed, the manufacturing system moves to the next time step. The physical state is collected by sensors and information systems. The context feature extraction module in step 2 is run again to recalculate the relationships between all entities. The forward propagation of the Transformer and the gated expert network yields new local context features. With global context features This leads to a new state. Based on the above interaction process, this application is an intelligent agent. At time step Constructing a six-tuple sample:

[0144]

[0145] in, For the first An intelligent agent in time Local state representation, For the first Each intelligent agent executes The local state representation at the next moment, For the first An intelligent agent in time The action, It is time The reward For intelligent agents The new policy network parameters are the policy function. This is a round termination flag variable. The above joint samples are stored in the context sample library, providing a data foundation for the centralized training phase.

[0146] During the strategy optimization phase, this application employs a shared, centralized value network to evaluate the global context state. Value Network by As input, output a value estimate for this global state, where These are the parameters for the value network. For each time step... The value estimation error is calculated using the time-series difference method:

[0147]

[0148] in This is a discount factor used to balance current rewards with future rewards. Indicates time Lower timing difference error, For time The global state under the current conditions. Take the total state within a training batch. At each time step, the squared TD errors are averaged to obtain the loss function of the value network. :

[0149]

[0150] Through the Update parameters using gradient descent This allows the value network to gradually approximate the real value function, providing a reliable value reference for strategy improvement.

[0151] To improve the stability and expressiveness of the dominance estimation, this application employs the generalized dominance estimation (GAE) method to construct the dominance function. In a given length... In the interaction trajectory, for each time step definition:

[0152]

[0153] in, The smoothing coefficient is the dominant factor. For time Lower timing difference error, This represents the time step offset index. To avoid excessive differences in the scale of the dominance values ​​between different training batches, which could lead to training instability, this application standardizes the dominance values ​​of all time steps within the current batch, and denotes the mean and standard deviation of the dominance for that batch as follows: and Then the normalization advantage for:

[0154]

[0155] Regarding policy updates, this application employs the PPO (Policy Objective of Policy) pruning to construct a stable policy gradient. For the agent... Define the old and new strategies in the state. Next action The probability ratio is:

[0156]

[0157] in, For intelligent agents In time Local state observation, For intelligent agents In time The action performed For intelligent agents New strategy network parameters, For intelligent agents The old strategy network parameters, For the policy function, Represents intelligent agents In time The ratio of the probability of the new and old strategies being in the same state-action pair.

[0158] Based on this, the objective function of the strategy is constructed as follows:

[0159]

[0160] in This is a parameter to limit the range of strategy changes. The truncation operator is used to... Limited to Within the range, to prevent excessively large policy updates from causing a sudden drop in performance, This represents the expectation of the state-action-reward samples sampled in the current training batch. This is a minimum value function. To enhance the policy's exploration capability, this application adds an entropy regularization term to the policy objective, defining the policy entropy as follows:

[0161]

[0162] in, For intelligent agents In time and status The policy entropy below, Indicated by network parameters The policy function represented Indicates the distribution from the current policy Actions obtained from sampling This indicates that the policy distribution is followed. Mathematical expectation of the action samples In the state Select Action The logarithmic probability.

[0163] Construct the total loss function for the Actor network:

[0164]

[0165] in is the entropy weighting coefficient, used to balance exploration and convergence.

[0166] To improve the stability of convergence during training, this application employs both learning rate decay and gradient clipping mechanisms. The learning rate decay and gradient clipping mechanisms are used, assuming the first... The learning rate for the next iteration is It adopts an exponential decay form:

[0167] ;

[0168] in, The initial learning rate, To minimize the learning rate, For the first The decay coefficient in each iteration; by gradually reducing the learning rate in the later stages of training, the oscillations in parameter updates can be reduced, and the convergence smoothness can be improved. The gradient vector is obtained in each backpropagation. Subsequently, this application calculates its second norm. ,like Then the gradient is scaled:

[0169]

[0170] in, >0 is the gradient norm threshold. This represents the clipped gradient vector. Gradient clipping effectively suppresses gradient explosion and further improves the numerical stability of the multi-agent PPO training process.

[0171] Regarding the multi-agent collaborative architecture, this application adopts a centralized training and distributed execution model. During the centralized training phase, all agents share the global context features generated in step 2. And through a unified value network, the global state is monitored. The value is evaluated; each agent's policy network is based on its own state. Optimize and achieve collaborative learning through shared value signals.

[0172] Step 4: In the distributed execution phase, each device agent is deployed in an edge control node or local controller. When the physical workshop enters any decision-making moment... At that time, the context feature extraction module in step 2 calculates in real time. and intelligent agent Only depends on local state and the converged policy network It can independently output control and scheduling decisions without needing to access centralized value networks or other intelligent agents' internal parameters online, thereby significantly reducing the computational overhead and communication pressure of real-time operation.

[0173] like Figure 2 As shown, this application first proposes an AgentBuilder, a device intelligence agent builder that integrates generative models, to automate the generation of intelligent agents for manufacturing equipment of different brands, models, and interface protocols. The device intelligence agent adopts a three-layer structure of "adaptation layer—communication layer—intelligent decision-making layer," enabling it to directly control specific physical equipment and perform advanced decision-making tasks such as scheduling, collaboration, and local optimization. The entire construction process is as follows: Figure 1 As shown, by automatically parsing heterogeneous information from multiple sources such as equipment parameters, protocol documents, process specifications, and operation manuals, an intelligent agent program that perfectly matches the equipment capabilities is generated. This enables underlying machine tools, robots, AGVs, and warehousing nodes to access the multi-agent manufacturing system using a unified interface format. To achieve the above functions, the Agent Builder constructed in this application adopts the following... Figure 3 The hybrid generative model response service architecture shown in this embodiment is a cloud-edge hybrid generative model response service architecture. The cloud-side generative large model is responsible for complex tasks such as semantic understanding, cross-modal knowledge extraction, mechanism reasoning, and rule generation. By parsing text data such as process documents, development documents, scheduling rules, processing strategies, and communication protocol descriptions, it automatically generates the initial logical structure, interface specifications, and behavioral models of the device's intelligent agent. The edge-side lightweight model is deployed on the edge control nodes of each device and is responsible for instruction parsing, real-time action control, execution of security protection logic, and status monitoring, ensuring that the intelligent agent can operate stably in industrial scenarios with extremely high real-time requirements.

[0174] During the automatic construction of the device's intelligent agent, the flexible adaptation engine first receives relevant device documents, including communication protocol specifications (such as Modbus, OPC UA, EtherCAT, HNC system commands, etc.), register mapping tables, machine tool operation manuals, tool compensation strategies, and process constraint specifications, and sends them as input. Figure 2 The generative large-scale model inference layer is shown. The generative model performs structured parsing of the aforementioned document, extracting device register definitions, motion axis parameters, supported G / M codes, alarm structures, interaction modes, and processing flow information, and automatically generates adaptation layer code. The adaptation layer is responsible for the underlying motion control of the device, including register read / write, motion command issuance, alarm detection, emergency stop protection, and processing status acquisition; to ensure the reliability of the generated code, this application... Figure 1The adapter layer module introduced interface connectivity verification, compilation consistency verification and semantic consistency verification mechanisms are used to verify the consistency between the generated code and the actual device communication protocol, code compilability, and whether the logical behavior conforms to the equipment manual and process specifications, thereby ensuring that the adapter layer has deployability, security and reusability in industrial production.

[0175] like Figure 1 As shown, a communication layer is built on top of the adaptation layer, with its core implemented by a semantic negotiation engine. The communication layer extracts task dependencies, device collaboration roles, data exchange formats, and communication action templates from collaboration templates, device capability descriptions, MES / ERP interface documents, and typical task flows using a generative model. It also automatically generates cross-device communication protocol parsing modules, semantic matching modules, conflict detection modules, and collaboration response modules. Based on the workshop topology, the communication layer establishes semantic-level connections between machine tools, robots, AGVs, and warehousing equipment, enabling real-time interaction of various messages such as process completion notifications, material requests, processing readiness, path occupancy reminders, and anomaly negotiation and recovery instructions. This allows multiple intelligent agents to collaborate in a consistent manner, avoiding the need for manually writing complex communication logic.

[0176] Above the communication layer, such as Figure 1 and Figure 2 The jointly demonstrated intelligent decision-making layer is responsible for the high-level planning and strategy execution of the intelligent agents. This application... Figure 2 A dynamic knowledge base has been constructed, including a basic equipment parameter library, a kinematic parameter library, a process and safety constraint library, a scheduling strategy experience library, and a semantic knowledge graph. For example... Figure 4 As shown, a swarm intelligence decision optimization method based on contextual collaborative learning of multi-agent PPO is constructed.

[0177] Example 1:

[0178] This application provides a case study of a dynamic, flexible workshop where workpieces are randomly generated from six types. The platform's basic components include an automated storage and retrieval system (AS / RS, i.e., one warehouse), two lathes (M1, M2), two milling machines (M3, M4), and two drilling machines (M5, M6). The AS / RS is equipped with a warehouse for storing workpiece blanks, while the machine tools are equipped with a buffer zone for temporarily storing workpieces awaiting processing. Based on scenarios K1 to K10, the generated perturbation instances are denoted as DK1 to DK10, as follows: DK1: one machine failure and one new job arrival; DK2: random fluctuation in processing time; DK3: insertion of two new jobs; DK4: changes in processing time accompanied by the arrival of six new jobs; DK5: three machine failures; DK6: one new job arrival, two machine failures, and processing time fluctuations; DK7: six new jobs arriving and one machine failure; DK8: two machine failures; DK9: one machine failure, three new jobs arriving, and random fluctuations in processing time; DK10: one new job arrival and random changes in processing time. The method of this application is compared with four different composite rules, two different multi-agent reinforcement learning algorithms including the unmodified MAPPO and MADDPG algorithms, and two heuristic algorithms PSO and GA, as shown in Table 1. This embodiment selects some representative scheduling rules, including two job ordering rules (PDR) and two machine allocation rules, and combines them into four composite rules for comparison with the proposed framework. Three job sorting rules are included: First-Come, First-Served (FIFO) and Maximum Remaining Work First (MWKR); two machine allocation strategies are included: Shortest Processing Time First (SPT) and Earliest Finish Time First (EET). By combining job selection rules and machine selection rules, four composite priority scheduling rules (PDR) are derived: Rule 1, Rule 2, Rule 3, and Rule 4. Under ten typical dynamic disturbance scenarios (DK1–DK10), the proposed method significantly outperforms the comparative algorithms in both maximum completion time (Cmax) and average machine utilization (f). For maximum completion time, the proposed method achieves globally optimal or near-optimal scheduling results in most high-disturbance scenarios (such as K02, K03, K04, K07, K08, K09, and K10), effectively mitigating processing congestion and increased waiting time caused by new job arrivals, machine failures, and fluctuations in processing time. In complex scenarios with frequent machine failures or multiple perturbations (such as DK4, DK6, and DK9), this application still maintains stable scheduling performance and does not experience a sudden drop in performance like traditional rule-based or unimproved reinforcement learning algorithms.Meanwhile, regarding the average machine utilization rate f, this application achieved the highest utilization rate in most of the ten scenarios. For example, in K02, K03, K04, K09, and K10, this application significantly outperformed MAPPO, MADDPG, and the four composite scheduling rules. This indicates that the proposed context-aware feature modeling and multi-agent collaborative optimization strategy can more fully mobilize system resources, enabling machine tools to maintain a more continuous and balanced processing rhythm in dynamic environments. Overall, this application demonstrates superior stability and generalization ability in dealing with dynamic disturbances, shortening completion time, and improving machine utilization, effectively supporting the high-dynamic scheduling needs in actual manufacturing workshops.

[0179] Table 1

[0180]

[0181] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, for the device embodiments, the above descriptions are merely preferred embodiments of the present invention. Since they are fundamentally similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention, without departing from the principle of the present invention, should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamic optimization of multi-agent manufacturing systems based on generative models, characterized in that, Comprising the following steps: Step 1, using the device agent constructor of the fusion generative model, the device parameter document, communication protocol description, register mapping table, operation manual, tool compensation rule and process procedure of the physical devices such as machine tool, robot, automatic guided vehicle AGV and storage unit in the workshop are parsed, and the corresponding device agent is automatically generated; Step 2, during the operation of the manufacturing system, the running state, task attribute, logistics information and order insertion, fault information of machine tool, workpiece or process, AGV and storage or buffer unit are collected in real time, and the original state characteristics containing multiple entities and their attributes are constructed; Based on the processing feasible relationship between entities, the constraint relationship before and after the process, the resource competition relationship, the material interaction relationship and the path conflict relationship, a relationship matrix set is constructed, and the entity attributes and their relationships are aggregated through multi-level features by relationship Transformer network; After each layer of relationship Transformer network, a gated hybrid expert attention mechanism is introduced, specifically, the device health, task and process, logistics path and disturbance risk semantics are mapped to different expert networks, and the output of each expert network is adaptively fused by the gating network according to the running state of the current manufacturing system, to obtain the high-dimensional context representation of each entity; The local context features corresponding to each device agent are generated by output mapping, and the global context features reflecting the overall load, resource occupation and disturbance risk distribution of the system are generated by attention-based weighted aggregation of all entity representations; Step 3, the local context features and global context features are taken as state input, and a multi-agent proximal policy optimization PPO model is constructed for each manufacturing device agent, which is trained in a centralized training manner and adopts a distributed execution architecture, specifically: For the device agent , the new and old policies define a probability ratio of an action in a state : ； wherein, is an agent at time a local state observation, is an agent at time an action performed, are new policy network parameters for an agent are old policy network parameters for an agent is a policy function, denotes an agent at time a probability ratio of the new and old policies on the same state-action pair;​​ Constructing a cropped policy objective function of proximal policy optimization PPO using multi-agent : ； wherein is a policy variation range limit parameter, is a truncation operator for truncating to be within an interval, denotes an expectation over state-action-return samples sampled in the current training batch, is a minimum function; An entropy regularization term is added to the clipped policy objective function, and the policy entropy is defined as: ； in, For intelligent agents In time and status The policy entropy below, Indicated by network parameters The policy function represented Indicates the distribution from the current policy Actions obtained from sampling This indicates that the policy distribution is followed. Mathematical expectation of the action samples In the state Select Action The logarithmic probability; Constructing the actor network total loss function : ； wherein is an entropy weight coefficient; The learning rate is set as , and the exponential decay form is adopted: , and the exponential decay form is adopted: ； wherein, is an initial learning rate, is a minimum learning rate, is a decay coefficient for the th iteration; Backpropagation yields the gradient vector , compute its 2-norm , if then scale the gradient: ； wherein, > 0 is a gradient norm threshold, denotes the clipped gradient vector; Interactive learning in a digital twin workshop or simulation environment; The proximal policy optimization PPO model is trained through advantage estimation, clipped policy target, entropy regularization, learning rate decay and gradient clipping mechanism, to realize stable convergence and performance improvement of multi-agent strategy; Step 4, after the convergence of the proximal policy optimization PPO model, the strategy model of each device agent is deployed on the edge control node or local controller.

2. The method of claim 1, wherein, The device agent adopts a three-layer architecture of adaptation layer, communication layer and intelligent decision layer, the adaptation layer is used for bottom action instruction issuing, register reading and writing, alarm monitoring, safety logic execution and device running state acquisition; The communication layer is responsible for semantic level message interaction between devices, and the intelligent decision layer is responsible for local planning, strategy selection and online optimization; The device agent accesses the manufacturing system through a unified semantic interface.

3. The method of claim 1, wherein, In step 1, the device agent constructor adopts a cloud-edge hybrid generative model inference system, which comprises: The cloud-side full-parameter generation model is used for semantic understanding, mechanism reasoning and cross-document knowledge extraction on device parameter documents, communication protocol descriptions, register mapping tables, operation manuals, tool compensation rules and process procedures, and generates adaptive layer control logic, communication semantic templates and initial decision rules. The edge-side lightweight model is deployed on the device control node and is used for executing real-time instruction analysis, action control and safety response.

4. The method of claim 1, wherein, The relationship matrix set is , represents the time of the manufacturing system, represents the current time, represents the relationship type index, represents the number of relationship types; wherein the relationship types cover the machining feasible relationship between the workpieces and the equipment, the process before and after constraint relationship between the workpieces, the parallel machining and resource competition relationship between the equipment, the material interaction relationship between the AGV and the machine tool or the storage unit, and the path conflict relationship between the AGVs.

5. The method of claim 1, wherein, In step 2, the initial state vector and the structure position vector are spliced, and the input features of the relationship Transformer network are generated through linear mapping: ； in, Represents entity nodes in a manufacturing system. Indicates at time Lower entity The initial embedding vector at layer 0 of the relational Transformer network Representing entities At any moment The initial state vector, For entities At any moment The structural position vector, For embedding matrix, For bias terms, To hide dimensions, The dimension of the initial state feature vector. For the structural location information vector dimension, Represents the real number field. This is a vector concatenation symbol; The relational Transformer is used to model deep semantic context information, and the Transformer network is stacked by multiple encoding layers. In the first encoding layer, the entity is generated query vector: ； wherein, is the number of layers, is the number of layers, is a query vector type, denotes the feature representation of the entity at time step after being updated by the relation Transformer network at the -th layer, denotes the query vector generated by the relation Transformer network at the -th layer for the entity at time step . For each relationship type , for the entities associated with the entity under the relationship type, a key vector and a value vector are generated using the specialized key, value mapping matrix corresponding to the relationship type . ； wherein, are the first and second entity representations, respectively, is a key and value transformation matrix for the relation type , denotes the feature representation of the entity at time step after being updated by the relation Transformer network at layer , and denote the value vector and key vector, respectively, generated for the relation type at time step for the entity at layer , based on the query vector and key vectors define a relation attention scoring function: ； wherein, is a relation type corresponding learnable bias scalar, denotes the existence of a relation between entity at time for relation type at the layer and entity relation attention score, is a hidden feature dimension, is a relation type , whether a relation exists between entity and entity at time ; The scoring values of all entity pairs satisfying are normalized by softmax to obtain the relationship type of the lower entity The attention weight of each adjacent entity ​ ； wherein, denotes the attention score of the entity at time for the relation type at layer to the entity , denotes whether there is a relation between the entity and the entity for the relation type at time , is an exponential function, denotes the attention weight of the entity at time for the relation type at layer to the entity , and the value vector is weighted and aggregated according to the weight, and the intermediate feature of the entity at layer for the relation type is generated at time : ； For an entity , the relationship importance weight is calculated based on the intermediate features of each relationship type, the intermediate features under each relationship type are weighted and aggregated based on the weight, and the relationship fusion features of the entity are obtained: ； ； wherein, represents a learnable relation score vector, represents the fusion feature of the entity , the relative importance weight of the entity to the relation Transformer network, represents the fusion feature of the entity at time , the fusion feature of the relation Transformer network Pre-set A semantic expert network Each expert corresponds to a different semantic subspace, and for each entity The relationship fusion feature A set of expert weight distributions is generated through the gating network ​ ； wherein, and denotes the representation of the entity in the current context; the number of information that should be obtained from the th expert; combined with the output of each semantic expert network, the final representation of the entity in the th layer: ； wherein, is a number of layers of nonlinear fully connected networks that fuse features contains a number of layers of information, which integrates multi-source semantic information from device state, task constraint, resource occupation, logistics path and disturbance risk layer by layer to form high-dimensional features with contextual meaning; When the number of layers reaches , the final semantic representation set of all entities at time step , where represents a set composed of all entity nodes in the manufacturing system, represents an entity , and represents the final semantic representation obtained after the first layer of the relationship Transformer network is processed; for an entity corresponding to the first device intelligent agent, the local context feature at time is calculated through an output mapping matrix :​ ； wherein the local context features for use as observation input to the device agent policy network in step 3; Constructing global context features , first introduce global attention vectors , compute importance weights for all entities: ； wherein, and denotes an entity For time point Contribution degree in global context, denotes an entity At time point , the final semantic representation obtained after the first layer processing of the relation Transformer network, denotes a global attention vector; weighted sum of all entity representations, to obtain the global context feature vector of time step : ； Local context features Global context features Contextual features collectively constitute a manufacturing system at a time step ​ 6. The method of claim 1, wherein, In step 3, the interactive learning in the digital twin workshop or the simulation environment is specifically: The global context state is evaluated using a shared centralized value network, which is trained to estimate the value of a global state in time The global state is inputted, where is the number of equipment agents, the output is the value estimation of the global state at the current time, which is the evaluation result of the global state of the manufacturing system, where is the value network parameter; For time step , the value estimation error is computed by the time-difference method: ； where is a discount factor that balances current rewards with future rewards, denotes time The next-time-step error, is the global state at time ; take a training batch of time steps, average the squared TD errors to get the loss function for the value network : ； By updating the parameters through gradient descent The value network gradually approaches the true value function.​ Based on the evaluation result of the global state, a generalized advantage estimation GAE method is used to construct an advantage function, and in an interaction trajectory with a length of , an advantage function is defined for each time step . ； wherein, is the dominant smoothing coefficient, is the time is the down temporal difference error, denotes the time step offset index; The advantage values of all time steps within the current training batch are normalized by the mean and standard deviation of the batch advantage and , respectively. The normalized advantage is given by: 。

Images