A workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions

Abstract

The application provides a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions, relates to the technical field of production scheduling and industrial automation, and comprises the following steps: constructing a multi-dimensional time sequence matrix by collecting production data in real time through a distributed sensor network; establishing a three-dimensional decision space containing time, resource and task dimensions, and adopting an improved multi-agent deep deterministic policy gradient algorithm; dynamically calculating the real-time production urgency of production batches, analyzing the collaborative efficiency factor generated by cross-unit cooperation; combining the two to optimize the scheduling scheme through a curriculum learning strategy; and verifying the scheme in a virtual twin environment and feeding back and updating the model parameters. The application breaks through the limitations of traditional scheduling, such as insufficient dynamic adaptability and low efficiency of multi-target cooperation, realizes real-time response to dynamic production conditions such as equipment failure and order adjustment, balances time efficiency, energy consumption economy and abnormal fault tolerance through a multi-agent cooperation and closed-loop verification mechanism, and improves the resource cooperation efficiency under complex production constraints.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of production scheduling and industrial automation, in particular to a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions. BACKGROUND

[0002] With the acceleration of the intelligent transformation of manufacturing industry, the workshop production scheduling, as the core link of the manufacturing system, is facing an increasingly complex dynamic environment and multi-objective optimization demand. Traditional scheduling methods are mostly based on static rules or offline optimization models, which are difficult to adapt to sudden equipment failures, order priority adjustments, material supply fluctuations and other real-time changes in production conditions. Especially in the multi-variety, small-batch customized production scene, the process connection is high in tightness and the resource competition is fierce. The existing scheduling system often causes problems such as decreased production efficiency, increased energy consumption and abnormal event propagation due to response lag or decision rigidity, and it is urgent to break through the inherent limitations of the static scheduling mode.

[0003] The current mainstream scheduling technology has significant shortcomings in dynamic adaptability, multi-objective coordination, intelligent algorithm practicality and real-time data fusion. Based on fixed rules (such as first-come-first-served) or periodic rescheduling strategies, it is difficult to real-time perceive dynamic factors such as equipment state and environmental parameters, resulting in a disconnection between the scheduling scheme and the actual situation, for example, it is difficult to adjust the processing order in time to avoid quality defects when the equipment temperature is abnormal, and it is also difficult to quickly reconstruct the path optimization scheme when the material flow is delayed; mostly using single objective optimization (such as shortest completion time) or simple weighted sum method, it is difficult to balance the conflict between energy economy, time efficiency and abnormal fault tolerance, and it may cause chain problems or cause waste of production capacity due to excessive pursuit of a certain target; reinforcement learning and other intelligent algorithms are prone to local optimization in multi-agent collaboration scenarios, and rely on simulation environments that lack high-fidelity modeling capabilities for training, resulting in a significant decline in effectiveness when the strategy is migrated to the actual workshop; most systems lack a full-dimensional data perception system and online feedback mechanism, making it impossible to achieve a closed-loop iteration of "decision-making - execution - verification - optimization", resulting in a disconnection between decision-making and actual production process.

[0004] The industry has tried to improve the dynamic response capability of the scheduling system through technologies such as Internet of Things perception and digital twinning, but still faces multiple challenges: the real-time fusion and feature extraction efficiency of multi-source heterogeneous data (such as vibration spectrum, temperature curve, material flow trajectory) is low, the learning stability and explainability of multi-agent collaboration strategies under complex production constraints are insufficient, the randomness and propagation effect of abnormal events are difficult to model quantitatively, and the multi-objective trade-off in dynamic environment relies on human experience and lacks self-adaptive optimization mechanism. Under this background, a scheduling method that deeply integrates real-time perception, intelligent decision-making and closed-loop verification is needed to break through the limitations of traditional technology and achieve efficient collaboration of workshop resources and dynamic optimization of production process. SUMMARY

[0005] In order to solve the technical problems of the prior art, such as insufficient dynamic adaptability of workshop production scheduling, difficulty in real-time response to sudden equipment failure, order priority adjustment, material supply fluctuation and other dynamic production conditions, low multi-target coordination efficiency, inability to effectively balance the conflict between energy economy, time efficiency and abnormal fault tolerance, intelligent algorithm being easy to fall into local optimum in the multi-agent collaboration scene and relying on simulation environment lacking high-fidelity modeling capability leading to poor strategy migration effect, lack of full-dimensional data perception system covering equipment, material and environment and "decision-making-execution-verification-optimization" closed loop mechanism, low efficiency of real-time fusion and feature extraction of multi-source heterogeneous data, insufficient stability and explainability of multi-agent collaborative strategy learning under complex production constraints, difficulty in quantitative modeling of abnormal event randomness and propagation effect leading to lack of robustness of scheduling scheme, and multi-target trade-off relying on artificial experience and lacking self-adaptive optimization mechanism in dynamic environment, the present application provides a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions.

[0006] The technical scheme provided by the present application is as follows:

[0007] The workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions provided by the present application comprises:

[0008] S1, real-time production data acquisition: real-time acquisition of processing state data of each production unit through a distributed sensor network, including device operating parameters, material flow progress and environmental monitoring indicators, and construction of a multi-dimensional time series data matrix;

[0009] S2, dynamic scheduling decision modeling: a three-dimensional decision space including time dimension, resource dimension and task dimension is established, and a multi-agent deep deterministic policy gradient algorithm is used to cooperatively schedule the model, each agent corresponding to an independent decision unit;

[0010] S3, real-time production urgency calculation: based on process connection tightness, remaining processing time margin and equipment load balancing degree, the real-time production urgency of each production batch is dynamically calculated;

[0011] S4, collaborative efficiency factor generation: by analyzing the cross-unit material flow path, device collaborative working mode and abnormal event propagation path, a multi-dimensional collaborative efficiency factor is generated;

[0012] S5, online scheduling decision optimization: taking the real-time production urgency and the collaborative efficiency factor as state input features, the agent exploration rate is dynamically adjusted through an improved curriculum learning strategy, and a joint scheduling scheme including process sequencing, device allocation and material path is output;

[0013] S6, scheduling scheme verification and feedback: a virtual twin simulation environment is established, multi-objective verification is carried out on the generated scheduling scheme, the verification indexes include energy economy, time efficiency and abnormal fault tolerance, and the verification result is fed back to the scheduling model for online parameter updating.

[0014] The technical scheme provided by the application has at least the following beneficial effects:

[0015] (1) In the application, multi-dimensional production data is collected in real time through a distributed sensor network, a three-dimensional decision space including time, resources and tasks is constructed, and dynamic calculation of real-time production urgency and collaborative efficiency factors is combined, so that the scheduling system can perceive changes in device state, material flow and environmental parameters in real time, effectively cope with dynamic production conditions such as device sudden failure and order priority adjustment, and break through the lagging limitations of traditional static scheduling;

[0016] (2) In the application, an improved multi-agent deep reinforcement learning algorithm is used, an attention mechanism and a variance constraint term are introduced to optimize the agent cooperation strategy, and a curriculum learning strategy is used to dynamically adjust the exploration rate, so that adaptive balance can be achieved between time efficiency, energy economy and abnormal fault tolerance, and the risk of production waste or chain failure caused by single target optimization is avoided, and the resource collaborative efficiency under complex production constraints is improved;

[0017] (3) In the application, a virtual twin simulation environment is established to verify the scheduling scheme, comprehensive evaluation and Pareto frontier analysis are carried out on indexes such as energy economy, time efficiency and abnormal fault tolerance, a closed-loop feedback mechanism of "decision-execution-verification-optimization" is formed, the problem of disconnection between offline verification of traditional scheduling system and actual production is solved, online iterative optimization of the scheduling model is realized, and the robustness and long-term adaptability of the scheme are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A flowchart of a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions is provided for the embodiments of the application.

[0020] Figure 2 A flowchart of a three-dimensional decision space construction method of a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions is provided for the embodiments of the application.

[0021] Figure 3 A real-time production urgency P calculation method process schematic diagram of a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions is provided for an embodiment of the present application.

[0022] Figure 4 A collaborative efficiency factor CE calculation method process schematic diagram of a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the present application will be described below with reference to the drawings.

[0024] In the embodiments of the present application, the words such as "example", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0025] In the embodiments of the present application, "image" and "picture" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent. "Of", "corresponding" and "corresponding" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent.

[0026] In the embodiments of the present application, sometimes the subscript such as may be mistakenly used in the form of non-subscript such as W1. When the distinction is not emphasized, the meanings expressed are consistent.

[0027] To make the technical problems, technical solutions and advantages to be solved by the present application clearer, specific embodiments will be described in detail below with reference to the drawings.

[0028] Reference is made to the drawings accompanying the specification, which show a process schematic diagram of a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions provided by an embodiment of the present application. Figure 1

[0029] The present application provides a workshop multi-agent deep reinforcement learning scheduling method based on real-time production conditions. The processing flow can include the following steps:

[0030] ​S1, real-time production data collection: real-time collection of processing state data of each production unit through a distributed sensor network, including equipment operating parameters, material flow progress and environmental monitoring indicators, to construct a multi-dimensional time series data matrix.

[0031] It should be noted that the multi-dimensional time series data matrix is constructed by a distributed sensor network, and the core function is to realize comprehensive situation awareness of the production site. The distributed architecture avoids single point failure risk, and the introduction of environmental monitoring indicators can capture parameters such as temperature and humidity that have an implicit impact on processing quality, providing multi-physical field coupling data support for subsequent decision-making. Compared with traditional single-dimensional data collection, this method can effectively retain the dynamic correlation characteristics in the production process through time series data organization, laying a data foundation for building accurate scheduling decision models.

[0032] S2, dynamic scheduling decision modeling: a three-dimensional decision space including time dimension, resource dimension and task dimension is established, and a multi-agent deep deterministic policy gradient (MADDPG) algorithm is used to coordinate the scheduling model. Each agent corresponds to an independent decision unit.

[0033] It should be noted that this step constructs a three-dimensional decision space, which orthogonally decomposes the three dimensions of time, resources and tasks. The sliding window mechanism of the time dimension realizes the organic combination of historical experience and future prediction, the device capacity vector of the resource dimension first introduces the maintenance demand index and the temperature influence coefficient into the quantitative evaluation system, and the feature matrix of the task dimension breaks through the limitation of traditional scheduling methods only considering delivery period. The MADDPG algorithm is used to construct a collaborative scheduling model, which effectively solves the contradiction between device resource competition and cooperation through a multi-agent architecture, providing an algorithm framework for distributed decision-making in complex production environments.

[0034] In one possible implementation, as shown in Figure 2 The construction method of the three-dimensional decision space in S2 includes:

[0035] S201, the time dimension adopts a sliding window mechanism, and the historical data of the time period is intercepted forward from the current time as the reference, and the scheduling demand of the time period is predicted backward;

[0036] S202, the resource dimension establishes a device capacity vector E=( , ,..., ), wherein , is the real-time utilization rate of device i, ​​a maintenance demand index, a temperature influence coefficient, , , a dynamic weight parameter;

[0037] S203, constructing a feature matrix in a task dimension wherein represents a quantized value of the task j in the i-th feature dimension, and the feature dimension includes process complexity, quality requirement, and delivery period sensitivity.

[0038] It can be understood that the structured solution element improves the model interpretability. The time dimension sliding window mechanism (S201) combines historical data analysis and future demand prediction to realize dynamic time modeling. The adaptive adjustment mechanism of the window length can match different production rhythm requirements. The resource dimension equipment capacity vector (S202) breaks through the traditional single evaluation mode of utilization rate, introduces the maintenance demand index and the temperature influence coefficient, wherein the maintenance demand index integrates the equipment vibration spectrum characteristics and the maintenance cycle data, and the temperature influence coefficient is related to the material thermal expansion coefficient and the machining precision requirement. The task dimension feature matrix (S203) quantifies the processing difficulty level through the process complexity, maps the detection standard strictness through the quality requirement dimension, and introduces the customer level weight through the delivery period sensitivity, forming a task portrait system facing multiple constraint conditions, and providing structured input for subsequent agent decision-making.

[0039] It should be noted that the dynamic weight parameter in S202 specifically includes:

[0040] dynamic weight parameter , , The determination method includes:

[0041] S501, constructing a judgment matrix wherein represents the importance of the x-th index relative to the y-th index;

[0042] S502, calculating information entropy , is a standardization coefficient;

[0043] S503, calculating the difference coefficient ;

[0044] S504, the maximum weight , , .

[0045] It can be understood that the traditional method is improved to realize data-driven dynamic weight allocation. The judgment matrix construction (S501) adopts an improved three-scale method (1-3-5 scale), and the relative importance scores between indexes are automatically generated through the equipment operation log, so as to eliminate subjective judgment deviation. The information entropy calculation (S502) introduces Laplace smoothing to avoid zero probability problem, and the standardization coefficient k=1 / ln(3) ensures that the entropy value interval is reasonable. The difference coefficient conversion (S503) quantifies the information uncertainty as the basis for weight allocation, and the final normalization processing (S504) ensures the mathematical completeness of the weight parameter. The method makes the weight ratio of equipment utilization, maintenance demand and temperature influence reflect the current production environment characteristics in real time, automatically increases the temperature coefficient weight in high temperature season, and strengthens the maintenance demand weight in the equipment aging stage.

[0046] It should be noted that the improvement of the multi-agent deep deterministic policy gradient algorithm MADDPG algorithm in S2 includes:

[0047] S801, an attention mechanism is added in the critic network to calculate the correlation weight between agents:

[0048]

[0049] wherein, is the query vector of the current agent, is the key vector of other agents, is the vector dimension;

[0050] S802, a double-delay update strategy is adopted, the main network parameter update period =1000 steps, the target network update period =2000 steps;

[0051] S803, a variance constraint term is introduced in the policy gradient: , , t is the current training step, is the maximum training step.

[0052] It can be understood that the multi-agent collaboration efficiency is improved through algorithm-level innovation. The attention mechanism (S801) adopts a multi-head attention structure (4 heads), and the d=64-dimensional vector space ensures feature decoupling, and the Q and K matrices are generated by splicing the device type code and the real-time state. The double-delay update strategy (S802) sets the main network update step length η=0.001, the target network soft update coefficient τ=0.005, and balances the learning speed and stability. The variance constraint term (S803) introduces moving average variance calculation, the window size is set to 100 experience samples, the β coefficient linearly decays with the training progress, encourages exploration in the early stage (β=0.1), and focuses on utilization in the later stage (β=0.01).

[0053] S3, real-time production urgency calculation: based on process coupling tightness, remaining processing time margin and equipment load balancing, dynamically calculate the real-time production urgency of each production batch.

[0054] It should be noted that the three-dimensional urgency evaluation system is creatively proposed in this step, which solves the problem of rigid priority setting in traditional scheduling methods. Process coupling tightness quantifies the coupling relationship in the production process, remaining processing time margin introduces an exponential decay function to enhance time sensitivity, and equipment load balancing uses standard deviation to avoid local overload. Through the adaptive adjustment of dynamic weight coefficient, the dynamic balance between emergency order response and system stable operation is realized. The calculation of this parameter provides a quantitative decision basis for subsequent scheduling optimization, making the system have the flexibility to respond to unexpected situations.

[0055] In one possible implementation, as shown in Figure 3 The calculation method of real-time production urgency P includes:

[0056] S301, calculate the time urgency component: , is the remaining processing time, is the maximum allowed delay time, , is the process half-life;

[0057] S302, calculate the coupling component: , is the coupling degree of the current process and the downstream process, is the threshold coupling degree;

[0058] S303, calculate the load balancing component: , is the standard deviation of the load of the same type of equipment, , is the reference load difference value;

[0059] S304, synthesize the total urgency: , , , is the dynamic weight coefficient, satisfying and > 0.

[0060] It can be understood that the establishment of a sub-item weighted model solves the problem of multi-source parameter fusion. The time urgency component (S301) uses an exponential decay function to simulate the nonlinear change of process urgency, and the process half-life The setting makes different process types (such as heat treatment and machining) present different attenuation curves. The coupling component (S302) eliminates the inherent coupling difference between processes through threshold normalization processing, and the logarithmic function compresses the extreme value influence to avoid decision shock. The load balancing component (S303) innovatively uses the relative standard deviation measurement, and the benchmark load difference value According to the dynamic calculation of the equipment group capacity, the evaluation deviation caused by the heterogeneity of the equipment is eliminated. The dynamic weight coefficient (S304) introduces a hidden Markov model to track the production state in real time, automatically increases the time weight during the order peak period, and strengthens the load balancing weight during the equipment failure period, so as to realize intelligent weight migration.

[0061] S4, synergy efficiency factor generation: by analyzing the cross-unit material flow transfer path, equipment synergy working mode and abnormal event propagation path, a multi-dimensional synergy efficiency factor is generated.

[0062] It should be noted that this step is aimed at the cross-unit synergy scheduling problem, and a multi-dimensional evaluation model integrating path optimization, resource synergy and abnormal propagation is proposed. The path optimization potential value quantifies the material flow transfer efficiency improvement space, the resource synergy benefit reveals the equipment utilization rate improvement potential, and the abnormal influence factor assesses the risk transmission effect through a probability multiplication model. The min-max normalization strategy is used to process the fusion problem of different dimension parameters, and the dynamic adjustment mechanism of the balance coefficient η makes the system automatically focus on the efficiency or stability target in different production stages, which significantly improves the overall synergy performance of the complex production network.

[0063] In one possible implementation, as shown in Figure 4 The calculation method of the synergy efficiency factor CE in S4 includes:

[0064] S401, calculate the path optimization potential value: , wherein is the original path length, is the optimized path length;

[0065] S402, calculate the resource synergy benefit: , is the theoretical maximum utilization rate of the equipment, is the current utilization rate;

[0066] S403, calculate the abnormal influence factor: , is the occurrence probability of the kth abnormal event, is the influence coefficient of the corresponding abnormality;

[0067] S404, generate the final synergy efficiency factor: , wherein , is the current cumulative processing time, As a time threshold, , For benefit weight and .

[0068] Understandably, constructing a multi-path collaborative evaluation model addresses the efficiency-robustness trade-off. The path optimization potential value (S401) uses a relative improvement rate as a metric to avoid the dimensional differences between different transport vehicles, where the optimized path length... pass A hybrid strategy combining algorithm and tabu search is used for generation. Resource synergy benefits (S402) are incorporated to maximize theoretical utilization. Its value is dynamically adjusted based on the historical peak value of the equipment's OEE (Overall Equipment Effectiveness). The anomaly impact factor (S403) is constructed through fault tree analysis. and The quantitative relationship and the multiplication formula accurately characterize the risk of chain reactions of multiple anomalies. The final synthesis formula (S404) uses the min function to constrain the synergistic short-board effect, and the sigmoid function design of the balance coefficient η enables the system to automatically focus on anomaly prevention when the cumulative processing time approaches the threshold, forming an environment-adaptive synergistic optimization mechanism.

[0069] S5. Online scheduling decision optimization: Using real-time production urgency and collaborative efficiency factors as state input features, the agent's exploration rate is dynamically adjusted through an improved course learning strategy, and a joint scheduling scheme including process sequencing, equipment allocation, and material path is output.

[0070] It should be noted that this step achieves progressive agent training through an improved course learning strategy, addressing the problem of low exploration efficiency in traditional reinforcement learning. A high exploration rate in the initial stage ensures the learning of the basic scheduling strategy; the introduction of sinusoidal function fluctuations into the dynamic adjustment formula prevents local optima; and the exponential decay mechanism enhances strategy stability in later training stages. By using urgency and coordination factors as joint state features, the agent simultaneously focuses on task urgency and system coordination. The output three-dimensional scheduling scheme (process, equipment, path) forms a complete decision-making loop, ensuring the executability and optimality of the scheme.

[0071] In one possible implementation, the course learning strategy in S5 includes:

[0072] S601, Initial Stage Exploration Rate Setting =0.9, prioritize learning single-device scheduling tasks;

[0073] S602. After the model's scheduling accuracy on the validation set reaches 85% for three consecutive epochs, introduce device collaboration constraints and adjust the exploration rate to... , is the current training step number, is a period parameter;

[0074] S603, when the device utilization rate reaches the 75% threshold, the abnormality processing module is activated, and the exploration rate is reduced to .

[0075] It can be understood that the progressive training framework is designed to solve the complex scheduling learning problem. In the initial stage (S601), the learning difficulty is reduced by simplifying the state space (only containing the basic parameters of the device), and the high exploration rate of 0.9 ensures sufficient traversal of the single-device scheduling strategy. The accuracy triggering mechanism (S602) introduces the early stopping method to prevent overfitting, and the device cooperation constraint condition gradually increases the material buffer limit and the energy consumption threshold. The sinusoidal fluctuation term in the exploration rate dynamic formula breaks the strategy convergence deadlock, and the setting of the period parameter T=5000 steps matches the typical production batch rhythm. The abnormality processing module activation condition (S603) is related to the device MTBF (Mean Time Between Failures) data, and the exponential decay function makes the exploration rate decrease rate negatively related to the device reliability, realizing a risk adaptive exploration strategy.

[0076] S6, scheduling scheme verification and feedback: a virtual twin simulation environment is established to verify the generated scheduling scheme, the verification indexes include energy economy, time efficiency and abnormal fault tolerance, and the verification results are fed back to the scheduling model for online parameter updating.

[0077] It should be noted that the virtual twin simulation environment constructed in this step breaks through the limitations of traditional offline verification, and realizes comprehensive evaluation of the scheme through a multi-objective verification system. The energy economy index is related to the device power and running time, the time efficiency index adopts a normalized delay penalty function, and the abnormal fault tolerance index quantifies the system robustness through weighted frequency. The Pareto frontier analysis in the three-dimensional evaluation space ensures the non-inferiority of the scheme, and the online parameter updating mechanism forms a "decision-verification-optimization" reinforcement learning closed loop, which significantly improves the continuous optimization ability of the scheduling system.

[0078] In one possible implementation, the specific method of multi-objective verification in S6 includes:

[0079] S701, calculating the energy economy index: , is the power of device i, is the running time, is the maximum allowed power consumption of the system;

[0080] S702, calculating the time efficiency index: , is the delivery period, is the actual completion time;

[0081] S703, calculate the abnormal fault tolerance index: , is the weight of the kth abnormality, is the occurrence frequency;

[0082] S704, establish a three-dimensional evaluation space, and screen the non-inferior solution set through Pareto frontier analysis.

[0083] It can be understood that the quantitative evaluation system solution solves the multi-attribute optimization problem. The energy economic index (S701) introduces the power factor correction coefficient, According to the dynamic adjustment of the power distribution capacity of the workshop, the risk of overload is avoided. The time efficiency index (S702) adopts a segmented penalty function, sets a 3-fold penalty weight for key customer orders, and strengthens the VIP guarantee capability. The abnormal fault tolerance index (S703) determines the weight through failure mode and effects analysis (FMEA), and the frequency statistical window is set to a rolling 24-hour period. The three-dimensional evaluation space (S704) uses the NSGA-II algorithm to search for the Pareto frontier, sets the crowding distance threshold value to 0.1 to maintain the diversity of the solution set, and provides a strategy pool containing 5-8 non-inferior solutions for decision makers.

[0084] It should be noted that the construction method of the virtual twin simulation environment in S6 includes:

[0085] S901, establish a digital model of the equipment, import historical processing data to train an LSTM prediction network, and predict the state transition probability of the equipment;

[0086] S902, construct a directed graph G=(V,E) of material flow, where the node V represents the processing site, the edge E represents the transmission path, and the edge weight includes transmission time and energy cost;

[0087] S903, design an abnormal event injection module, generate equipment failure events according to Weibull distribution, and generate material shortage events according to Poisson process;

[0088] S904, set the simulation acceleration factor According to the GPU computing power, dynamically adjust the simulation speed.

[0089] It can be understood that a high-fidelity simulation environment is created to realize closed-loop verification. The device digital model (S901) integrates a physical engine to simulate mechanical transmission errors. The LSTM network input includes 12-dimensional features such as current waveform and noise spectrum. The material flow chart (S902) edge weight calculation fuses time cost (0.5 yuan / minute) and energy cost (1.2 yuan / kWh). The path optimization constraint is a maximum turning angle of 45 degrees. In the abnormal event injection (S903), the Weibull distribution shape parameter β=2.1 simulates wear and tear failure, and the Poisson process intensity λ=0.02 times / minute fits material shortage. The simulation acceleration (S904) uses time scaling technology, and when the GPU memory is sufficient, double-precision floating-point operation is enabled to ensure numerical stability.

[0090] In one possible implementation, a dynamic parameter adjustment method is further included.

[0091] S1001, real-time monitoring of device temperature fluctuation rate , is the historical average temperature;

[0092] S1002, when >0.15, triggering an emergency cooling strategy, inserting a forced maintenance time window in the scheduling scheme;

[0093] S1003, dynamically adjusting the process priority weight: , is the base weight;

[0094] S1004, updating the time threshold in the calculation of the synergy efficiency factor , t is the cumulative running time.

[0095] It can be understood that an environment-responsive adaptive mechanism is constructed. Temperature monitoring (S1001) uses a sliding Z-score algorithm to detect abnormal fluctuations, with a window size N=30 minutes and a threshold Z=2.5. The forced maintenance strategy (S1002) inserts a 15-minute cooling time window, during which the standby device takes over the task. Priority adjustment (S1003) sets the base weight =0.7, and the temperature compensation coefficient is limited to the interval [0.7, 1.3] to prevent weight out of control. Time threshold decay (S1004) introduces a learning rate decay mechanism, reducing the initial threshold by 5% every 1000 steps, but sets the minimum threshold =0.6 to avoid excessive compression of time margin.

[0096] The technical scheme provided by the embodiment of the application has at least the following beneficial effects:

[0097] (1) In the present application, multi-dimensional production data is collected in real time by a distributed sensor network, a three-dimensional decision space containing time, resources and tasks is constructed, and the dynamic calculation of real-time production urgency and collaborative efficiency factors enables the scheduling system to perceive the changes of equipment state, material flow and environmental parameters in real time, effectively cope with dynamic production conditions such as equipment sudden failure and order priority adjustment, and break through the lagging limitations of traditional static scheduling;

[0098] (2) In the present application, an improved multi-agent deep reinforcement learning algorithm is used, the attention mechanism and variance constraint term are introduced to optimize the agent cooperation strategy, and the exploration rate is dynamically adjusted combined with the curriculum learning strategy, which can realize adaptive balance between time efficiency, energy economy and abnormal fault tolerance, avoid the waste of production capacity or the risk of chain failure caused by single target optimization, and improve the resource collaborative efficiency under complex production constraints;

[0099] (3) In the present application, a virtual twin simulation environment is established to verify the scheduling scheme, through the comprehensive evaluation and Pareto frontier analysis of indicators such as energy economy, time efficiency and abnormal fault tolerance, a closed-loop feedback mechanism of "decision-execution-verification-optimization" is formed, solving the problem that the traditional scheduling system is out of touch with actual production, realizing online iterative optimization of the scheduling model, and significantly improving the robustness and long-term adaptability of the scheme.

[0100] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0101] The following points need to be explained:

[0102] (1) The drawings of the embodiments of the present application only involve the structures involved in the embodiments of the present application, and other structures can refer to the usual design.

[0103] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present application, the thickness of the layers or regions is exaggerated or reduced, that is, the drawings are not drawn according to the actual proportion. It can be understood that when elements such as layers, films, regions or substrates are referred to as being "on" or "under" another element, the element can be "directly" on or under another element or there can be an intermediate element.

[0104] (3) In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other to obtain new embodiments.

[0105] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A real-time production status based workshop multi-agent deep reinforcement learning scheduling method, characterized in that, The application relates to a real-time production scheduling method based on multi-agent deep reinforcement learning. The method comprises the following steps: S1, real-time production data acquisition: real-time acquisition of processing state data of each production unit through a distributed sensor network, including equipment operation parameters, material flow progress and environmental monitoring indexes, to construct a multi-dimensional time sequence data matrix; S2, dynamic scheduling decision modeling: a three-dimensional decision space is established, which comprises a time dimension, a resource dimension and a task dimension; a multi-agent deep deterministic policy gradient algorithm is used to construct a collaborative scheduling model, and each agent corresponds to an independent decision unit; S201、time dimension adopts sliding window mechanism, intercepts forwardly with current time as benchmark history data of time period, backward prediction schedule demand of time period; S202、resource dimension establishment device capability vector E=( , ,..., ),wherein , is the real-time utilization rate of device i, is the maintenance demand index, is the temperature influence coefficient, , , is the dynamic weight parameter; S203, constructing a feature matrix in a task dimension wherein denotes the quantized value of task j in the i-th feature dimension, the feature dimensions including process complexity, quality requirement, and delivery period sensitivity; The method for constructing the three-dimensional decision space in S2 comprises the following steps: S3, real-time production urgency calculation: based on three dimensions of process connection tightness, remaining processing time margin and equipment load balancing degree, the real-time production urgency of each production batch is dynamically calculated; the synergic efficiency factor: wherein , is the current cumulative processing time, is a time threshold, , is a benefit weight and , denotes a path optimization potential value, denotes a resource synergic benefit, denotes an anomaly impact factor; S4, collaborative efficiency factor generation: by analyzing cross-unit material flow transfer paths, equipment collaborative working modes and abnormal event propagation paths, multi-dimensional collaborative efficiency factors are generated; S5, online scheduling decision optimization: taking the real-time production urgency and the collaborative efficiency factor as state input characteristics, an improved curriculum learning strategy is used to dynamically adjust the agent exploration rate, and a joint scheduling scheme comprising process sequencing, equipment allocation and material path is outputted; 2. The method of claim 1, wherein, S6, scheduling scheme verification and feedback: a virtual twin simulation environment is established, the generated scheduling scheme is subjected to multi-objective verification, the verification indexes comprise energy consumption economy, time efficiency and abnormal fault tolerance, and the verification result is fed back to the scheduling model for online parameter updating. S301、calculating a time urgency component: wherein is the remaining processing time, is the allowed maximum delay time, , is the process half-life; S302, calculate the coupling degree component: , is the coupling degree of the current process and the downstream process, is the threshold coupling degree; S303、Calculate the load balancing component: , is the load standard deviation of the same type of device, , is the reference load difference value; S304、Synthesize total urgency: wherein , , is a dynamic weight coefficient, satisfying and >

0. 3.The workshop multi-agent deep reinforcement learning scheduling method based on real-time production status according to claim 1, characterized in that, The method for calculating the real-time production urgency P comprises the following steps: S401、Calculate the path optimization potential value: wherein is the original path length, is the optimized path length; S402, computing resource synergy benefit: , is the theoretical maximum utilization of the device, is the current utilization rate; S403、calculate the abnormal influence factor: , is the probability of occurrence of the kth abnormal event, is the influence coefficient corresponding to the abnormality.

4. The method of claim 1, wherein, The method for calculating the collaborative efficiency factor CE in S4 comprises the following steps: The dynamic weight parameter , , The determination method comprises: S501、constructing a judgment matrix wherein represents the importance of the xth index relative to the yth index; S502, calculate information entropy , is a standardization coefficient; S503、Calculate the difference coefficient ; S504, the most weight , , .

5. The method of claim 1, wherein, The dynamic weight parameter in S202 comprises the following steps: S601, set the exploration rate in the initial stage = 0.9, prefer to learn single-device scheduling tasks; S602、When the model reaches 85% accuracy for 3 consecutive epochs in the validation set, introduce device coordination constraints and adjust the exploration rate to , is the current training step, is the period parameter; S603, when the device utilization reaches the 75% threshold, activate the anomaly handling module, while reducing the exploration rate to .

6. The method of claim 1, wherein, The implementation of the curriculum learning strategy in S5 comprises the following steps: S701, calculate the energy consumption economy index: , is the power of the device i, is the running time, is the maximum allowed power consumption of the system; S702, calculate time efficiency index: , is the delivery time, is the actual completion time; S703, calculate the abnormal fault tolerance index: , is the weight of the kth type of anomaly, is the occurrence frequency; The specific method for multi-objective verification in S6 comprises the following steps:

7. The method of claim 1, wherein, S704, a three-dimensional evaluation space is established, and a non-inferior solution set is screened through Pareto frontier analysis. The improvement of the multi-agent deep deterministic policy gradient algorithm in S2 comprises the following steps: wherein, is a query vector of the current agent, is a key vector of other agents, is a vector dimension; S802, adopt double delay update strategy, main network parameter update period =1000 steps, target network update period =2000 steps; S803, introduce a variance constraint term in the policy gradient: , , t is the current training step, is the maximum training step. 8.The workshop multi-agent deep reinforcement learning scheduling method based on real-time production status according to claim 1, wherein, S801, an attention mechanism is added to a critic network to calculate the correlation weight between agents: The method for constructing the virtual twin simulation environment in S6 comprises the following steps: S901, a digital equipment model is established, historical processing data is imported to train an LSTM prediction network, and an equipment state transition probability is predicted; S902, a material flow transfer directed graph G=(V, E) is constructed, a node V represents a processing site, an edge E represents a transmission path, and an edge weight comprises transmission time and energy consumption cost; S904, setting an emulation acceleration factor , dynamically adjusting the emulation speed according to the GPU computing capability. 9.The workshop multi-agent deep reinforcement learning scheduling method based on real-time production status according to claim 1, wherein, S903, an abnormal event injection module is designed, equipment failure events are generated according to a Weibull distribution, and material shortage events are generated according to a Poisson process; The method further comprises a dynamic parameter adjustment method. S1001、Real-time monitoring of device temperature fluctuation rate , is the historical average temperature; S1002、when > 0.15, an emergency cooling strategy is triggered, and a mandatory maintenance time window is inserted in the dispatching scheme; S1003, dynamically adjusting the process priority weight: , is the base weight; S1004, updating the time threshold in the synergy efficiency factor calculation t is the cumulative run time.

Images