A production scheduling method based on hierarchical strategy model and lean management

Abstract

The application provides a production scheduling method based on a hierarchical strategy model and lean management, comprising the following steps: constructing a hierarchical strategy model comprising a high-level strategy model and a low-level strategy model; constructing a global state vector according to the running data of a production workshop; the high-level strategy model generates an initial probability distribution for all action options according to the global state vector, calculates the priority of each action option, adaptively adjusts the initial probability distribution of all action options according to the priority, and outputs the current optimal action option according to the adjusted probability distribution; the low-level strategy model generates specific actions comprising equipment processing speed, transportation frequency of a transport vehicle and buffer zone capacity adjustment ratio according to the action option output by the high-level strategy model; and the cumulative sequence of the action options selected by the high-level strategy model at each time step is integrated with the continuous control action sequence generated by the low-level strategy model, so as to form a final production scheduling scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of production scheduling and intelligent manufacturing technology, in particular to a production scheduling method based on a hierarchical strategy model and lean management. BACKGROUND

[0002] Under the promotion of intelligent manufacturing, production scheduling is evolving from static rule-driven to dynamic intelligent decision-making. Facing dynamic complexity scenarios such as multi-variety small batch, urgent order insertion and equipment disturbance, the traditional scheduling system exposes structural contradictions such as difficulty in real-time updating of experience-rules, localized decision-making perspective, and high cost of cross-scenario migration, which urgently need methodological upgrading to adapt to rapidly changing demands.

[0003] As the core paradigm of traditional industrial optimization, the production scheduling method of lean management significantly improves efficiency and resource utilization in a stable production environment through standardized processes and experience accumulation; its core value lies in accurate identification and elimination of waste. With value stream optimization, Kanban system pull and balanced production, enterprises can reduce inventory backlog, shorten delivery cycle and strengthen inter-departmental collaboration, especially in scenarios with high repeatability and small demand fluctuations, which can form a replicable optimization paradigm.

[0004] However, the lean management methodology has been limited by static rule base and artificial experience dependence for a long time. With the evolution of production environment towards small batch customization and high frequency disturbance, its limitations have gradually emerged: dependence on preset rules and artificial experience leads to dynamic response lag, making it difficult to respond to sudden situations such as equipment failure or urgent order insertion; local optimization logic lacks a global perspective, and multi-objective coordination is prone to suboptimal solutions; cross-scenario migration highly depends on expert experience refinement, with long adaptation period and high cost for new products or new production lines. The current lean scheduling exposes two bottlenecks of dynamic response lag and multi-objective optimization contradiction, which directly restrict its applicability in the era of intelligent manufacturing. SUMMARY

[0005] The present application proposes a production scheduling method based on a hierarchical strategy model and lean management, which solves the problem of relying on preset rules and artificial experience in the prior art, making it difficult to respond to sudden situations.

[0006] To solve the above technical problems, the present application provides a production scheduling method based on a hierarchical strategy model and lean management, comprising the following steps:

[0007] Step S1: Construct a hierarchical strategy model including a high-level strategy model and a low-level strategy model;

[0008] Step S2: Collecting the running data of the production workshop from the manufacturing system, constructing a global state vector, the high-level strategy model generating an initial probability distribution for all action options according to the global state vector, calculating the priority of each action option according to the relative value gap, exploration potential and maturity discount, adaptively adjusting the initial probability distribution of all action options according to the priority, and the high-level strategy model outputting the current optimal action option according to the adjusted probability distribution;

[0009] Step S3: The low-level strategy model generates specific actions including equipment processing speed, transportation frequency of the transport vehicle and buffer capacity adjustment ratio according to the action option output by the high-level strategy model;

[0010] Step S4: Accumulating the action options selected by the high-level strategy model at each time step into an action option sequence, accumulating the corresponding continuous control actions generated by the low-level strategy model at each time step into a control action sequence, and forming a production scheduling scheme for the multi-variety small-batch production scene.

[0011] Preferably, in step S2, when the high-level strategy model generates an initial probability distribution for all action options according to the global state vector, a reward function is constructed based on the equipment comprehensive efficiency, order delivery rate, energy consumption cost and switching penalty, the high-level strategy model calculates the initial probability distribution of all action options according to the reward function, and the expression of the reward function of the high-level strategy model is:

[0012] ;

[0013] ;

[0014] ;

[0015] ;

[0016] ;

[0017] In the formula, is the reward function of the high-level strategy model; is the weight coefficient of the equipment comprehensive efficiency, order delivery rate, energy consumption cost and switching penalty; is the equipment comprehensive efficiency.

[0018] Preferably, in step S3, the expression for calculating the priority of each action option according to the relative value gap, exploration potential and maturity discount is:

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] In the formula, Action options In state Priority; The maximum Q value for all action options; Action options for the current moment In state Q value; , This is a learnable adjustment coefficient; To explore potential indicators; Maturity factor; Action options for the current moment In state The Q value.

[0024] Preferably, the expression for adaptively adjusting the initial probability distribution of all action options according to the priority in step S3 is:

[0025] ;

[0026] ;

[0027] In the formula, This is the revised high-level strategy; This is the original high-level strategy; To dynamically explore the intensity of incentives; Action options In state Exploration priority; To switch the penalty coefficient; For the indicator function, if the action option The value is 1 if the device was recently switched to, otherwise it is 0. The learning rate; The switching frequency of action options in the last 100 steps; Switch frequencies for the target.

[0028] Preferably, the intensity of the dynamic exploration incentive According to recent The difference between the average reward of each step and the historical average reward Perform adaptive adjustments:

[0029] ;

[0030] In the formula, The learning rate; Reward the target.

[0031] Preferably, the low-level strategy model in step S1 is a deep deterministic policy gradient network (DDPG), which includes an actor network for generating actions and a critic network for evaluating action values.

[0032] Preferably, the actor network adopts a two-way branch design, one way inputs order queue features into a long short-term memory network (LSTM), and the other way inputs device comprehensive efficiency and production line beat features into a fully connected layer, and the outputs of the two ways are spliced to generate the continuous control action.

[0033] Preferably, after constructing the hierarchical strategy model in step S1, the hierarchical strategy model is trained in stages, including the following steps:

[0034] Pre-training stage: freeze the high-level strategy model and only train the low-level strategy model;

[0035] Joint training stage: after the low-level strategy model converges, unfreeze the high-level strategy model, and jointly optimize the high-level strategy model and the low-level strategy model.

[0036] Preferably, a partitioned priority experience replay pool is used in the process of the staged training, which includes a high-value sample area and a regular sample area, the high-value sample area is used to store interaction trajectory samples related to device failure, order emergency change and logistics bottleneck, and the regular sample area uses time difference error as a priority standard, and gives the samples in the high-value sample area a higher sampling weight than the regular sample area.

[0037] Preferably, the joint training stage in the staged training adopts a method of gradually increasing the complexity of the environment, and the complexity of the environment gradually increases from a level including only one processing device and fixed order input to a level including multiple stations, unexpected events and order demand fluctuations.

[0038] The beneficial effects of the present application at least include:

[0039] 1. The high-level strategy model can grasp the overall direction of production scheduling from a macro perspective, select action options based on key indicators such as device comprehensive efficiency, order delivery rate, energy consumption cost and switching penalty, focus on long-term benefits and overall optimization; the low-level strategy model focuses on specific production operation details, generates specific actions including device processing speed, transportation frequency of transportation vehicles and buffer capacity adjustment ratio according to high-level decision, realizes local precise control and optimization, thereby effectively combines global target and local execution, improves the comprehensiveness and effectiveness of the scheduling scheme;

[0040] 2. The high-level policy model can adaptively adjust the output action options according to the priority. This adaptive adjustment mechanism enables the model to dynamically optimize the scheduling strategy according to the changes in the production environment and the feedback of real-time information, without frequent manual intervention, improving the flexibility and intelligence of the model, and enabling it to better cope with uncertain factors such as frequent order changes and equipment failures in multi-variety and small-batch production scenarios;

[0041] 3. In the high-level policy model, the priority of each action option is calculated based on the relative value gap, exploration potential and maturity discount. This method can comprehensively weigh the current value, potential value and risk of the action options, providing a more scientific and accurate basis for the selection of action options, enabling the model to better balance between deterministic and exploratory decisions, and improving the quality and adaptability of the decisions. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The method flowchart of the embodiment of the present application is shown in the figure.

[0043] Figure 2 The interaction diagram of the high-level policy model and the low-level policy model in the hierarchical policy model of the embodiment of the present application is shown in the figure.

[0044] Figure 3 The flowchart of constructing and training the hierarchical policy model in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0046] As shown in the figure, the embodiment of the present application provides a production scheduling method based on hierarchical policy model and lean management, including the following steps: Figure 1

[0047] Step S1: Construct a hierarchical policy model including a high-level policy model and a low-level policy model.

[0048] Specifically, the hierarchical policy model including a high-level policy model and a low-level policy model is constructed, i.e. a two-level policy architecture is adopted to realize production scheduling optimization. As shown in the figure, Figure 2 ​As shown, the high-level policy model is responsible for macro decision-making, and the Option-Critic architecture is used to divide the scheduling policy into several discrete action options Options, each of which represents a macroscopic and goal-oriented sub-policy, such as an emergency order handling mode or a device maintenance mode. The low-level policy model is responsible for micro execution, and outputs specific continuous control action sequences according to the high-level selected Option, so as to decompose the complex scheduling task into two levels of high-level policy selection and low-level action execution.

[0049] This hierarchical design reduces the complexity of the large-scale state space and action space faced by a single policy model, and improves the adaptability of the model to dynamic production environments.

[0050] Step S2: Collecting operation data of the production workshop from the manufacturing system, constructing a global state vector, and the high-level policy model generating an initial probability distribution for all action options according to the global state vector, calculating the priority of each action option according to the relative value gap, exploration potential and maturity discount, and adaptively adjusting the initial probability distribution of all action options according to the priority, and the high-level policy model outputting the current optimal action option according to the adjusted probability distribution.

[0051] Specifically, for the high-level policy model, order, production plan, energy and cost, and inventory data are collected from the manufacturing system MES, ERP, EMS and WMS. The high-level model data is preprocessed and feature extracted to obtain order features, device features, energy consumption features and production features, and a global state vector is constructed.

[0052] Global state vector The global state vector includes order features, device features, energy consumption features and production features, wherein the order features include: normalized number of emergency orders, average remaining delivery time of orders; the device features include: overall equipment effectiveness OEE, fault prediction probability, average load rate; the energy consumption features include: current period electricity price, total energy consumption limit remaining, single piece energy consumption; the production features include: daily plan completion rate, work-in-process WIP, standardized inventory. The number of Options is set to 5, and one-hot encoding is used to represent the Options, and the global state vector and the Option encoding are spliced to serve as the joint input of the high-level policy model. The global state vector constructed in the embodiment of the present application As shown in Table 1.

[0053] Table 1 Global feature vector

[0054]

[0055] The high-level policy model includes a shared feature extractor, an upper policy network, and a termination discriminant network.

[0056] Shared feature extractor: responsible for extracting shared semantic representation from global state vector, for all option networks, its structure is fully connected layer and ReLU activation function, input dimension is state_dim, output dimension is 128;

[0057] Upper policy network: according to the shared feature, decide which action option should be selected in the current state, its structure is two layers, the first layer is fully connected layer and ReLU activation function, input dimension is 128, output dimension is 64, the second layer is fully connected layer, input dimension is 64, output dimension is N, N is the number of options.

[0058] Termination discriminator network: evaluate whether the executing option should be terminated in the current state, in order to switch to a more suitable option, its structure is two layers, the first layer is fully connected layer and a ReLU activation function, input dimension is 128, output dimension is 64, and a Sigmoid activation function is composed. The second layer is fully connected layer and Sigmoid activation function, which is composed of a linear layer to reduce the dimension from 64 to 1, and finally the output is compressed to 0 to 1 through Sigmoid activation function.

[0059] In addition, the high-level policy model of the embodiment of the application adopts a fully connected neural network structure, which contains three hidden layers: the first two layers use ReLU activation function, and the third layer uses tanh activation function, so as to ensure that the output policy changes smoothly within a certain range; the network weight parameter initialization adopts Xavier initialization method to maintain the stability of training.

[0060] Through the above design, the high-level policy model can generate the probability distribution of each Option policy based on the current global state vector, and evaluate the value of each Option by using the Critic network. The output Option of the high level represents the selection of a set of macro production scheduling strategies, for example, the mode selection in production planning, device load allocation, energy use strategy, etc.

[0061] In order to realize multi-objective optimization, the embodiment of the application designs a comprehensive multi-objective reward function in the high-level policy model, to evaluate the pros and cons of each Option by weighted sum of four types of indicators: among them, the device comprehensive efficiency and the order on-time delivery rate are positive income, and the energy consumption cost and the Option switching penalty are negative cost:

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] In the formula, This is the reward function for the high-level strategy model; In this embodiment of the invention, weighting coefficients are set for overall equipment efficiency, order delivery rate, energy consumption cost, and switching penalty. ; For overall equipment efficiency.

[0068] Based on this, two key value functions are constructed:

[0069] Option-Value Function Used to assess the state Select action options below The long-term returns.

[0070] Action-Value Function Used to evaluate action options Execute specific actions The value of.

[0071] Gradient optimization is employed to optimize the policy output by a high-level policy model. The goal of gradient optimization is to optimize the internal policy parameters θ of the high-level policy model using the policy gradient theorem, maximizing the expected reward of each option. Simultaneously, the termination condition parameters are optimized using the termination gradient theorem. If the current option performs poorly, its termination probability is adaptively increased, thereby creating an opportunity to switch to a more promising strategy.

[0072] The explore-exploitation dilemma in reinforcement learning is a challenging problem, where an agent needs to strike a balance between exploring unknown strategies to find better solutions and exploiting known optimal strategies to obtain maximum rewards. To address this, this invention presents a highly intelligent explore-exploitation coordinator.

[0073] The core principle of this coordinator is to intelligently allocate limited exploration resources to the most promising, least mature, and currently performing sub-policies with the largest gap from the best level. This enables the model to perform meta-inference on its own knowledge boundaries, thereby achieving a purposeful, self-directed learning process that is far more efficient than simple random exploration.

[0074] The coordinator quantifies the exploration value of each option by a priority calculator, which includes three parts, relative value gap, exploration potential indicator and maturity discount. The relative value gap is used to measure the performance gap between the current strategy and the optimal strategy. The exploration potential indicator is a composite indicator determined by the unexplored state density, state access entropy and Q value sensitivity. When the model encounters an unknown and never-accessed state, the unexplored state density will soar, triggering the system to allocate resources to the related options and start focused exploration. The maturity discount is used to suppress the priority of an option when it has been fully trained and utilized, preventing the waste of computing resources on repeated learning. The final expression of the priority is:

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] wherein, is the priority of action option in state ; is the maximum Q value of all action options; is the Q value of action option in state at the current moment; , are learnable adjustment coefficients; is the exploration potential indicator; is the maturity factor; is the Q value of action option in state at the current moment.

[0080] wherein the unexplored state density is the proportion of states not accessed by the current Option; the state access entropy is the Shannon entropy of the historical state distribution, and the higher the entropy, the more uniform the exploration; and the Q value sensitivity is the standard deviation of the recent Q value.

[0081] The priority calculator does not directly control the strategy selection, but dynamically adjusts the probability distribution of the high-level strategy through a strategy correction function:

[0082] ;

[0083] wherein, is the corrected high-level strategy; is the original high-level strategy; The initial value of the dynamic exploration incentive strength is 0.5; The action option In the state The exploration priority; The switching penalty coefficient; The indicator function, if the action option Is switched recently, it is 1, otherwise it is 0.

[0084] Where the dynamic exploration incentive strength According to the difference between the average reward of the last Steps and the historical average reward Adaptive adjustment is made:

[0085] ;

[0086] In the formula, The learning rate; The target reward.

[0087] When the average reward obtained in the near future is improved, The value will increase, encouraging the model to continue valuable exploration; otherwise it will decrease. This mechanism ensures that high-value exploration behavior can be strengthened in time, and ineffective exploration will be inhibited.

[0088] In addition, in order to prevent the strategy from frequently oscillating in a complex environment, a switching penalty coefficient Flexible control is introduced:

[0089] ;

[0090] In the formula, The learning rate; The switching frequency of the action option in the last 100 steps; The target switching frequency.

[0091] When the switching frequency of Option exceeds the target frequency, The value will increase cumulatively, thereby increasing the switching cost, effectively inhibiting unnecessary frequent switching, reducing device start-stop energy consumption and enhancing system stability.

[0092] Step S3: The low-level policy model generates specific actions including device processing speed, transport vehicle transport frequency and buffer zone capacity adjustment ratio according to the action options output by the high-level policy model.

[0093] Specifically, for the low-level policy model, through industrial sensors, high-frame-rate cameras and acoustic devices, real-time acquisition of timing data, image data and acoustic data in the processing process is realized.

[0094] Preprocessing of the collected data: using multi-Besse wavelet transform to decompose the time series data into three levels, filter out high-frequency noise, and perform Z-score normalization; crop and enhance the defect area of the image data, set the cropped area size to 128x128 pixels, and rotate the image by ±15, adjust the brightness interval to [0.8, 1.2]; convert the acoustic signal into a mel-frequency spectrum graph after short-time Fourier transform, with a window size of 25ms and a frame shift of 10ms, set 128 mel filters, and the frequency range is 0-8kHz; convert the continuous sensor data into 32 intervals by equal-width binning method, each interval width is 5Hz, and convert it into a 32-dimensional one-hot vector as input features.

[0095] The dynamic features of time series data are extracted by one-dimensional convolutional neural network (1D-CNN) and long short-term memory network (LSTM), the defect area features of image data are extracted by lightweight Vision Transformer network, and the features of acoustic data are extracted by two-dimensional convolutional neural network processing mel-frequency spectrum graph.

[0096] The embodiment of the application adopts a deep deterministic policy gradient (DDPG) model to process low-level continuous control problems. The state space of DDPG includes three dimensions of equipment comprehensive efficiency, line beat and order queue length. The design of the action space aims at continuous control, covering the dynamic adjustment capability of key line resources, which is specifically defined as a three-dimensional continuous vector, including:

[0097] Device processing speed: ranging from 0% to 100%, used to adjust the running rate of the production unit.

[0098] Transportation frequency of intelligent logistics transport vehicle AGV: ranging from 1 to 5 times / minute, used to adjust the material distribution rhythm.

[0099] Buffer capacity adjustment ratio: allows dynamic expansion and contraction within ±20% to cope with short-term material accumulation or resource fluctuations.

[0100] DDPG is composed of two core networks: Actor network and Critic network.

[0101] The Actor network takes a 128-dimensional fused state vector as input and employs a dual-branch design. One branch inputs order queue features into a 64-unit LSTM to capture the dynamic impact of historical order changes; the other branch inputs equipment overall efficiency and production line cycle time features into a 64-node fully connected layer to extract equipment and production line information. The two outputs are concatenated and then pass through a Layer Normalization layer to stabilize the gradient distribution during deep network training. Two fully connected layers are then concatenated, with the second layer using tanh activation. Since the output of the Actor network is […] 1,1], equipment speed [0,100]%, AGV frequency [1,5] times / minute, buffer adjustment [ [0.2, +0.2], therefore, a general linear formula is used to transform the network output into values ​​of the device rate, AGV frequency and buffer capacity range, respectively, and finally output a three-dimensional continuous motion vector, which is mapped to the actual control range.

[0102] The Critic network is responsible for evaluating the value of state-action pairs. It includes a cross-attention fusion module and a value evaluation module. In the fusion phase, 128-dimensional state features are associated with 3-dimensional action vectors through a multi-head cross-attention module, dynamically calculating the attention weights between state and action, and outputting a 256-dimensional fused representation. The value evaluation phase consists of two fully connected layers, both using ReLU activation, ultimately outputting a Q-value prediction. To stabilize training, a target Critic network is maintained outside the main Critic network. Its structural parameters are completely identical to the main network, but the parameters are gradually synchronized through a soft update mechanism. This target network is used to calculate the TD objective and reduce training oscillations.

[0103] To ensure training stability and convergence, a soft update strategy is employed to synchronize the parameters of the Critic main network and the target network. This soft update mechanism synchronizes the parameters of the main and target networks every 100 training steps. The update method uses an exponentially weighted moving average, gradually updating the target network with a very small learning rate, i.e.:

[0104] ;

[0105] In the formula, The learning rate is set to 0.005 in this embodiment of the invention. For target network parameters; These are online network parameters.

[0106] This strategy effectively alleviates instability during training and prevents drastic fluctuations in the target value.

[0107] Step S4: The action options selected by the high-level policy model at each time step are accumulated into an action option sequence, and the corresponding continuous control actions generated by the low-level policy model at each time step are accumulated into a control action sequence, to form a production scheduling scheme for the multi-variety small-batch production scene.

[0108] Specifically, as shown in Figure 3 To ensure the stability and robustness of the model, the embodiment of the application adopts a phased collaborative training strategy in the model training phase. And in order to improve the training efficiency and enhance the influence of key samples in the learning process, a partitioned priority experience replay pool is constructed, and the training samples are stored in partitions according to importance. The high-value samples related to key events such as equipment failure, urgent order change, and logistics bottleneck are stored separately and obtain three times the sampling weight of regular samples, ensuring that the model can repeatedly learn from these key experiences. The regular samples use TD-error as the priority reference, and the sampling weight is updated in real time, so that the training can focus on the difficult samples that have not been mastered.

[0109] The training process is divided into two stages:

[0110] Pre-training stage: In the early stage of training, the Option network parameters of the high layer are frozen, and only the DDPG network of the bottom layer is trained to ensure the stability and convergence of the continuous control action. During the training process, training samples are periodically sampled from the experience pool, and the TD loss of the Critic network and the policy gradient loss of the Actor network are calculated in turn, and the respective parameters are updated using the back propagation algorithm. At the same time, after completing each update step, a soft update is performed to adjust the target network parameters, so as to maintain the stable convergence of the learning process. The purpose of this stage is to establish a stable and reliable bottom layer continuous control foundation for the system, and to ensure that it has basic action execution capability, so as to avoid the mutual interference of the high layer and the low layer in the non-converged state.

[0111] Joint training phase: After the underlying policy converges, gradually unfreeze the high-level Option layer network and enter the joint optimization phase of the upper and lower layers. In this phase, the complexity of the environment will gradually increase from simple to complex, and is divided into five levels. Level 1 contains only one processing device, and the order input quantity is fixed and stable; level 2 expands to two to three workstations in series, and the order arrival rate fluctuates ± 10%; level 3 contains a complete production line and introduces sudden events such as temporary device shutdown; level 4 has priority differences between orders; and level 5 has a dramatic change in order demand over time, with an increased random failure rate of devices. Every 200,000 steps will be evaluated, and if the average return increases by less than 2%, the environment complexity level will be automatically increased. During training, the option strategy selects the most suitable option based on the current state, and the underlying DDPG generates specific actions based on the option scenario. This curriculum learning approach enables the system to gradually adapt to more challenging real-world scenarios, effectively enhancing the model's generalization ability and robustness.

[0112] To verify the effectiveness of the method of the embodiment of the application, an empirical test was conducted in a small-batch workshop with a batch size of 5-50 pieces, an average of 4 orders per day, and a failure rate of 12%.

[0113] By deploying Internet of Things sensors to collect real-time workshop data, a state snapshot is generated every 5 seconds. These data are processed and constructed into an 11-dimensional global state vector reflecting the real-time status of orders, devices, energy consumption, and production in four dimensions. As shown in Table 2, the global state vector of the embodiment of the application is as follows.

[0114] Table 2 Global state vector of a small-batch workshop

[0115]

[0116] The preprocessed global state vector is spliced with the five Option encodings and input into the Option-Critic network to generate a policy distribution. During this process, the exploration coordinator dynamically adjusts the gamma value, optimizing the Option switching interval from the initial 120s to the stable period of 300s, and compressing the decision response time to an average of 80ms. Table 3 shows the probability of ω selection when an urgent order appears.

[0117] Table 3 Probability of ω selection when an urgent order appears

[0118]

[0119] As can be seen from Table 3, the Q value of ω2 is 1.24, which is much higher than the others, and the probability of 32% is also the highest, which shows that the system correctly identifies the emergency order and preferentially selects ω2 specially for dealing with it. But the probability is not too large, which shows that other options are also considered, such as the 20% of ω3, which may be used to handle abnormal or high-load situations.

[0120] The DDPG network receives high-level Option instructions and device states, and generates specific actions: reduce the material processing speed from 65% to 42% to deal with the risk of failure; reduce the AGV transportation frequency from 3.5 times / minute to 1.8 times / minute to reduce the logistics load; at the same time, adjust the buffer zone capacity to +18% to buffer the work-in-process. The Critic network evaluates the Q value with a failure risk weight of 0.4, and the noise parameter σ is exponentially decayed from 0.25 to 0.08 with training.

[0121] The Actor network of DDPG outputs the parameters of machine tool speed control, as shown in Table 4.

[0122] Table 4 Machine tool speed control parameters

[0123]

[0124] When the machine failure condition occurs, the unexplored state density soars to 0.42, triggering the core formula:

[0125] ;

[0126] The priority of the Option related to emergency failure is instantly raised, and this Option obtains 83% of the exploration resources in the next 2 hours, and the iteration speed of the new cooling strategy is increased from the regular 5 times / hour to 17 times / hour.

[0127] The original strategy probability distribution of the high-level is [0.10, 0.15, 0.05, 0.20, 0.50], which is modified by the following formula:

[0128] ;

[0129] After modification, the processing delay of machine failure events is compressed from 8.5 minutes to 6.2 minutes, and the scrap rate is reduced by 23%.

[0130] In this process, the value of γ is dynamically adjusted according to the reward feedback of order delay reduction, and the value of γ is like the breathing rhythm of exploration intensity, with a daily fluctuation of ±0.03, ensuring that high-value exploration is focused on the fluctuation window period.

[0131] The switching frequency of the Option during the failure peak period is 0.28 times / step, triggering the constraint:

[0132] .

[0133] After 3 times of overrunning The value accumulates to 0.31, effectively inhibits the shock, and the switching frequency is reduced from 35 times / hour to 23 times / hour, and the energy consumption of the equipment start-stop is reduced by 21%. This shows that the dynamic adjustment mechanism provided by the method of the embodiment of the application not only improves the decision efficiency, but also fundamentally enhances the stability and economy of the system. The optimization effect of the exploration coordinator is shown in Table 5, and the comprehensive effect comparison is shown in Table 6.

[0134] Table 5 Optimization effect of exploration coordinator

[0135]

[0136] Table 6 Comprehensive effect comparison

[0137]

[0138] Table 5 and Table 6 clearly quantify the breakthrough progress brought by the method of the embodiment of the application, and the significant shortening of the delivery time of the emergency order and the improvement of the equipment OEE directly reflect the strong ability of the system in improving efficiency. The significant decrease in abnormal downtime rate proves the robustness of the system to dynamic disturbance. The reduction of peak power single-piece energy consumption and the significant compression of single Option decision time directly reflect the efficiency and reliability of the system from the economic and technical aspects.

[0139] When the hierarchical strategy model is integrated into lean production scheduling, the structural contradiction between experience solidification and dynamic adaptation in traditional methods should be solved. The rationality of task decomposition, the quantitative expression of multi-objective coordination, and the contradiction between calculation efficiency and real-time performance should be considered. Although the hierarchical architecture can reduce the complexity of the state space of a single layer, the interaction of multi-level strategies may increase the decision delay, and a balance between strategy and response speed needs to be sought. Therefore, the embodiment of the application effectively solves the limitations of traditional lean management in the face of dynamic and complex environments by constructing a double-layer intelligent agent architecture. By decoupling macro-strategy planning and micro-continuous control, the complex multi-objective scheduling problem becomes manageable and learnable, and the efficiency and accuracy of decision-making are significantly improved. At the same time, through an intelligent exploration coordinator, the system can spontaneously focus learning resources on the unknown areas that need to be explored the most, achieving more efficient knowledge discovery and learning than traditional methods. Through phased pre-training and joint training, and gradually increasing the complexity of the environment, the model ensures smooth convergence and strong generalization ability in simple to complex scenarios.

[0140] The technical features of the above embodiments can be combined in any manner. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, and only the preferred embodiments of the present application are expressed, which are described in more detail and in a more specific manner, but it should not be understood as a limitation on the scope of the present application. As long as the combinations of these technical features do not contradict each other, they should be considered as the scope of the present application.

[0141] It should be noted that, for those skilled in the art, there are still several modifications and improvements without departing from the concept of the present application, which are within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A production scheduling method based on hierarchical policy model and lean management, characterized in that, The method comprises the following steps: Step S1: constructing a hierarchical policy model comprising a high-level policy model and a low-level policy model; Step S2: collecting running data of a production workshop from a manufacturing system, constructing a global state vector, the high-level policy model generating an initial probability distribution for all action options according to the global state vector, calculating a priority of each action option according to a relative value gap, an exploration potential and a maturity discount, adaptively adjusting the initial probability distribution of all action options according to the priority, and the high-level policy model outputting a current optimal action option according to the adjusted probability distribution; Step S3: the low-level policy model generating specific actions comprising a device processing speed, a transport frequency of a transport vehicle and a buffer capacity adjustment ratio according to the action option output by the high-level policy model; Step S4: accumulating the action option selected by the high-level policy model at each time step into an action option sequence, accumulating the corresponding continuous control action generated by the low-level policy model at each time step into a control action sequence, and forming a production scheduling scheme for a multi-variety small-batch production scene.

2. The production scheduling method based on hierarchical policy model and lean management according to claim 1, characterized in that: When the high-level policy model generates an initial probability distribution for all action options according to the global state vector in step S2, a reward function is constructed based on a device comprehensive efficiency, an order delivery rate, an energy consumption cost and a switching penalty, the high-level policy model calculates the initial probability distribution of all action options according to the reward function, and an expression of the reward function of the high-level policy model is: ； ； ； ； ； In the formula, is a reward function of a high-level policy model; is a weight coefficient of the equipment comprehensive efficiency, order delivery rate, energy consumption cost and switching penalty; is the equipment comprehensive efficiency.

3. The production scheduling method based on hierarchical policy model and lean management according to claim 1, characterized in that: An expression for calculating a priority of each action option according to a relative value gap, an exploration potential and a maturity discount in step S3 is: ； ； ； ； wherein is the action option in state ; is the maximum Q-value for all action options is the Q-value of the action option in state at the current time instant , is the learnable adjustment coefficient is the exploration potential indicator is the maturity factor is the Q-value of the action option in state at the current time instant 4. The production scheduling method based on hierarchical policy model and lean management according to claim 1, characterized in that: An expression for adaptively adjusting the initial probability distribution of all action options according to the priority in step S3 is: ； ； wherein, is the modified high-level policy; is the original high-level policy; is the dynamic exploration incentive strength; is the action option is the state is the exploration priority; is the switching penalty coefficient; is the switching penalty coefficient of the next time step; is the indicator function, which is 1 if the action option has been switched recently, otherwise 0; is the learning rate; is the switching frequency of the action option in the last 100 steps; is the target switching frequency.

5. The production scheduling method based on hierarchical policy model and lean management according to claim 4, characterized in that: Dynamic exploration of incentive intensity According to recent Difference between average reward of step and historical average reward Adaptive adjustment: ； In the formula, is a learning rate; is a target reward; is a dynamic exploration incentive strength at the next time step.

6. The production scheduling method based on hierarchical policy model and lean management according to claim 1, characterized in that: The low-level policy model in step S1 is a deep deterministic policy gradient network (DDPG), which comprises an Actor network for generating actions and a Critic network for evaluating action values.

7. The production scheduling method based on hierarchical policy model and lean management according to claim 6, characterized in that: The Actor network adopts a two-way branch design, one way inputs order queue features into a long short-term memory network (LSTM), and the other way inputs device comprehensive efficiency and production line beat features into a fully connected layer, and the outputs of the two ways are spliced to generate the continuous control action.

8. The production scheduling method based on hierarchical policy model and lean management according to claim 1, characterized in that: After constructing the hierarchical policy model in step S1, the hierarchical policy model is trained in stages, comprising the following steps: A pre-training stage: freezing the high-level policy model and only training the low-level policy model; A joint training stage: after the low-level policy model converges, unfreezing the high-level policy model, and jointly optimizing the high-level policy model and the low-level policy model.

9. The production scheduling method based on hierarchical policy model and lean management according to claim 8, characterized in that: A partitioned priority experience replay pool is used in the process of the staged training, the partitioned priority experience replay pool comprises a high-value sample area and a regular sample area, the high-value sample area is used to store interaction trajectory samples related to device failures, order emergency changes and logistics bottlenecks, and the regular sample area uses a time difference error as a priority standard and gives the samples in the high-value sample area a higher sampling weight than the samples in the regular sample area.

10. The production scheduling method based on hierarchical policy model and lean management according to claim 8, characterized in that: The joint training phase in the staged training adopts a way of gradually increasing the environment complexity for training, and the environment complexity gradually increases from a level containing only one processing device and fixed order input to a level containing multiple stations, emergencies and order demand fluctuations.

Images