A two-stage batch flow flexible job shop dynamic scheduling method and device

Abstract

The present application belongs to the technical field of workshop scheduling, and discloses a two-stage batch flow flexible job shop dynamic scheduling method and equipment, which comprises the following steps: (1) extracting product processing technology features to construct a multi-dimensional batch feature set, using a genetic programming algorithm to automatically select and combine batch features in the multi-dimensional batch feature set, and then generating multiple batch scheduling rules through evolution operations, and the multiple batch scheduling rules constitute an initial batch scheduling rule set; (2) using a deep reinforcement learning algorithm to dynamically optimize the weights of the initial batch scheduling rule set, and then determining the current optimal workpiece sorting and machine selection strategy, thereby obtaining a scheduling scheme. Through the cooperative work of the genetic programming algorithm and the deep reinforcement learning algorithm, the present application realizes real-time response and adaptive scheduling to dynamic disturbances such as new workpiece arrival and machine failure in the production process, effectively improves the optimization efficiency, and improves the workshop production efficiency and resource utilization rate.

Description

Technical Field

[0001] This invention belongs to the technical field of workshop scheduling, and more specifically, relates to a dynamic scheduling method and equipment for a two-stage batch flow flexible operation workshop. Background Technology

[0002] As the manufacturing industry gradually shifts from large-batch, single-product production to small-batch, multi-product flexible production, production scheduling in batch flow workshops has become crucial for manufacturing enterprises to improve production efficiency and reduce production costs. Batch flow workshop production is characterized by batch processing of workpieces, highly dynamic production tasks, and a high degree of flexibility in processing equipment. Scheduling decisions must not only consider the diverse process requirements of different batches of workpieces but also take into account the efficient allocation of equipment resources and the rational distribution of tasks. Traditional static scheduling methods are difficult to effectively adapt to the frequent dynamic disturbances in batch flow workshops, such as order changes, equipment failures, and emergency order insertions, which seriously affect the enterprise's production efficiency and delivery timeliness, and restrict the enterprise's ability to respond quickly to changes in market demand.

[0003] To address scheduling challenges in the dynamic environment of batch flow workshops, researchers have proposed a series of rule-based or heuristic-based dynamic scheduling strategies in recent years. While these strategies have improved production efficiency and responsiveness to some extent, they still suffer from problems such as over-reliance on expert experience in rule design, poor rule adaptability, and insufficient dynamic optimization capabilities, failing to effectively handle the dynamic changes and uncertainties in batch flow workshop production processes. Furthermore, there is a close interaction between sub-batch partitioning and task scheduling in batch flow workshops. How to effectively handle the collaborative optimization between batch partitioning, process sequencing, and machine allocation is a key technical issue for further improving batch flow workshop scheduling methods. Genetic Programming (GP) can automatically evolve and optimize scheduling rules, adapting to different batch characteristics and process constraints, and can generate high-quality batching, sequencing, and machine selection rules in batch flow environments. Deep Reinforcement Learning (DRL) possesses powerful adaptive learning capabilities, enabling it to perceive the workshop environment status in real time and optimize the weights of scheduling rules, thereby dynamically adjusting production decisions and improving the flexibility of scheduling schemes. However, there is currently little research that effectively combines genetic programming with deep reinforcement learning, especially in batch workflow environments, where efficient dynamic scheduling methods based on rule-based adaptive evolution and continuous weight optimization are still incomplete. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a two-stage batch flow flexible workshop dynamic scheduling method and equipment, which aims to solve the problem of low optimization efficiency of existing batch flow flexible workshop scheduling.

[0005] To achieve the above objectives, according to one aspect of the present invention, a two-stage batch flow flexible job shop dynamic scheduling method is provided, comprising the following steps: (1) Extract product processing technology features to construct a multi-dimensional batch feature set. After the batch features in the multi-dimensional batch feature set are automatically selected and combined by the genetic programming algorithm, multiple batch scheduling rules are generated through evolutionary operation. The multiple batch scheduling rules form the initial batch scheduling rule set. (2) The initial batch scheduling rule set is dynamically optimized by using a deep reinforcement learning algorithm, and then the current optimal workpiece sorting and machine selection strategy is determined, thereby obtaining the scheduling scheme; Batch scheduling rules include batching rules and scheduling rules. Batching rules are used to determine how workpieces are divided into batches. Scheduling rules include sub-batch sorting rules and machine selection rules, which are used to determine the processing order of sub-batches and the corresponding processing equipment. The initial set of batch scheduling rules is represented by a binary tree structure. The internal nodes of the binary tree are operators used to represent the operational relationships between child nodes. After evolving to a certain stage, the batching rules obtained through evolution are fixed and no longer changed; only the scheduling rules continue to evolve. Based on the fixed batching rules, the genetic programming algorithm generates a set of scheduling rules suitable for a specific batching scheme through continuous crossover, mutation, and selection operations.

[0006] Furthermore, when a new order arrives, the workpieces in the new order are first divided into batches using fixed batching rules; after the batch division is completed, the determined sub-batch workpieces are added to the task pool; then, the processing priority of each sub-batch is evaluated based on the current workshop production environment, and the initial batching scheduling rule set obtained through evolution is used to initially determine the sub-batch sorting and machine selection strategy.

[0007] Furthermore, for each specific scheduling decision, the weight distribution of the initial batch scheduling rule set generated by genetic programming is updated in real time based on the real-time production status information of the workshop.

[0008] Furthermore, after each specific scheduling action is executed, the workshop environment immediately provides a real-time reward signal to evaluate the efficiency of the current scheduling decision. At the same time, after the sub-batch of workpieces is completed, the parameters of the deep reinforcement learning algorithm are dynamically adjusted in real time based on the latest production status information and reward feedback in the workshop, thereby optimizing the weight combination corresponding to the batch scheduling rules for subsequent decisions in real time.

[0009] Furthermore, before assigning weights, the evaluation values ​​calculated by all rules are uniformly normalized; based on the normalization results, the workpiece sorting rules and machine selection rules are weighted to obtain a weighted comprehensive score, and based on the weighted comprehensive score, the sub-batch processing sequence of the current decision point and the machine allocation strategy for the sub-batch workpieces are determined respectively.

[0010] Furthermore, collaborative genetic programming is used to optimize the batching rules and scheduling rules; and dual-tree genetic programming is used to optimize the sub-batch sorting rules and machine selection rules within the scheduling rules.

[0011] The present invention also provides a two-stage batch flow flexible job shop dynamic scheduling system. The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it performs the two-stage batch flow flexible job shop dynamic scheduling method as described above.

[0012] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the two-stage batch flow flexible job shop dynamic scheduling method as described above.

[0013] In summary, compared with the prior art, the two-stage batch flow flexible workshop dynamic scheduling method and equipment provided by the present invention have the following beneficial effects: 1. This invention achieves real-time response and adaptive scheduling to dynamic disturbances such as the arrival of new workpieces and machine failures during the production process through the collaborative work of genetic programming algorithm and deep reinforcement learning algorithm. It effectively improves optimization efficiency, enhances workshop production efficiency and resource utilization, and is suitable for dynamically changing production environments.

[0014] 2. This invention can obtain a set of batch scheduling rules based on genetic programming evolution, including batch rules, sub-batch sorting rules and machine selection rules. It also optimizes the weight combination of each scheduling rule in real time according to the dynamic environment through deep reinforcement learning algorithm. Compared with existing manually designed or experience-based scheduling rules, it can obtain more scientific results in a shorter time and significantly improve the overall optimization capability of workshop scheduling decisions.

[0015] 3. This invention proposes a batching rule based on evolutionary generation using a genetic programming algorithm. Utilizing the multidimensional production characteristics of a batch production environment and combining statistical methods to extract batch features, it achieves an efficient and dynamic batching method suitable for batch production workshops. Compared with traditional batching methods, the feature-driven batching rule designed in this invention can more accurately reflect actual production needs, adapt to dynamically changing batch production environments, and effectively improve the production efficiency of flexible batch production workshops.

[0016] 4. This invention normalizes the original priority calculation values ​​of sub-batches under each rule, enabling fair comparison and effective integration of evaluation indicators between different rules. This avoids decision-making bias caused by differences in rule numerical dimensions, further improving the rationality and stability of decision-making. Attached Figure Description

[0017] Figure 1 This is a flowchart of a two-stage batch flow flexible workshop dynamic scheduling method provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the optimization process of the hybrid genetic programming algorithm provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the interaction strategy between the intelligent agent and the workshop environment provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a weight-based workpiece selection conforming to rules provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] This invention provides a two-stage dynamic scheduling method for flexible job shops in batch flow, which is based on Genetic Programming (GP) and Deep Reinforcement Learning (DRL). The scheduling method aims to minimize the maximum completion time and addresses the problems of high coupling between batch division and sub-batch scheduling and the reliance on manual rule design in batch flow production by constructing a structured two-stage optimization framework. The first stage extracts product processing technology features to construct a multi-dimensional batch feature set. The GP algorithm automatically selects and combines batch features, evolving to form an initial set of batch flow scheduling rules. Simultaneously, Collaborative Genetic Programming (CCGP) and Dual-Tree Genetic Programming (TTGP) are combined to optimize the workpiece batching strategy and sub-batch scheduling strategy, forming a high-quality scheduling rule set suitable for batch flow environments. The second stage introduces a Deterministic Policy Gradient Algorithm (DDPG) deep reinforcement learning method to adaptively adjust the weights of the sub-batch scheduling rules generated by the GP algorithm, generating composite scheduling strategies in real time to adapt to dynamic disturbance events and continuously optimizing sub-batch sorting and machine allocation results. This method can respond to different dynamic scenarios in real time, providing a more flexible and efficient scheduling solution, greatly improving the adaptability and solution efficiency of production scheduling, optimizing the production process, and creating greater production benefits for enterprises.

[0020] The aforementioned scheduling method is a dynamic scheduling method for batch flow flexible job shops that effectively combines the rule evolution capability of genetic programming algorithms with the adaptive decision optimization capability of deep reinforcement learning algorithms. It can dynamically generate and optimize rules, adjust rule weights in real time, and flexibly respond to various dynamic events that occur during the production process in a batch flow production environment, thereby significantly improving the production efficiency of batch flow workshops and promoting the sustainable development of agile production and intelligent manufacturing for enterprises.

[0021] Please see Figure 1 , Figure 2 , Figure 3 and Figure 4 The scheduling method mainly includes the following steps: Step 1: Extract product processing technology features to construct a multi-dimensional batch feature set. Use a genetic programming algorithm to automatically select and combine batch features in the multi-dimensional batch feature set, and then generate multiple batch scheduling rules through evolutionary operations. These multiple batch scheduling rules form the initial batch scheduling rule set.

[0022] In one implementation, an intelligent agent system for a batch-flow flexible workshop is constructed with the optimization objective of minimizing the maximum completion time. This system employs an optimization framework combining genetic programming (GP) and deep reinforcement learning (DRL), and establishes an interaction strategy between the agent and the workshop environment to achieve dynamic batching and intelligent scheduling of workshop production tasks. Simultaneously, a task pool for storing unprocessed sub-batches and an equipment pool for storing available processing machines are constructed, with the optimization objective of minimizing the maximum completion time of workpieces in the workshop. The agent extracts batch features from production data and uses the genetic programming (GP) algorithm to evolve and generate an initial set of batching and scheduling rules. Upon entering the dynamic scheduling phase, the agent determines whether there are dynamic disturbance events such as new order arrivals or equipment failures, and updates the states of the task pool and equipment pool accordingly. Based on the real-time status information of the workshop, the agent dynamically optimizes the weights of the scheduling rules through a deep reinforcement learning (DRL) algorithm, selects sub-batches to be processed from the task pool, and determines the processing sequence and processing machines. After each decision action is executed, the workshop environment provides real-time feedback on reward values ​​and the latest status feature information to guide the reinforcement learning agent in updating and optimizing parameters. The above decision-making process is continuously executed in a loop until all workpieces in the task pool are processed, and finally outputs an adaptive optimized production scheduling plan for the batch flow flexible operation workshop, thereby continuously improving the workshop's production efficiency and dynamic response capabilities.

[0023] A set of batch scheduling rules is generated through evolution using a genetic programming (GP) algorithm. These rules include batching rules and scheduling rules. Batching rules determine how workpieces are divided into batches, primarily based on characteristics such as workpiece processing time, priority, number of selectable machines, and process time distribution. Scheduling rules encompass sub-batch sorting rules and machine selection rules, used to determine the processing order of sub-batches and their corresponding processing equipment. Through the initial evolutionary process of the genetic programming algorithm, an initial set of batch scheduling rules, including both batching and scheduling rules, is formed.

[0024] After the genetic programming algorithm evolves to a certain stage, to ensure the stability of the batching scheme, the evolved batching rules are fixed and no longer changed. Only the scheduling rules (including sub-batch sorting rules and machine selection rules) continue to evolve to further optimize the scheduling strategy. Based on the fixed batching rules, the genetic programming algorithm generates a set of high-quality scheduling rules suitable for a specific batching scheme through continuous crossover, mutation, and selection operations.

[0025] When a new order arrives, the overall intelligent agent (the agent of the intelligent agent system) first divides the workpieces in the order into batches using the aforementioned fixed batching rules. After batching, the determined sub-batch workpieces are added to the task pool. The overall intelligent agent evaluates the processing priority of each sub-batch based on the current workshop production environment (such as the real-time status of machines, work-in-process status, and workpiece delivery date requirements), and uses the initial batching scheduling rule set obtained through evolution to initially determine the sub-batch sorting and machine selection strategies to form a preliminary scheduling scheme, corresponding to the initial batching scheduling rule set.

[0026] For each specific scheduling decision, the overall agent employs a deep reinforcement learning algorithm to update the weight distribution of the initial batch scheduling rule set generated by genetic programming in real time, based on real-time production status information in the workshop (including current machine idle status, workpiece queue status, sub-batch processing priority, etc.). The reinforcement learning agent utilizes a deep neural network to output the weights of each batch scheduling rule in real time according to the environmental state, forming a weighted composite batch scheduling rule scheme. Based on this composite batch scheduling rule scheme, the agent determines the specific sorting and machine allocation strategy for the current sub-batch to minimize the maximum workpiece completion time in the workshop.

[0027] After each specific scheduling action, the reinforcement learning agent receives real-time reward signals from the shop floor environment to evaluate the efficiency of the current scheduling decision. Simultaneously, once a sub-batch of workpieces is completed, the reinforcement learning agent receives the latest production status information and reward feedback from the shop floor in real time. This information is then input into the reinforcement learning algorithm to dynamically adjust the agent's network parameters, thereby optimizing the weight combinations corresponding to the batch scheduling rules for subsequent decisions. Based on the real-time feedback, the reinforcement learning agent updates the status of the sub-batches awaiting processing and the list of processing resources in the task pool, before proceeding to the next scheduling decision stage.

[0028] When dynamic disturbances occur in the workshop, the reinforcement learning agent responds rapidly at the corresponding decision points. The agent immediately batches newly arriving workpieces according to fixed batching rules and updates the task pool. Simultaneously, the agent quickly adjusts the weight distribution of the batching scheduling rules to adapt to the current disturbance environment, optimizing task sequencing and resource allocation strategies to ensure that the impact of disturbances on production scheduling is minimized.

[0029] Repeat the above steps until all workpieces are processed and generate the final workshop production schedule to ensure the efficiency and flexibility of the production process.

[0030] Please see Figure 2 The initial batch scheduling rule set is represented by a binary tree structure; the internal nodes of the binary tree are operators used to represent the operational relationships between child nodes.

[0031] The leaf nodes of the binary tree are the basic features of the workshop environment; the features used in the genetic programming algorithm for workpiece batch division, sub-batch sorting and machine selection specifically include: workpiece batching rule features, sub-batch sorting rule features and machine selection rule features.

[0032] The operators include: "+", "-", "×", "÷", "Max", and "Min"; when the divisor is 0, the result of "÷" returns a preset value; "Max" means taking the maximum value and "Min" means taking the minimum value.

[0033] The characteristics of batching rules include: a) Average Total Processing Time (ATT) for each type of workpiece. b) Total number of jobs (JCT); c) Average Machine Selection (AMS) for each type of workpiece process. d) The sum of the maximum processing times required for each process, MaxPT (Maximum Processing Time Sum); e) The sum of the minimum processing times required for each process, MinPT (Minimum Processing Time Sum); f) Total Processing Time Standard Deviation (TSD) of the process time distribution. g) The ratio of standard deviation to mean, TCV (Total Coefficient of Variation).

[0034] The characteristics of sub-batch sorting in scheduling rules include: a) Processing time (PT) for sub-batch workpieces; b) Delivery date (DD) of sub-batch workpieces; c) Sub-batch weight W (Weight); d) Next Processing Time (NPT) of the sub-batch; e) Work Remaining (WKR) of sub-batch workpieces; f) Number of Operations Remaining (NOR) for sub-batch workpieces. g) Remaining Due Date (RDD) for sub-batch workpieces; h) Slack Time (SL) for sub-batches.

[0035] The characteristics of machine selection in scheduling rules include: a) Machine Waiting Time (MWT); b) The current work in the queue (WIQ). c) Total idle time of the equipment (WD). d) Work Processing Time (WPT) on the equipment; e) Work Utilization (WU); f) Work Queue Size (WQS) of the device.

[0036] The batching rule processing method also includes: (1) For each batch of parts, the terminal set is first constructed by extracting the features of that type of parts. The extracted features include multi-dimensional information such as the total number of parts in the batch, the longest processing time, the shortest processing time, and the number of selectable machines. These features constitute the input data for generating batching rules.

[0037] (2) Through the evolutionary process of the genetic programming algorithm, the extracted part features are used as the terminal set, and combined with the preset comprehensive function set (operation symbols), an expression representing the batching rule is generated. This expression represents the relationship between the part features and can calculate the corresponding value based on the actual workpiece data. For example, for each type of part, after calculating its total processing time, minimum processing time, and the number of machines that can be selected, the values ​​are substituted into the expression to obtain the value. ,in Indicates the first k The processing results for the parts. The genetic programming algorithm evolves by continuously optimizing these expressions, producing batching rules that optimize the scheduling process.

[0038] (3) After the batching rules are determined, calculate the value of each type of part. And further calculate the average value of all part classes. ,Right now:

[0039] in, K This represents the total number of parts. Indicates the first k Calculation results for the type of part. Define an average batch size. u This represents the target quantity for each batch. The formula for calculating the final batch quantity allocated to each type of part is:

[0040] in, Represents part class k The ratio between the average of all part classes and the average batch size, multiplied by the average batch size. u This yields the allocated batch quantity of this type of part in the overall batch flow.

[0041] The genetic programming algorithm also includes the following processing methods: (1) In the genetic programming algorithm, Collaborative Coevolutionary Genetic Programming (CCGP) is used to optimize the batching rules and scheduling rules. Specifically, there is a high degree of interdependence between the batching rules and the scheduling rules. CCGP can utilize the collaborative relationship between multiple rules to perform parallel evolution and optimization, so as to effectively coordinate the batch division and subsequent scheduling tasks, and avoid the local optima or rule conflicts that occur when rules evolve alone. Therefore, the CCGP method can effectively improve the rationality of workpiece batch division and the overall coordination of scheduling decisions in the batch production environment, thereby significantly improving the overall quality and adaptability of the initial batching scheme.

[0042] (2) Based on the batching rules optimized by CCGP, Two-Sub-Tree Genetic Programming (TTGP) is used to optimize the sub-batch sorting rules and machine selection rules within the scheduling rules. Sub-batch sorting and machine selection tasks in the scheduling process typically have a clear hierarchical structure and sequence. The TTGP method uses a two-tree structure to divide the scheduling rules into a decision tree specifically responsible for sub-batch sorting and a decision tree specifically responsible for machine selection. This tree structure of TTGP can effectively express the order and dependency in task scheduling, avoiding the problem that traditional single-tree structures cannot effectively handle the complex logic of scheduling tasks. TTGP can finely optimize the sub-batch sorting and machine selection rules, ensuring that the scheduling task selection and resource allocation strategies are more reasonable and efficient in terms of rule hierarchy and execution order.

[0043] The maximum completion time (makespan) of a workpiece is defined as the maximum of the actual completion times of all workpieces, and the calculation formula is:

[0044] in, Tj For the first j The actual completion time of each workpiece. J The total number of sub-batches. The reinforcement learning agent optimizes the weights of the scheduling rules to minimize the maximum completion time of the workpieces, thereby improving the overall efficiency and time sensitivity of shop floor production scheduling.

[0045] After the reinforcement learning agent completes the scheduling action at each decision point, the workshop environment immediately sends a real-time reward signal. For a reinforcement learning agent, the reward function is the difference between the maximum completion time (makespan) between the current decision point and the previous decision point, plus the change in machine utilization. The calculation formula is as follows:

[0046] in, This is the maximum completion time since the last decision. This represents the maximum completion time after the current decision. and These represent the overall machine utilization rates of the workshop after the previous decision and the current decision, respectively. Adjust flexibly according to actual scheduling needs.

[0047] When a decision effectively reduces the maximum completion time, the reward value is positive, indicating that the current scheduling action helps improve production efficiency; conversely, the reward value is negative, prompting the agent to continuously optimize the scheduling strategy. By maximizing the cumulative sum of this reward signal, the agent gradually achieves dynamic optimization of the scheduling rule weights, thereby continuously improving the real-time performance of shop floor scheduling and overall production efficiency.

[0048] Step two involves using a deep reinforcement learning algorithm to dynamically optimize the weights of the initial batch scheduling rule set, thereby determining the current optimal workpiece sorting and machine selection strategy, and thus obtaining the scheduling scheme.

[0049] Specifically, at each decision moment, the reinforcement learning agent corresponding to the deep reinforcement learning algorithm calculates the weight distribution of the initial batch scheduling rule set in real time based on the real-time status information of the workshop (including the current machine idle status, workpiece queue status, sub-batch processing priority, etc.). Based on the obtained weight distribution, the rules generated by the genetic programming algorithm are weighted and combined to determine the current optimal workpiece sorting and machine selection strategy.

[0050] The reinforcement learning agent evaluates each decision action in real time and updates its network parameters based on the evaluation results, gradually improving the quality of scheduling decisions. Simultaneously, after a sub-batch of workpieces completes processing, the agent receives the latest production status information and reward feedback from the workshop in real time and inputs this information into the reinforcement learning algorithm to dynamically adjust the deep reinforcement learning network parameters, thereby optimizing the weight combination of scheduling rules for subsequent decisions in real time. Based on the real-time feedback, the agent updates the status of the sub-batches awaiting processing and the list of processing resources in the task pool, entering the next scheduling decision stage.

[0051] A deep reinforcement learning method using the deterministic policy gradient algorithm (DDPG) is employed to dynamically optimize the weights of batch scheduling rules. Specifically, at each scheduling decision, the reinforcement learning agent dynamically generates a set of weights for batch scheduling rules based on the current production environment state to optimize the sub-batch sorting rule (JS) and machine selection rule (MS). The optimization process calculates the optimal rule weight distribution in real time based on the current production state. The agent's optimization objective is to minimize the maximum completion time, ensuring that the scheduling system can dynamically adjust the rule weights according to the actual production environment, thereby optimizing production scheduling efficiency. At each decision point, the agent's input includes real-time shop floor data such as workpiece characteristics, machine status, and work-in-process quantity in the current task pool. Based on the real-time shop floor state information, the reinforcement learning agent dynamically outputs corresponding rule weight vectors for the sub-batch sorting rule (JS) and machine selection rule (MS), respectively.

[0052]

[0053] in, For a moment t The generated set of rule weights, Indicates the first i The optimal weight of each sub-batch scheduling rule in the current production environment. I This represents the total number of sorting rules for the sub-batch. For a moment t The weight set of the generated machine selection rules. Indicates the first n The optimal weight of each machine selection rule in the current production environment. N The total number of rules selected for the machine.

[0054] Before calculating the weighted composite score of the sub-batch sorting rule (JS) and the machine selection rule (MS), directly adding the weighted scores due to potential differences in the magnitude or numerical range of the evaluation values ​​calculated by different rules can lead to inconsistent influence of different rules on the final decision, reducing the rationality of the scheduling decision. To address this issue, the reinforcement learning agent performs a unified normalization process on the evaluation values ​​calculated by all rules before assigning weights. Specifically, the normalization method is as follows: Normalization of sub-batch sorting rule priority:

[0055] Normalization of machine selection rule priority:

[0056] in: For the first i Under the first JS rule, the first j Evaluation value of the sub-batch sorting rule for each sub-batch; For the first n Under the MS rule, the first m The machine selection rule evaluation value for each machine; and This corresponds to the normalized rule value; This represents the total number of sub-batches currently awaiting processing; This represents the total number of available machines at the current decision point.

[0057] After the rule weight set W(t) is calculated, the agent performs weighted calculations on the workpiece sorting rules and machine selection rules based on this rule weight set, and finally obtains the scheduling decision result. Specifically:

[0058]

[0059] in: For a moment t Calculated sub-batch j Sorting scores in sub-batches; For a moment t The calculated machine m The machine-selected score; Based on the weighted composite score mentioned above, the agent determines the sub-batch processing sequence and the machine allocation strategy for the sub-batch workpieces at the current decision point. Specifically: First, the overall score is determined by sorting the sub-batches. Determine the processing sequence of the current sub-batch to be processed:

[0060] in: j Index for sub-batch; For the first j The overall score of each sub-batch at the current decision point t; Secondly, for the determined sub-batch, select the most suitable machine equipment for processing the sub-batch based on the comprehensive score corresponding to the machine selection rules:

[0061] in, M For machine indexing; Indicates the first m The machine is at all times t The overall score for machine selection for the current batch of sub-batch to be processed.

[0062] Through the above decision-making process, the intelligent agent can efficiently complete the decision on the processing sequence of sub-batch and the allocation of machine equipment, thereby optimizing the overall production efficiency of the workshop.

[0063] The network input states of the deep reinforcement learning agent include: (1) The processing time characteristics of sub-batch workpieces in the task pool; (2) Delivery time characteristics of sub-batch workpieces in the task pool; (3) Characteristics of the number of remaining processes for sub-batch workpieces in the task pool; (4) Real-time load characteristics of machines in the workshop; (5) Characteristics of machine queue length within the workshop; (6) Characteristics of machine waiting time for processing tasks in the workshop; (7) Overall characteristics of the quantity of work-in-process in the workshop.

[0064] Statistical methods are employed to extract features from the aforementioned state data. Specifically, the mean, standard deviation, maximum, and minimum values ​​are calculated for each category of original feature data to form a unified and scale-independent state feature input data. By uniformly calculating the mean, standard deviation, maximum, and minimum values ​​for each feature category, the agent's input features can effectively eliminate dependence on the specific scale of the workshop problem, reduce fluctuations in the dimensionality of state data, and significantly improve the training efficiency and generalization ability of the agent network. This ensures that it can stably and efficiently adapt to the dynamic changes in the workshop production environment.

[0065] The mechanism by which a reinforcement learning agent responds to dynamic perturbation events is as follows: (1) When a new order arrives, the reinforcement learning agent does not directly add the workpiece to the task pool, but first performs batch processing. The reinforcement learning agent creates new batches based on the workpiece characteristics of the order to ensure the stability and balance of batch production. After batching is completed, the reinforcement learning agent updates the task pool and re-evaluates key features such as batch priority, delivery time, and machine allocation. Based on the above steps, the reinforcement learning agent calculates the optimal scheduling strategy and adjusts the rule weights to optimize subsequent task scheduling.

[0066] (2) When a machine in the workshop malfunctions or stops, the reinforcement learning intelligence will remove the malfunctioning machine from the queue through the abnormal information of the machine and immediately reassign the task to other available machines, update the real-time status information of the workshop, and ensure that the workpiece allocation and machine selection strategy in the task pool can reflect the latest production environment, so as to reduce the impact of the malfunction on production scheduling.

[0067] The reinforcement learning agent is trained and updated using the Deep Deterministic Policy Gradient (DDPG) algorithm, which is suitable for continuous action spaces. The input state of the reinforcement learning agent at time t... Seven types of input features, after statistical processing, are used to reflect the real-time production status of the workshop. The reinforcement learning agent operates at any given time... Continuous output action Defined as the batch scheduling rule weight vector generated by the policy network (Actor) corresponding to reinforcement learning, i.e.:

[0068] in, This is the set of weights for the sub-batch sorting rules. The set of weights for the rules selected by the machine; the action dimension is the number of sub-batch sorting rules. I Number of machine selection rules N The sum of these weights. This batch scheduling rule weight vector is used to weight and combine the scheduling rules generated by genetic programming to form the current scheduling execution strategy.

[0069] In the above state ,action With rewards Based on this, reinforcement learning is used to develop corresponding value function networks (Critic networks) to... Input, output state-action value function Its loss function expression is:

[0070] in, and These represent the current state and the state at the next moment, respectively. The reward is given in real time after the current decision-making action. Discount factor; This is an estimate of the next state-action pair calculated using the objective value function network; and These are the parameters of the current value function network and the target value function network, respectively; The number of training samples randomly drawn from the experience replay pool.

[0071] By continuously optimizing the loss function to update the parameters of the Critic network, the agent can accurately evaluate and guide the policy network to generate more effective rule weight combinations, thereby achieving continuous optimization of shop floor scheduling decisions.

[0072] The present invention also provides a two-stage batch flow flexible job shop dynamic scheduling system. The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it performs the two-stage batch flow flexible job shop dynamic scheduling method as described above.

[0073] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the two-stage batch flow flexible job shop dynamic scheduling method as described above.

[0074] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A two-stage batch flow flexible job shop dynamic scheduling method, characterized in that, The steps are as follows: (1) Extract product processing technology features to construct a multi-dimensional batch feature set. After the batch features in the multi-dimensional batch feature set are automatically selected and combined by the genetic programming algorithm, multiple batch scheduling rules are generated through evolutionary operation. The multiple batch scheduling rules form an initial batch scheduling rule set. The batch scheduling rules include batch rules and scheduling rules. The batch rules are used to determine the batch division method of the workpiece. The scheduling rules include sub-batch sorting rules and machine selection rules, which are used to determine the processing order of the sub-batch and the corresponding processing equipment. The initial batch scheduling rule set is represented by a binary tree structure. The internal nodes of the binary tree are operation symbols, which are used to represent the operation relationship between the child nodes. After evolving to a certain stage, the batching rules obtained by evolution are fixed and no longer changed; only the scheduling rules continue to evolve. Based on the fixed batching rules, the genetic programming algorithm generates a set of initial batching scheduling rules suitable for a specific batching scheme through continuous crossover, mutation and selection operations. (2) The scheduling rules in the initial batch scheduling rule set are dynamically optimized by using a deep reinforcement learning algorithm to determine the current optimal workpiece sorting and machine selection strategy, thereby obtaining the scheduling scheme; When a new order arrives, the workpieces in the new order are first divided into batches using fixed batching rules; after the batch division is completed, the determined sub-batch workpieces are added to the task pool. Then, based on the current workshop production environment, the processing priority of each sub-batch is evaluated, and the initial batch scheduling rule set obtained from the evolution is used to preliminarily determine the sub-batch sorting and machine selection strategy.

2. The two-stage batch flow flexible workshop dynamic scheduling method as described in claim 1, characterized in that: In step (2), the deep reinforcement learning algorithm is used to dynamically optimize the weights of the scheduling rules in the initial batch scheduling rule set, including: for each specific scheduling decision, the weight distribution of the initial batch scheduling rule set generated by genetic programming is updated in real time according to the real-time production status information of the workshop.

3. The two-stage batch flow flexible workshop dynamic scheduling method as described in claim 2, characterized in that: In step (2), the use of deep reinforcement learning algorithm to dynamically optimize the weights of the scheduling rules in the initial batch scheduling rule set also includes: after each specific scheduling action is executed, the workshop environment immediately feeds back a real-time reward signal to evaluate the efficiency of the current scheduling decision; at the same time, after the sub-batch of workpieces is completed, the parameters of the deep reinforcement learning algorithm are dynamically adjusted in real time according to the latest production status information and reward feedback in the workshop, so as to optimize the weight combination corresponding to the batch scheduling rules of subsequent decisions in real time.

4. The two-stage batch flow flexible workshop dynamic scheduling method as described in claim 1, characterized in that: In step (2), the use of deep reinforcement learning algorithm to dynamically optimize the scheduling rules in the initial batch scheduling rule set also includes: before assigning weights, uniformly normalizing the evaluation values ​​calculated by all rules; calculating the workpiece sorting rules and machine selection rules by weighting according to the normalization results to obtain a weighted comprehensive score, and determining the sub-batch processing order and machine allocation strategy of the sub-batch workpieces at the current decision point based on the weighted comprehensive score.

5. The two-stage batch flow flexible workshop dynamic scheduling method as described in claim 1, characterized in that: In step (1), the processing method of the genetic programming algorithm also includes: using collaborative genetic programming to optimize the batching rules and scheduling rules; and using dual-tree genetic programming to optimize the sub-batch sorting rules and machine selection rules within the scheduling rules.

6. A two-stage batch flow flexible workshop dynamic scheduling system, characterized in that: The system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it performs the two-stage batch flow flexible workshop dynamic scheduling method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the two-stage batch flow flexible job shop dynamic scheduling method according to any one of claims 1-5.

Images