A production line scheduling method based on data and model hybrid driving

Abstract

The application discloses a production line scheduling method based on data and model hybrid driving, first, according to the resource position and layout, the number and type of operation orders, random noise disturbance and other elements in the flexible production line, a flexible production line scheduling model under random disturbance is constructed; then, according to the data and model hybrid driving framework, the scheduling model is solved; in the data and model hybrid driving framework, the data driving refers to the multi-agent reinforcement learning algorithm based on value decomposition, the model driving refers to the heuristic dispatching rule and experience knowledge, and the hybrid driving of the two mainly embodies three aspects: one is that the scheduling rule is used for guiding the agent strategy selection, two is that the reward function is designed based on the scheduling scene knowledge, and three is that the historical experience and data are supervised to train the noise detection and denoising module; finally, in the disturbance environment, the production line scheduling is carried out by adopting the data and model hybrid driving method, and the application is used for solving the flexible production scheduling under random disturbance.

Description

Technical Field

[0001] This invention relates to flexible production line scheduling, and more specifically to a production line scheduling method based on a hybrid data and model approach. Background Technology

[0002] In such complex manufacturing scenarios, scheduling decisions face challenges such as the inability to accurately model or even the absence of models, making it difficult to meet the requirements for rapid model response. Meanwhile, the popular data-driven artificial intelligence algorithms of recent years rely on large amounts of data for fitting, making it difficult to handle situations where the acquired data is inaccurate. Therefore, there is an urgent need for a new scheduling model that integrates manufacturing scenarios and artificial intelligence. To this end, this paper conducts complementary research on data and model-driven approaches. Summary of the Invention

[0003] In view of this, this invention addresses the problem of poor adaptability of a single-drive model under the customized production mode of multiple varieties and small batches, and proposes a data and model hybrid-drive method to solve flexible production scheduling under random disturbances.

[0004] The technical solution for implementing the present invention is as follows:

[0005] A data- and model-driven hybrid production line scheduling method is proposed. First, a flexible production line scheduling model under random disturbances is constructed based on factors such as resource location and layout, the number and type of work orders, and random noise disturbances. Then, the scheduling model is solved using the data- and model-driven hybrid framework. In this framework, data-driven refers to a multi-agent reinforcement learning algorithm based on value decomposition, while model-driven refers to heuristic assignment rules and empirical knowledge. The hybrid approach is mainly reflected in three aspects: first, using scheduling rules to guide agent strategy selection; second, designing a reward function based on scheduling scenario knowledge; and third, using historical experience and data for supervised training of the noise detection and denoising module. Finally, this data- and model-driven hybrid approach is used for production line scheduling under disturbance conditions.

[0006] Furthermore, the specific steps include:

[0007] Step 1: Model the flexible production line scheduling problem under random disturbances;

[0008] Step 2: Based on the model obtained in Step 1, propose a data and model hybrid driving framework for solving the model;

[0009] Step 3: Based on the data and model hybrid driving framework proposed in Step 2, the scheduling rules are used to guide the agent's policy selection;

[0010] Step 4: Based on the data and model hybrid driving framework proposed in Step 2, design a reward function based on scheduling scenario knowledge;

[0011] Step 5: Based on the data and model hybrid driving framework proposed in Step 2, construct a noise detection and denoising module based on data and model hybrid driving.

[0012] Step 6: Under disturbance conditions, based on the strategy constructed in Step 3, the reward function designed in Step 4, and the detection and denoising module trained in Step 5, adaptive scheduling of the production line is performed using this data and model hybrid driving method.

[0013] Furthermore, in step 1, the specific method for constructing the flexible production line scheduling problem under random disturbances is as follows: based on the resource location and layout of the production line... Quantity and type of work orders Random noise disturbance A flexible production line scheduling model under random perturbations is constructed. Decision points are defined as the end of a certain operation, the insertion of a new order, and machine failure. The model defines the behavior that minimizes the Q-value when a disturbance is observed. ,in, express All permissible disturbances in the surrounding area were observed. It is the predicted mean of the Gaussian distribution.

[0014] Furthermore, in step 2, the data and model hybrid driving framework for solving the scheduling model is constructed using the following method: combining a knowledge model with multi-agent reinforcement learning. Specifically, the data and model hybrid driving approach uses reinforcement learning algorithms as the main body, and leverages rule knowledge as an auxiliary method in three ways to improve the robustness of the algorithm. These three methods are:

[0015] (1) Use scheduling rules to guide agent policy selection and design rule-based scheduling algorithms. Greedy algorithm;

[0016] (2) Design a reward function based on scheduling scenario knowledge and use transportation distance as a penalty for a single agent;

[0017] (3) Construct a noise detection and denoising module driven by a combination of data and model to identify production line disturbances.

[0018] Furthermore, in step 3, the specific method for using the scheduling rules to guide the agent's strategy selection is as follows: when selecting the priority scheduling agent, two heuristic rules are added: maximum remaining task quantity (MLF) and minimum remaining task quantity (LLF), as shown in equation (7).

[0019] (7)

[0020] When selecting the workstation to prioritize for this task, a shortest distance first (SDF) rule is incorporated to design a rule-based approach. The greedy algorithm is shown in equation (8).

[0021] (8)

[0022] Furthermore, in step 4, a reward function is designed based on scheduling scenario knowledge as shown in equation (9).

[0023] (9)

[0024] in, express Move the number of jobs constantly. express Waiting for the number of assignments to be completed. express Current travel time express Maximum distance traveled at any given moment Indicates the maximum completion time. This indicates the maximum completion time for processing all orders. , To set a constant.

[0025] Furthermore, in step 5, the specific method for constructing the noise detection and denoising module based on data and model hybrid driving is as follows: a variational autoencoder structure is used to construct the detection module and the denoising module respectively.

[0026] The variational encoder structure consists of two parts: an encoder and a decoder. The encoder extracts features from the original information, and the decoder reconstructs the extracted features.

[0027] The detection module reports abnormal observations when the decoded observations differ significantly from the encoded observations. It uses historical data free from noise disturbances to pre-train the network by minimizing the reconstruction error. The reconstruction error is shown in Equation (10).

[0028] (10)

[0029] The denoising module maps the abnormal observation data detected by the detection module to the real normal observation, as shown in Equation (12).

[0030] (12)

[0031] Furthermore, in step 6, the specific method for production line scheduling using this data and model hybrid driving approach is as follows: Under a disturbed environment, adaptive production line scheduling is performed based on the strategy constructed in step 3, the reward function designed in step 4, and the detection and denoising module trained in step 5. First, the scheduling task is loaded, job and workstation information is initialized, and the experience replay pool and reinforcement learning value network parameters are initialized. and policy network parameters Then, it checks if the set of jobs to be scheduled is empty. If it is not empty, it retrieves the currently available actions. Calculate the distance between the current location and the available workstation locations, in order to Probability-based model-driven heuristic rule selection for actions, in order to Probability Selection The action with the highest value is selected. Next, this scheduling action is executed, and the reward from the environment is received. The current task is removed from the scheduling set, and the next task is added. Scheduling and processing times are calculated, and the environment state is updated. The above steps are repeated until the scheduling set is empty. Finally, the scheduling process sequence and maximum completion time are output.

[0032] Beneficial effects

[0033] This invention studies the scheduling problem of flexible production lines. Addressing the shortcomings of current scheduling algorithms in single-drive mode—namely, the inability to simultaneously overcome the drawbacks of long offline training times, ineffective exploration in the early stages of training, sparse rewards, and difficulty in accurate modeling in model-driven mode—this invention integrates a scheduling knowledge model into a multi-agent reinforcement learning algorithm. Utilizing the knowledge inherent in scheduling rules, worker experience, and historical data, it accelerates data-driven convergence from three perspectives: strategy selection, reward convergence, and detection denoising. This improves the effectiveness and superiority of the scheduling algorithm, providing an optimized control scheme for modular control in intelligent manufacturing and production scheduling under noise interference. Attached Figure Description

[0034] Figure 1 A scheduling framework driven by a hybrid approach of data and models.

[0035] Figure 2 A data-driven framework for knowledge enhancement.

[0036] Figure 3 This is a framework for error detection and noise reduction modules for production line scheduling.

[0037] Figure 4 This is a structural diagram of an automatic encoder.

[0038] Figure 5 A comparison chart of rewards for algorithms driven by a hybrid data and model approach.

[0039] Figure 6This chart compares the maximum completion time of each algorithm with that of the hybrid data and model-driven algorithm.

[0040] Figure 7 This is a comparison chart of the reward and baseline before and after noise reduction. Detailed Implementation

[0041] A production line scheduling method based on a hybrid data and model approach, characterized by the following steps:

[0042] Step 1: Model the flexible production line scheduling problem under random disturbances. The problem model for flexible production line scheduling under random disturbances includes three parts: scheduling environment, problem description, and disturbance noise.

[0043] The specific method for constructing a flexible production line scheduling environment under random disturbances is as follows:

[0044] The flexible production line scheduling environment includes n scheduling jobs: and m workstations: The work is modeled as an agent in reinforcement learning, where each workstation has a fixed location and contains different manufacturing resources. Each scheduled job contains One operation: In terms of specific operations, as long as the manufacturing resources include the first... Resources required for each operation This is considered as an operation that can be processed; the processing time for each operation by the manufacturing resources is shown in equation (1):

[0045] (1)

[0046] During scheduling, jobs are assigned to specific workstations at decision points, and manufacturing resources with available resources are selected to complete the operations sequentially. Any two scheduled jobs may differ in the "number of operations," "operation type," and "operation order." A decision point is defined as:

[0047] (1) At the end of the operation, it refers to the completion time of the last operation allocated to a certain manufacturing resource;

[0048] (2) When a new order is inserted, it is indicated that a random job has arrived, and the task of the newly added agent is included in the job pool to be completed;

[0049] (3) In case of machine failure, it is indicated as At this point, a certain workstation or a certain type of resource on a workstation becomes unavailable.

[0050] The specific method for describing the problem of constructing a flexible production line under random perturbations is as follows:

[0051] The flexible job scheduling problem is modeled as a seven-tuple. . This represents the global state, i.e., the global information of the current scheduling environment, such as the processing progress of all jobs and available robots. It is expressed as follows:

[0052] (2)

[0053] in, Indicates the location of currently available manufacturing resources. This represents the remaining operations for all jobs. The scheduling objective is to allocate available workstation resources to each job agent at the decision point to minimize the maximum completion time. The transition time matrix for manufacturing resources is defined as follows: The maximum completion time is shown in equation (3).

[0054] (3)

[0055] In each step of the action, each task agent Choose an action based on a certain strategy. The actions of multiple agents are combined into a single coordinated action. Causes changes in the state of the environment; Indicates job agent The strategy model, discount factor All job agents share the same reward function. Each job agent Observation status It is expressed as follows:

[0056] (4)

[0057] in, This indicates the distance between the current job agent i and other manufacturing resources. This indicates the number of remaining operations to be processed in this job. This indicates the next step to be taken. Indicates the time of the operation. This indicates interference noise.

[0058] The specific method for constructing interference noise in a flexible production line under random perturbation is as follows:

[0059] Interference noise Using the ground-truth surrounding the observations The norm region is used to represent it, as shown in equation (5).

[0060] (5)

[0061] in, yes The radius of the norm region. Defined as the factor that minimizes the Q-value when interference observations occur, i.e., the agent's choice of policy. The observation with the smallest Q value, and the interfering observation. As shown in equation (6).

[0062] (6)

[0063] in, express All permissible disturbances in the surrounding area were observed. It is the predicted mean of the Gaussian distribution.

[0064] Step 2: Based on the model obtained in Step 1, a data- and model-driven hybrid framework for solving the model is proposed. This framework combines knowledge models with multi-agent reinforcement learning, such as... Figure 1 As shown. Among them, the data-driven approach mainly adopts the VDN, QMIX and QTRAN_alt algorithms in Multi-Agent Reinforcement Learning (MARL), while the model-driven approach mainly adopts historical experience data and three manually set scheduling rules, namely First-In-First-Out (FIFO), Least Remaining Operands First-Come-Out (MLF) or Least Remaining Operands First-Come-Out (LLF) and Shortest Distance First (SDF) rules.

[0065] Specifically, the data- and model-driven approach primarily uses reinforcement learning algorithms, employing rule-based knowledge as an auxiliary method in three ways to enhance the algorithm's robustness, such as... Figure 2 As shown. The three methods are as follows:

[0066] (1) Use scheduling rules to guide agent policy selection and design rule-based scheduling algorithms. Greedy algorithm;

[0067] (2) Design a reward function based on scheduling scenario knowledge and use transportation distance as a penalty for a single agent;

[0068] (3) Construct a noise detection and denoising module driven by a combination of data and model to identify production line disturbances.

[0069] Step 3: Based on the data and model hybrid driving framework proposed in Step 2, the specific method for using scheduling rules to guide agent policy selection is as follows: In the scheduling problem, the agent's policy selection action can be divided into two steps: first, selecting the job to be scheduled first, and second, selecting the workstation that the job should go to first. When selecting the job agent to be scheduled first, two heuristic rules, maximum remaining task volume (MLF) and minimum remaining task volume (LLF), are added, as shown in Equation (7).

[0070] (7)

[0071] When selecting the workstation to prioritize for this job, incorporate the Shortest Distance First (SDF) rule. The random exploration part of the greedy algorithm is replaced with exploration based on heuristic rules; a rule-based approach is designed. The greedy algorithm is shown in equation (8).

[0072] (8)

[0073] Step 4: Based on the data and model hybrid driving framework proposed in Step 2, the specific method for designing the reward function based on scheduling scenario knowledge is as follows: In addition to considering giving a large reward when the scheduling task is completed, appropriate rewards or penalties are also given for the parameter variables involved in the scheduling process. Specifically, this includes rewarding the number of jobs moved at the decision point, penalizing the number of jobs waiting at the decision point, and penalizing the long distance traveled from the current workstation to the next workstation. The reward function is shown in Equation (9).

[0074] (9)

[0075] in, express Move the number of jobs constantly. express Waiting for the number of assignments to be completed. express Current travel time express Maximum distance traveled at any given moment Indicates the maximum completion time. This indicates the maximum completion time for processing all orders. , It is a constant.

[0076] Step 5: Based on the data and model hybrid driving framework proposed in Step 2, the specific method for constructing the noise detection and denoising module based on data and model hybrid driving is as follows: A variational autoencoder structure is used to construct the detection module and the denoising module respectively, such as... Figure 3 As shown.

[0077] The variational encoder structure consists of two parts: an encoder and a decoder, as follows: Figure 4 As shown. The encoder extracts features from the raw information, with the raw observations as input. These observations are passed through a recurrent neural network (GRU) layer and two distinct linear layers to generate a Gaussian distribution with mean and log-variance. Features are sampled from this Gaussian distribution. The extracted features are then passed to the decoder; the decoder reconstructs the features to make the reconstructed output as close as possible to the original information. The decoder uses a GRU layer and a linear layer to process the data. Decode it.

[0078] The detection module reports abnormal observations when the decoded and encoded observations differ significantly. It pre-trains the network using historical data free of noise disturbances and by minimizing the reconstruction error. The reconstruction error is defined as the input observation... and output observations The mean square error between them; if this value is greater than the threshold of the anomaly detection index. The detection module will then report an anomaly, as shown in equation (10).

[0079] (10)

[0080] The objective function of the detection module is shown in equation (11).

[0081] (11)

[0082] in, For encoder parameters, These are the parameters for the decoder; It is a moment Normal observation value; and These are the hidden states and latent variables of the encoder and decoder. Sample from the encoder-parameterized distribution. For latent variables... Decoding can reconstruct the input observations. To achieve the goal of reconstruction, It is the distribution generated by the encoder. The KL divergence between the distribution and the prior distribution.

[0083] The denoising module maps the abnormal observation data detected by the detection module to the real normal observation, as shown in Equation (12).

[0084] (12)

[0085] The objective function of the noise reduction module is shown in equation (13).

[0086] (13)

[0087] Objective function of the detection module In comparison, encoder The input is normal observation value and hidden state The encoder of the noise reduction module will be replaced. and hidden state Mapping to a latent variable This variable is input into the decoder. Generate real observations The corrected observations are used to generate actions through a policy network.

[0088] Step 6: Under perturbed conditions, based on the strategy constructed in Step 3, the reward function designed in Step 4, and the detection and denoising module trained in Step 5, the specific method for production line scheduling using this data and model hybrid driving approach is as follows:

[0089] In a perturbed environment, the policy constructed in step 3, the reward function designed in step 4, and the detection and denoising module trained in step 5 are integrated into a value decomposition-based multi-agent reinforcement learning algorithm (VDN, QMIX, and QTRAN).

[0090] First, load the scheduled task, initialize job and workstation information, and initialize the experience replay pool and reinforcement learning value network parameters. and policy network parameters Then, it checks if the set of jobs to be scheduled is empty. If it is not empty, it retrieves the currently available actions. And calculate the distance between the current location and the available workstation locations, in order to Probability-based model-driven heuristic rule selection for actions, in order to The action with the highest Q-value is selected based on probability. This action is then executed, and the reward from the environment is received. The current task is removed from the scheduling set, and the next task for this job is added. Simultaneously, scheduling and processing times are calculated, and the environment state is updated. These steps are repeated until the scheduling set is empty. Finally, the scheduling sequence and maximum completion time are output.

[0091] The effectiveness of the above method is verified by simulation using Python, as shown in the following example.

[0092] To evaluate the overall performance of different scheduling algorithms, we conducted a comparative simulation experiment.

[0093] Experiment 1: Under four different production line scenarios (Scenario 1: Workstation coordinates of type A, 8 agents, QMIX algorithm; Scenario 2: Workstation coordinates of type A, 10 agents, QMIX algorithm; Scenario 3: Workstation coordinates of type B, 10 agents, VDN algorithm; Scenario 4: Workstation coordinates of type A, 10 agents, QTRAN algorithm), by changing the workstation coordinates, the number of agents, and the multi-agent reinforcement algorithm, the hybrid driving algorithm proposed in this invention was determined to be superior in all situations. A comparison was made between the single data-driven algorithm and the data-and-model hybrid driving algorithm. Set it to 0.05 and record the Reward curve during training as follows: Figure 5 As shown.

[0094] Experiment 2: Using maximum completion time as the evaluation metric, and taking the QTRAN_alt algorithm as an example, compare the data- and model-driven MARL algorithm with random action selection scheduling, model-based rule scheduling algorithms (FIFO, MLF, LLF), and data-driven MARL algorithms. Figure 6 As shown in Table 1, to ensure the fairness of the experimental results, we conducted multiple sets of experiments and recorded the maximum completion times of the single model-driven algorithm, the single data-driven algorithm, and the hybrid data model-driven algorithm.

[0095] Experiment 3: Taking the QMIX algorithm as an example, the algorithm was trained 15,000 times using a mixture of noise-free data and the model as a baseline. The reward curves before and after denoising were compared as follows: Figure 7 As shown.

[0096] The results show that, as seen in Experiment 1, the oscillation amplitude and convergence speed of the reward curve of the data- and model-driven algorithm are both superior to those of the single data-driven algorithm; as seen in Experiment 2, in Figure 6 The random action selection scheduling time is the longest. The single data-driven algorithm is better than the single model-driven algorithm, while the data and model hybrid driving algorithm has the shortest maximum completion time. In 16 out of the 18 cases listed in Table 1, the data and model hybrid driving algorithm has the shortest maximum completion time, which proves the effectiveness of the proposed method. As can be seen from Experiment 3, after the detection and denoising module, the stability and upper limit of the reward curve have increased. The constructed detection and denoising module has improved the performance of the scheduling algorithm under noise interference to a certain extent.

[0097]

[0098] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A production line scheduling method based on data and model hybrid driving, characterized in that, Firstly, according to the elements of resource location and layout in flexible production line, order quantity and type, random noise disturbance, a flexible production line scheduling model under random disturbance is constructed; Then, according to the data and model hybrid driving framework, the scheduling model is solved; in the data and model hybrid driving framework, data driving refers to the value decomposition based multi-agent reinforcement learning algorithm, model driving refers to heuristic dispatching rules and experience knowledge, and the hybrid driving of the two is reflected in three aspects: first, the scheduling rules are used to guide the agent strategy selection, second, the reward function is designed based on the scheduling scene knowledge, and third, the historical experience and data are supervised to train the noise detection and denoising module; Finally, in the disturbance environment, the data and model hybrid driving method is used for production line scheduling; The steps of the method comprise: Step 1: modeling the flexible production line scheduling problem under random disturbance; Step 2: according to the model obtained in step 1, a data and model hybrid driving framework for solving the model is proposed; Step 3: according to the data and model hybrid driving framework proposed in step 2, the scheduling rules are used to guide the agent strategy selection; Step 4: according to the data and model hybrid driving framework proposed in step 2, the reward function is designed based on the scheduling scene knowledge; Step 5: according to the data and model hybrid driving framework proposed in step 2, a noise detection and denoising module based on data and model hybrid driving is constructed; Step 6: in the disturbance environment, according to the strategy constructed in step 3, the reward function designed in step 4 and the detection and denoising module trained in step 5, the production line adaptive scheduling is carried out in the data and model hybrid driving manner; The specific method for constructing the flexible production line scheduling problem under random disturbance in step 1 is as follows: according to the resource position and layout of the production line , the number and type of operation orders , random noise disturbance , a flexible production line scheduling model under random disturbance is constructed; the decision point is defined as the end of a process operation, the insertion of a new order and the failure of a machine; when the disturbance observation appears, , wherein represents all allowed disturbance observations around, is the mean of the predicted Gaussian distribution; In step 2, the data and model hybrid driving framework for solving the scheduling model is constructed in the following specific method: the knowledge model is combined with the multi-agent reinforcement learning, specifically, the data and model hybrid driving method is to take the reinforcement learning algorithm as the main body, and use rule knowledge as auxiliary to improve the robustness of the algorithm in three ways: (1) The scheduling rules are used to guide the agent policy selection, and a rule-based Greedy algorithm (2) the reward function is designed based on the scheduling scene knowledge, and the transportation distance is used as the punishment for a single agent; (3) a noise detection and denoising module based on data and model hybrid driving is constructed to identify the disturbance of the production line; In step 3, the specific method of using scheduling rules to guide agent strategy selection is to add two heuristic rules of maximum remaining task quantity MLF and minimum remaining task quantity LLF when selecting the priority scheduling job agent; In selecting the workstations to which the job is preferentially forwarded, a shortest distance first (SDF) rule is added, and a rule-based Greedy algorithm In step 4, the reward function designed based on the scheduling scene knowledge is: wherein, represents the number of moving jobs at the time, represents the number of waiting jobs at the time, represents the current travel time at the time, represents the maximum travel time at the time, represents the maximum completion time, represents the maximum completion time for completing all orders, , is a set constant; In step 5, the specific method of constructing the noise detection and denoising module based on data and model hybrid driving is to use a variational autoencoder structure to construct the detection module and the denoising module respectively; The variational autoencoder structure comprises an encoder and a decoder, the encoder extracts features from the original information, and the decoder reconstructs the extracted features; The detection module reports abnormal observation values when the decoded observation values and the encoded observation values are significantly different, and the network is pre-trained by minimizing the reconstruction error using historical data without noise disturbance; The denoising module maps the abnormal observation data discovered by the detection module to real normal observation data: In step 6, the specific method for production line scheduling using this data and model hybrid driving approach is as follows: Under a disturbed environment, adaptive scheduling of the production line is performed based on the strategy constructed in step 3, the reward function designed in step 4, and the detection and denoising module trained in step 5. First, the scheduling task is loaded, job and workstation information is initialized, and the experience replay pool and reinforcement learning value network parameters are initialized. and policy network parameters Then, determine if the set of jobs to be scheduled is empty; if not, obtain the currently available actions. Calculate the distance between the current location and the available workstation locations, in order to Probability-based model-driven heuristic rule selection for actions, in order to Probability Selection The action with the highest value is selected; then, the scheduling action is executed and the reward given by the environment is received. The current process is removed from the set to be scheduled, and the next process is added; the scheduling and processing time are calculated, and the environment state is updated; the above steps are repeated until the set of jobs to be scheduled is empty. Finally, the scheduling process sequence and the maximum completion time are output.

Images