A non-correlated parallel machine scheduling method based on deep reinforcement learning

Abstract

The present invention belongs to the field of industrial intelligence technology and is a non-correlated parallel machine scheduling method based on deep reinforcement learning. The method comprises the following steps: constructing a mathematical model suitable for non-correlated parallel machine scheduling; collecting and normalizing the processing time of each workpiece on each heterogeneous machine; initializing a genetic algorithm scheduling population and a deep Q network; constructing a deep reinforcement learning training framework; using average fitness, optimal fitness, and optimal individual encoding to represent the state vector; using action space to control the operating parameters of the genetic algorithm and using a reward function to update the parameters of the deep Q network; and using the trained deep reinforcement learning model to dynamically control operator selection during the genetic algorithm iteration process to obtain the optimal scheduling solution and implement the scheduling of non-correlated parallel machines. By analyzing the data model of the non-correlated parallel machine scheduling problem, the present invention provides a deep reinforcement learning algorithm for solving similar problems in the field of manufacturing production scheduling, thereby improving production efficiency.

Description

Technical Field

[0001] The present invention relates to the field of industrial intelligence technology, and in particular to a non-correlated parallel machine scheduling method based on deep reinforcement learning. Background Art

[0002] The Unrelated Parallel Machine Scheduling Problem (UPMSP) is a typical combinatorial optimization problem in the field of scheduling optimization, widely used in complex production scenarios such as manufacturing systems, computing resource allocation, and logistics scheduling. In an unrelated parallel machine environment, the processing time of different workpieces on different machines varies. This processing time depends not only on the workpiece's own properties but also on the type of machine to which it is assigned, reflecting a high degree of heterogeneity and uncertainty.

[0003] Traditional problem-solving methods, such as heuristic algorithms, genetic algorithms (GAs), simulated annealing (SAs), and particle swarm optimization (PSOs), can achieve acceptable solutions to a certain extent. However, they are prone to falling into local optima when dealing with large-scale instances or dynamic environments, are highly parameter-sensitive, and lack adaptability. In recent years, deep reinforcement learning (DRL), an intelligent optimization method that combines perception and decision-making, has demonstrated strong potential in complex environment control and policy learning, providing a new paradigm for solving scheduling problems.

[0004] However, most current scheduling research is still limited to fixed rules or static parameter configurations, making it difficult to dynamically adjust optimization strategies based on the current search state. There is also a lack of systematic approaches that combine deep reinforcement learning with traditional evolutionary algorithms for scheduling unrelated parallel machines. Therefore, an intelligent scheduling framework that balances global search capabilities with local adaptive adjustment mechanisms is urgently needed to improve solution quality and generalization capabilities. Summary of the Invention

[0005] In response to the technical problems raised above, a method for scheduling non-correlated parallel machines based on deep reinforcement learning is provided. The present invention mainly utilizes a genetic algorithm combined with a deep reinforcement learning algorithm to achieve an efficient and intelligent solution to the heterogeneous parallel machine scheduling problem.

[0006] The technical means adopted in the present invention are as follows:

[0007] A non-correlated parallel machine scheduling method based on deep reinforcement learning, comprising:

[0008] Construct a mathematical model for scheduling non-correlated parallel machines;

[0009] Collect the processing time of each workpiece on various heterogeneous machines and perform normalization;

[0010] Initialize the genetic algorithm scheduling population and deep Q network;

[0011] Build a deep reinforcement learning training framework and define the state space, action space, and reward function;

[0012] The state vector is represented by using average fitness, optimal fitness and optimal individual encoding;

[0013] Use the action space to control the operating parameters of the genetic algorithm and use the reward function to update the parameters of the deep Q network;

[0014] By utilizing the trained deep reinforcement learning model, the operator selection is dynamically controlled during the genetic algorithm iteration process to obtain the optimal scheduling solution and realize the scheduling of non-correlated parallel machines.

[0015] Furthermore, the mathematical model for scheduling non-coherent parallel machines specifically includes:

[0016] Define the basic set and symbols of scheduling problem, let the job set , a collection of heterogeneous machines , each job Can be assigned to any machine , but homework The processing time on different machines is different, which constitutes a non-correlation feature; the processing time matrix is ​​expressed as , Indicates homework In the machine Processing time on

[0017] The scheduling decision variables are defined as:

[0018]

[0019] Establish the scheduling optimization objective function, set For homework The optimization objective function is to minimize the total completion time of all jobs. , establish scheduling constraints:

[0020] Each job must be assigned to one and only one machine:

[0021]

[0022] On the same machine, any operation cannot be overlapped. 、 Assigned to the same machine , then it is necessary to constrain the processing order and define the variable Indicates homework Start processing time;

[0023] Completion time constraint, the completion time of each job is the sum of the start time and the processing time:

[0024]

[0025] Among them, the variable value range .

[0026] Furthermore, the processing time of each workpiece on each heterogeneous machine is Structured into matrix format:

[0027]

[0028] in, Indicates the i workpieces; represents the jth machine; For workpiece In the machine processing time;

[0029] The matrix The missing values ​​in are set to = , and forcibly exclude the corresponding job allocation in the scheduling constraints, eliminate extreme values, and unify the data scale by normalizing the matrix:

[0030] .

[0031] Furthermore, the initialization of the genetic algorithm scheduling population and the deep Q network specifically includes:

[0032] Assuming the initial population size is N, the initialized genetic algorithm scheduling population includes the job allocation code and processing sequence:

[0033] The job assignment encoding is represented as: ;

[0034] With the goal of minimizing the total completion time of all jobs, the individual fitness is calculated and the fitness function is defined as:

[0035]

[0036] Introducing Deep Q Network , estimate the long-term expected return value of each action combination under the scheduling state, combine the experience revisit pool D, sample data from the experience revisit pool each time, and perform TD target update:

[0037]

[0038] in, Represents the samples in the experience revisit pool The expected value of the quadruple, Indicates the possible actions to be taken in the next state. Indicates the current network parameters, represents the target network parameters, represents the discount factor, Indicates the reward obtained from the action in this round.

[0039] Furthermore, the state space S is used to represent the current evolution progress and reflect the structure of the current optimal scheduling solution;

[0040] The action space Every action in Corresponding to a set of genetic operator configurations, the total number of action spaces is:

[0041]

[0042] in, Indicates a selection operation. Indicates the cross mode, Indicates the variation mode, Indicates ratio;

[0043] The reward function uses the reward system in reinforcement learning to feedback the scheduling effect of the current selection operation; the reward value Expressed as:

[0044]

[0045] Among them, The total completion time of the optimal individual in each round is , No. The optimal individual completion time for each round is .

[0046] Furthermore, the average fitness represents the average of the total completion time of all individuals in the population. Expressed as:

[0047]

[0048] The optimal fitness is the completion time of the current optimal scheduling solution, representing the quality of the optimal solution in the current search space, expressed as:

[0049]

[0050] in, Represents all individuals The objective function value of The minimum value in ;

[0051] The optimal individual coding Expressed as:

[0052]

[0053] The state vector is expressed as:

[0054]

[0055] in, Represents the normalized iteration progress;

[0056] Furthermore, the training process of the deep reinforcement learning model specifically includes:

[0057] Extract the state vector of the current population ; Input the state vector into the trained deep Q network , output the Q value of all actions;

[0058] Use the ε-greedy strategy to select the current action:

[0059]

[0060] Execute an action , update the population according to the selected operator ; Calculate the new population fitness value and evaluate the optimal fitness ;Store state transfer quad to the experience pool; use batch samples to update parameters in Q network training to minimize TD error;

[0061] When the stopping condition is met, the current optimal individual is output as the final scheduling solution, including the machine allocation and processing sequence of all workpieces:

[0062]

[0063] in, Represents the total completion time of the scheduling solution.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] The non-correlated parallel machine scheduling method based on deep reinforcement learning provided by the present invention constructs a mathematical model suitable for non-correlated parallel machine scheduling; collects the processing time of each workpiece on each heterogeneous machine and performs normalization processing; initializes the genetic algorithm scheduling population and deep Q network; constructs a deep reinforcement learning training framework, defines the state space, action space and reward function; uses average fitness, optimal fitness and optimal individual encoding to represent the state vector; uses the action space to control the operating parameters of the genetic algorithm, and uses the reward function to update the parameters of the deep Q network; uses the trained deep reinforcement learning model to dynamically control operator selection during the genetic algorithm iteration process to obtain the optimal scheduling solution and realize the scheduling of non-correlated parallel machines. The present invention provides a deep reinforcement learning algorithm that can be directly used to solve similar problems in the field of manufacturing production scheduling by analyzing the data model research of non-correlated parallel machine scheduling problems and combining it with an intelligent solution algorithm based on reinforcement learning algorithm. Compared with previous hyper-heuristic algorithms, it will improve algorithm performance and make more reasonable scheduling solutions.

[0066] Based on the above reasons, the present invention can be widely promoted in fields such as industrial intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0068] Figure 1 This is a flow chart of the non-correlated parallel machine scheduling method based on deep reinforcement learning in the present invention.

[0069] Figure 2 This is a schematic diagram of the deep reinforcement learning scheduling control algorithm structure in the present invention.

[0070] Figure 3 Schematic diagram of the structure of the deep reinforcement learning algorithm in the present invention controlling the genetic algorithm to generate scheduling individuals. DETAILED DESCRIPTION

[0071] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0073] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0074] Unless otherwise specified, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention. At the same time, it should be clear that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of the exemplary embodiments can have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0075] like Figure 1 As shown, the present invention provides a non-correlated parallel machine scheduling method based on deep reinforcement learning, comprising:

[0076] Construct a mathematical model suitable for scheduling non-correlated parallel machines; in specific implementation, as a preferred embodiment of the present invention, the mathematical model for scheduling non-correlated parallel machines specifically includes:

[0077] Define the basic set and symbols of scheduling problem, let the job set , a collection of heterogeneous machines , each job Can be assigned to any machine , but homework The processing time on different machines is different, which constitutes a non-correlation feature; the processing time matrix is ​​expressed as , Indicates homework In the machine Processing time on

[0078] The scheduling decision variables are defined as:

[0079]

[0080] Establish the scheduling optimization objective function, set For homework The optimization objective function is to minimize the total completion time of all jobs. , establish scheduling constraints:

[0081] Each job must be assigned to one and only one machine:

[0082]

[0083] On the same machine, any operation cannot be overlapped. 、 Assigned to the same machine , then it is necessary to constrain the processing order and define the variable Indicates homework Start processing time;

[0084] Completion time constraint, the completion time of each job is the sum of the start time and the processing time:

[0085]

[0086] Among them, the variable value range .

[0087] The processing time of each workpiece on each heterogeneous machine is collected and normalized. In specific implementation, as a preferred embodiment of the present invention, the processing time data of each workpiece on each heterogeneous machine is collected. The scheduling model of the present invention requires a complete grasp of the processing time of each workpiece on different machines, so a two-dimensional processing time matrix needs to be established. The processing time of each workpiece on each heterogeneous machine is normalized. Structured into matrix format:

[0088]

[0089] in, Indicates the i workpieces; represents the jth machine; For workpiece In the machine processing time; since it is a non-correlated parallel machine problem, the values ​​of each column (machine) corresponding to each row (job) in the matrix may be different and have no proportional relationship.

[0090] The matrix reflects the processing heterogeneity of each workpiece and is the core feature that distinguishes the non-correlated parallel machine scheduling problem from the traditional parallel machine scheduling. The missing values ​​in are set to = , and forcibly exclude the corresponding job allocation in the scheduling constraints, eliminate extreme values, and unify the data scale by normalizing the matrix:

[0091] .

[0092] Initializing the genetic algorithm scheduling population and the deep Q network; in specific implementation, as a preferred embodiment of the present invention, initializing the genetic algorithm scheduling population and the deep Q network specifically includes:

[0093] Set the initial population size to N, and initialize the genetic algorithm scheduling population to include job allocation coding and processing sequence:

[0094] The job assignment encoding is represented as: ;

[0095] The processing sequence of possible jobs on each machine is: , .

[0096] With the goal of minimizing the total completion time of all jobs, the individual fitness is calculated and the fitness function is defined as:

[0097]

[0098] Introducing Deep Q Network , estimate the long-term expected return value of each action combination under the scheduling state, combine the experience revisit pool D, sample small batches of data from the experience revisit pool each time, and perform TD target update:

[0099]

[0100] in, Represents the samples in the experience revisit pool The expected value of the quadruple, Indicates the possible actions to be taken in the next state. Indicates the current network parameters, represents the target network parameters, represents the discount factor, Indicates the reward obtained by this round of action. When implemented, the network structure As follows: The input layer dimension is the state vector dimension ;The output layer dimension is the action space ; A hidden layer is set in the middle, and its structure is as follows:

[0101]

[0102] Construct a deep reinforcement learning training framework and define the state space, action space, and reward function. In specific implementation, as a preferred embodiment of the present invention, the state space S is used to represent the current evolution progress and reflect the structure of the current optimal scheduling solution.

[0103] Action Space Every action in The total number of action spaces corresponding to a set of genetic operator configurations is:

[0104]

[0105] in, Indicates a selection operation. Indicates the cross mode, Indicates the variation mode, Indicates ratio;

[0106] During implementation, the action space consists of selection, crossover, and mutation operators. Selection operators include roulette wheel selection, tournament selection, and ranked selection. Roulette wheel selection is used to probabilistically select individuals, with the probability of selection proportional to their fitness. Tournament selection randomly selects individuals from the population and chooses the best individual for mating. Ranked selection ranks individuals according to their fitness, selecting them based on the ranking probability. Crossover operators include mapping crossover and sequential crossover. Mutation operator options include swap mutation and reverse mutation, with a configurable mutation rate. Crossover mutation is used to randomly swap the assignments of two artifacts; reverse mutation is used to reverse the selection of artifact subsequences.

[0107] The reward function uses the reward system in reinforcement learning to feedback the scheduling effect of the current selected operation; the reward value Expressed as:

[0108]

[0109] Among them, The total completion time of the optimal individual in each round is , No. The optimal individual completion time for each round is . Reward function This is used to evaluate the improvement in the quality of the currently selected genetic operator on the scheduling solution, with the ultimate goal of optimizing the total completion time. Through reward feedback, the reinforcement learning model can learn which actions contribute to the optimization of the scheduling solution, thereby improving the strategy.

[0110] The state vector is represented by using the average fitness, the best fitness and the best individual encoding; in specific implementation, as a preferred embodiment of the present invention, the average fitness represents the average value of the total completion time of all individuals in the population, and the average fitness is Expressed as:

[0111]

[0112] Optimal fitness is the completion time of the current optimal scheduling solution, representing the quality of the optimal solution in the current search space, expressed as:

[0113]

[0114] in, Represents all individuals The objective function value of The minimum value in , that is, the optimal value of the solution in the current search space.

[0115] Optimal individual coding Expressed as:

[0116]

[0117] The state vector is expressed as:

[0118]

[0119] in, Represents the normalized iteration progress.

[0120] Use the action space to control the operating parameters of the genetic algorithm and use the reward function to update the parameters of the deep Q network;

[0121] Utilizing a trained deep reinforcement learning model, we dynamically control operator selection during the genetic algorithm iteration process to obtain the optimal scheduling solution and achieve scheduling of non-correlated parallel machines. Utilizing a trained deep reinforcement learning module, we dynamically control operator selection during the genetic algorithm iteration process to gradually obtain the optimal scheduling solution. By combining the advantages of reinforcement learning and genetic algorithms, this invention uses a trained Q network to dynamically control operator selection in each round of the genetic algorithm, thereby optimizing the scheduling solution.

[0122] In specific implementation, as a preferred embodiment of the present invention, the training process of the deep reinforcement learning model specifically includes:

[0123] Extract the state vector of the current population ; Input the state vector into the trained deep Q network , output the Q value of all actions;

[0124] Use the ε-greedy strategy to select the current action:

[0125]

[0126] Execute an action , update the population according to the selected operator ; Calculate the new population fitness value and evaluate the optimal fitness ;Store state transfer quad to the experience pool; use batch samples to update parameters in Q network training to minimize TD error;

[0127] When the stopping condition is met, the current optimal individual is output as the final scheduling solution, including the machine allocation and processing sequence of all workpieces:

[0128]

[0129] in, Represents the total completion time of the scheduling solution.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-correlated parallel machine scheduling method based on deep reinforcement learning, characterized in that: include: Construct a mathematical model for scheduling non-correlated parallel machines; The mathematical model of the non-correlated parallel machine scheduling specifically includes: Define the basic set and symbols of scheduling problem, let the job set , a collection of heterogeneous machines , each job Can be assigned to any machine , but homework The processing time on different machines is different, which constitutes a non-correlation feature; the processing time matrix is ​​expressed as , Indicates homework In the machine Processing time on The scheduling decision variables are defined as: Establish the scheduling optimization objective function, set For homework The optimization objective function is to minimize the total completion time of all jobs. , establish scheduling constraints: Each job must be assigned to one and only one machine: On the same machine, any operation cannot be overlapped. 、 Assigned to the same machine , then it is necessary to constrain the processing order and define the variable Indicates homework Start processing time; Completion time constraint, the completion time of each job is the sum of the start time and the processing time: Among them, the variable value range ; Collect the processing time of each workpiece on various heterogeneous machines and perform normalization; Initialize the genetic algorithm scheduling population and deep Q network; Build a deep reinforcement learning training framework and define the state space, action space, and reward function; The state vector is represented by using average fitness, optimal fitness and optimal individual encoding; Use the action space to control the operating parameters of the genetic algorithm and use the reward function to update the parameters of the deep Q network; By utilizing the trained deep reinforcement learning model, the operator selection is dynamically controlled during the genetic algorithm iteration process to obtain the optimal scheduling solution and realize the scheduling of non-correlated parallel machines.

2. The non-correlated parallel machine scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The processing time of each workpiece on each heterogeneous machine Structured into matrix format: in, Indicates the i workpieces; represents the jth machine; For workpiece In the machine processing time; The matrix The missing values ​​in are set to = , and forcibly exclude the corresponding job allocation in the scheduling constraints, eliminate extreme values, and unify the data scale by normalizing the matrix: 。 3. The non-correlated parallel machine scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The initialization of the genetic algorithm scheduling population and the deep Q network specifically includes: Assuming the initial population size is N, the initialized genetic algorithm scheduling population includes the job allocation code and processing sequence: the job allocation code is expressed as: ; With the goal of minimizing the total completion time of all jobs, calculate the individual fitness and define the fitness function as: Introducing Deep Q Network , estimate the long-term expected return value of each action combination under the scheduling state, combine the experience revisit pool D, sample data from the experience revisit pool each time, and perform TD target update: in, Represents the samples in the experience revisit pool expected value of a quadruple; Indicates the possible actions to be taken in the next state. Indicates the current network parameters, represents the target network parameters, represents the discount factor, Indicates the reward obtained from the action in this round.

4. The non-correlated parallel machine scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The state space S is used to represent the current evolution progress and reflect the structure of the current optimal scheduling solution; The action space Every action in Corresponding to a set of genetic operator configurations, the total number of action spaces is: in, Indicates a selection operation. Indicates the cross mode, Indicates the variation mode, Indicates ratio; The reward function uses the reward system in reinforcement learning to feedback the scheduling effect of the current selection operation; the reward value Expressed as: Among them, The total completion time of the optimal individual in each round is , No. The optimal individual completion time for each round is .

5. The non-correlated parallel machine scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The average fitness represents the average of the total completion time of all individuals in the population. Expressed as: The optimal fitness is the completion time of the current optimal scheduling solution, representing the quality of the optimal solution in the current search space, expressed as: in, Represents all individuals The objective function value of The minimum value in ; The optimal individual coding Expressed as: The state vector is expressed as: in, Represents the normalized iteration progress.

6. The non-correlated parallel machine scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The training process of the deep reinforcement learning model specifically includes: Extract the state vector of the current population ; Input the state vector into the trained deep Q network , output the Q value of all actions; Use the ε-greedy strategy to select the current action: Execute an action , update the population according to the selected operator ; Calculate the new population fitness value and evaluate the optimal fitness ;Store state transfer quad to the experience pool; use batch samples to update parameters in Q network training to minimize TD error; When the stopping condition is met, the current optimal individual is output as the final scheduling solution, including the machine allocation and processing sequence of all workpieces: in, Represents the total completion time of the scheduling solution.

Images