Distributed heterogeneous flexible job shop energy-saving scheduling method

Abstract

The embodiment of the invention provides an energy-saving scheduling method for a distributed heterogeneous flexible job shop, and the method comprises the steps: obtaining the shop information and at least one piece of workpiece information of the distributed heterogeneous flexible job shop, and taking the minimization of the maximum completion time and the minimization of the total energy consumption as a scheduling target, establishing a multi-objective optimization model according to the workshop information and the information of the at least one workpiece; and a population is established and updated based on a non-dominated sorting genetic algorithm, and the population is evolved by using a D3QN deep reinforcement learning network in the population updating process, so that the multi-target optimization model is solved, and a scheduling scheme capable of realizing the scheduling target is obtained.

Description

Technical Field

[0001] The present invention relates to the field of manufacturing system scheduling, and in particular to an energy-saving scheduling method for a distributed heterogeneous flexible job shop. Background Art

[0002] With globalization and the increasing diversification of market demands, the manufacturing industry is facing increasing challenges. Especially in terms of production scheduling and energy efficiency, these challenges have driven the development of advanced manufacturing technologies and systems. In this context, the distributed flexible job shop, as an emerging production system model, shows great potential in improving production flexibility and efficiency. This workshop model is becoming more and more common in the manufacturing industry, especially in companies pursuing efficient energy management and sustainable production. Traditional studies assume that the machine types and processing capabilities are the same in each factory.

[0003] In the process of implementing the present invention, the applicant discovered that there are at least the following problems in the prior art:

[0004] Traditional methods are unable to achieve efficient energy management and sustainable production for distributed heterogeneous flexible job shops. Summary of the invention

[0005] The embodiment of the present invention provides a distributed heterogeneous flexible job shop energy-saving scheduling method to solve the problem that traditional methods cannot perform efficient energy management and sustainable production for distributed heterogeneous flexible job shops.

[0006] To achieve the above-mentioned purpose, an embodiment of the present invention provides a distributed heterogeneous flexible job shop energy-saving scheduling method, comprising:

[0007] Acquire shop information and at least one workpiece information of a distributed heterogeneous flexible job shop, take minimizing maximum completion time and minimizing total energy consumption as scheduling objectives, and establish a multi-objective optimization model according to the shop information and at least one workpiece information;

[0008] A population is established and updated based on a non-dominated sorting genetic algorithm, and in the process of updating the population, a D3QN deep reinforcement learning network is used to evolve the population, so as to solve the multi-objective optimization model and obtain a scheduling scheme that can achieve the scheduling goal;

[0009] Among them, the population includes multiple solutions, each solution corresponds to a scheduling plan, and each solution includes process selection information, machine selection information and workshop selection information; the process selection information records the processing sequence of all processes of all workpieces; the machine selection information records the machine corresponding to each process; and the workshop selection information records the workshop selected for each workpiece.

[0010] The above technical solution has the following beneficial effects: by establishing and updating a population based on a non-dominated sorting genetic algorithm, and in the process of updating the population, the D3QN deep reinforcement learning network is used to evolve the population, and the non-dominated sorting genetic algorithm (NSGA-II algorithm) is used to initialize and evolve the population to generate and update the population. In the process of updating the population, the D3QN deep reinforcement learning network is used to further evolve and update the population, so as to optimize the corresponding solution for the workshop state corresponding to each solution in the population, thereby realizing the evolution of the population; thereby, on the basis of maintaining the global search advantage of the non-dominated sorting genetic algorithm (NSGA-II), the search strategy can be dynamically adjusted through the deep reinforcement learning ability of the D3QN deep reinforcement learning network to improve the search efficiency and the quality of the solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 It is a flow chart of a distributed heterogeneous flexible job shop energy-saving scheduling method according to one embodiment of the present invention;

[0013] Figure 2 It is a schematic diagram of coding of a solution of a group in one embodiment of the present invention;

[0014] Figure 3 is a schematic diagram of a POX crossover algorithm according to one embodiment of the present invention;

[0015] Figure 4 is a schematic diagram of a UX crossover algorithm according to one embodiment of the present invention;

[0016] Figure 5 It is a schematic diagram of an improved POX crossover method and a mutation strategy of gene sequence inversion according to one embodiment of the present invention;

[0017] Figure 6 is a schematic diagram of an energy-saving scheduling strategy according to an embodiment of the present invention;

[0018] Figure 7 This is a schematic diagram of the structure of a D3QN deep learning network in one embodiment of the present invention;

[0019] Figure 8 It is another flow chart of a distributed heterogeneous flexible job shop energy-saving scheduling method according to one of the embodiments of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Distributed Heterogeneous Flexible Job Shop (DHFJS) is a complex extension of the traditional Flexible Job Shop (FJS). The problem needs to be broken down into several key parts: 1) allocating workpieces to each factory; 2) determining the production sequence in each factory; 3) selecting the right machine for each process. Due to its complexity, the distributed heterogeneous flexible job shop is considered to be an NP-hard (Non-deterministic Polynomial hard problem). Meta-heuristic algorithms such as the gray wolf algorithm, particle swarm algorithm, and genetic algorithm can be applied to solve this problem. Although traditional studies assume that the machine type and processing capacity in each factory are the same, in fact, the inventors found that this assumption is often not in line with reality due to limitations such as cost, space, and worker skills. In actual distributed manufacturing scenarios, factories are usually heterogeneous, which means that the number, quality, and processing time of machines may vary.

[0022] Deep reinforcement learning combines the feature extraction ability of deep learning and the decision-making ability of reinforcement learning. It can directly make the optimal decision output based on the input multi-dimensional data. It is an end-to-end decision control system that is widely used in dynamic decision-making, real-time prediction, simulation, game theory and other fields. It continuously interacts with the environment in real time, takes environmental information as input to obtain the experience of failure or success to update the parameters of the decision network, thereby learning the optimal decision. D3QN, the full name of Dueling Double Deep Q-Network, is developed on the basis of DQN and combines the advantages of DuelingDQN and Double DQN algorithms. The core of the D3QN algorithm is to solve the problem of overestimation of Q value that may occur in the training process of DQN, while improving the performance and stability of the algorithm. Considering the large state space of DHFJS, the inventor adopts D3QN to learn and select the optimal operator. The embodiment of the present invention proposes a dual population NSGA-Ⅱ (D3QN-NSGA-Ⅱ) algorithm based on D3QN, with the goal of minimizing the maximum completion time and the minimum total energy consumption, to solve the energy efficiency scheduling optimization problem of DHFJS.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a distributed heterogeneous flexible job shop energy-saving scheduling method, including:

[0024] Step S10, obtaining the workshop information and at least one workpiece information of the distributed heterogeneous flexible job shop, taking minimizing the maximum completion time and minimizing the total energy consumption as the scheduling objectives, and establishing a multi-objective optimization model according to the workshop information and at least one workpiece information;

[0025] Step S11, establishing and updating a population based on a non-dominated sorting genetic algorithm, and evolving the population using a D3QN deep reinforcement learning network in the process of updating the population, thereby solving the multi-objective optimization model and obtaining a scheduling solution that can achieve the scheduling goal;

[0026] Among them, the population includes multiple solutions, each solution corresponds to a scheduling plan, and each solution includes process selection information, machine selection information and workshop selection information; the process selection information records the processing sequence of all processes of all workpieces; the machine selection information records the machine corresponding to each process; and the workshop selection information records the workshop selected for each workpiece.

[0027] In some embodiments, a distributed heterogeneous flexible job shop (DHFJS) is an advanced production system consisting of multiple geographically dispersed factories, each with different types and quantities of machines. This workshop model allows workpieces to be dynamically allocated between different factories based on process requirements and resource availability to optimize production efficiency and energy consumption. It is characterized by the ability to handle a variety of different production tasks while maintaining a high degree of flexibility and production efficiency, and is particularly suitable for manufacturing environments with changing demands and customized production. The workshop information includes: the type and quantity of machines in each workshop, the processing capacity of each workshop, the load energy consumption and idle energy consumption of the workshop machines; the workpiece information includes: at least one process required to process the workpiece, the time required for processing on optional machines, and the optional machines for each process; the maximum completion time is the time consumed to complete all processes of all workpieces; the total energy consumption is the sum of the machine load energy consumption of all workpieces completing all processes and the energy consumption when the machine is idle; after the population is evolved using the non-dominated sorting genetic algorithm, the local search operator for each solution in the population is determined according to the workshop state corresponding to the solution using the D3QN deep reinforcement learning network. The local search operator is used to define the strategy for evolving the solution, and the solution is evolved according to the local search operator of the solution, thereby evolving the population evolved by the non-dominated sorting genetic algorithm again.

[0028] The embodiments of the present invention have the following technical effects: a population is established and updated based on a non-dominated sorting genetic algorithm, and in the process of updating the population, a D3QN deep reinforcement learning network is used to evolve the population, and a non-dominated sorting genetic algorithm (NSGA-II algorithm) is used to initialize and evolve the population to generate and update the population. In the process of updating the population, the D3QN deep reinforcement learning network is used to further evolve and update the population, so as to optimize the corresponding solution for the workshop state corresponding to each solution in the population, thereby realizing the evolution of the population; thereby, on the basis of maintaining the global search advantage of the non-dominated sorting genetic algorithm (NSGA-II), the deep reinforcement learning ability of the D3QN deep reinforcement learning network can also be used to dynamically adjust the search strategy to improve the search efficiency and the quality of the solution.

[0029] Furthermore, the non-dominated sorting genetic algorithm is used to establish and update the population, and in the process of updating the population, the D3QN deep reinforcement learning network is used to evolve the population, so as to solve the multi-objective optimization model and obtain a scheduling scheme that can achieve the scheduling goal, including:

[0030] Initialize the current number of iterations;

[0031] Initialize a population according to the workshop information, the at least one workpiece information and a preset encoding method, and use the initialized population as the current parent population; the preset encoding method satisfies a preset scheduling constraint condition;

[0032] Calculate the fitness value of each solution in the current parent population;

[0033] Calculate the non-dominated sorting and crowding degree for the current parent population, obtain the multi-level non-dominated layers corresponding to the current parent population and the crowding degree of each level of non-dominated layers, and take at least one solution in the first level of non-dominated layers corresponding to the current parent population as the current optimal solution to be determined; wherein each level of non-dominated layers includes at least one solution;

[0034] Use the reproductive capacity determination method to split the current parent population into reproductive sub-populations and common sub-populations;

[0035] Different evolutionary strategies are used to perform evolutionary updates on the reproductive sub-population and the common sub-population, respectively, to obtain the reproductive sub-population and the common sub-population after evolutionary updates;

[0036] Merge the evolutionarily updated breeding sub-population and the common sub-population into a new population;

[0037] Determine whether the current number of iterations is greater than the preset maximum number of iterations;

[0038] If it is determined that the current number of iterations is less than or equal to the preset maximum number of iterations, the current number of iterations is incremented by 1, and for each solution in the new population, the D3QN deep reinforcement learning network is used to select the local search operator corresponding to the solution in the preset action space, and the solution is evolutionarily updated according to the local search operator corresponding to the solution to obtain a new population after D3QN deep reinforcement learning, and the new population after D3QN deep reinforcement learning is used as the current parent population, and the process of calculating the non-dominated sorting and crowding degree for the current parent population is returned to obtain the multi-level non-dominated layers corresponding to the current parent population and the crowding degree of each level of the non-dominated layers, and the solution in the first level non-dominated layer corresponding to the current parent population is used as the current undetermined optimal solution.

[0039] If it is determined that the current number of iterations is greater than the preset maximum number of iterations, the current undetermined optimal solution is used as a scheduling solution that can achieve the scheduling target.

[0040] In some embodiments, the number of iterations is recorded by the current number of iterations. The initial value of the current number of iterations can be 0 or 1. Each time an iteration is completed, the current number of iterations will be incremented by 1. The current number of iterations is combined with the preset maximum number of iterations to limit the total number of iterations. The solutions in the population include process selection information, machine selection information, and workshop selection information; Figure 2 As shown, the process selection information, machine selection information and shop selection information are all organized in the form of arrays; Figure 2 The encoding method of a solution is shown, where OS represents the process selection (information), MS represents the machine selection (information) for each process, and FS represents the factory sequence (shop selection information) to which the workpiece is assigned. The length of OS and MS is the total number of processes, and the length of FS is the total number of workpieces. When decoding, all workpieces are assigned to each heterogeneous factory according to the FS vector. Then, according to the MS vector, each process selects an optional machine, and obtains O i,j Finally, the start time and completion time of each process are obtained, thereby obtaining the completion time and total energy consumption of the entire workshop. Specifically, when encoding, the process selection information (i.e. Figure 2 The OS segment in the , the workpiece number is directly encoded with the workpiece number, and the order in which the workpiece number appears indicates the order of processing between the workpiece processes, that is, it is compiled from left to right. For the jth appearance of workpiece number i, it indicates the jth process of workpiece i, and the total number of appearances of the workpiece number is equal to the total number of processes of the workpiece. Machine selection information (i.e. Figure 2 The MS segment in the table is arranged in sequence according to the workpiece and the workpiece process. Each integer represents the order of the processing machine selected for the current process in the optional machine set. Figure 2 The FS segment in the figure indicates the processing workshop to which each workpiece is assigned. Figure 2The first position in the OS segment is 2, which represents workpiece 2. The order in which this 2 appears in the OS segment among all the 2s in the OS segment represents the process number of workpiece 2, that is, the first process of workpiece 2, that is, O. 21 The second position of the OS segment is 1, which represents workpiece 1. The order in which this 1 appears in the OS segment among all the 1s in the OS segment represents the process number of workpiece 1, that is, it represents the first process of workpiece 1, i.e., c. The third position of the OS segment is 2, which represents workpiece 2. The order in which this 2 appears in the OS segment among all the 2s in the OS segment represents the process number of workpiece 2, that is, it represents the second process of workpiece 2, i.e., o. 22 , the numbers in other positions of the OS segment are explained as follows. Each process has at least one machine that can be selected. The machine selection information (MS segment) records the machine number selected for each process of all workpieces. The number stored in each position in the MS segment is the machine number selected for the process corresponding to the position. The process of each workpiece is arranged continuously according to the workpiece number in the machine selection information, and the process of each workpiece is also arranged in the order of the process number. For example, Figure 2 There are three workpieces in the system, and workpiece 1 has two processes O 11 and O 12 , workpiece 2 has three processes O 21 , O 22 and O 23 , workpiece 3 has two processes O 31 and O 32 ; The first two positions in the MS segment correspond to the two processes of workpiece 1 in the process sequence, the process immediately following the process of workpiece 1 corresponds to the three processes of workpiece 2 in the process sequence, and the process immediately following the process of workpiece 2 corresponds to the two processes of workpiece 3 in the process sequence. It can be seen from the MS segment that the first process O of workpiece 1 11 The second process O of the workpiece 1 is processed on the machine 1 among the multiple machines that can be selected for this process. 12 Processed on machine 2 among multiple machines available for this process; each workpiece can only be produced in one workshop, and the position in the FS segment corresponding to the same workpiece in the MS segment defines the workshop where the machine in MS is located, for example Figure 2It can be seen from the FS and MS segments that workpiece 1 (two processes) is processed in workshop 2, the three processes of workpiece 2 are processed in workshop 1, and the two processes of workpiece 3 are processed in workshop 2. When decoding, first, extract the set of workpieces assigned to each workshop from the FS segment. Next, traverse the vector segment sequentially, from left to right, to determine the processing order and assign each process to the corresponding workshop. Then, traverse the MS segment sequentially, also from left to right, to determine the machine selection for each process and schedule the process on the machine in the corresponding factory. Next, calculate the start and end times of all processes in each factory, and calculate the waiting time of each processing position to ensure that all start processing times meet the preset scheduling constraints of the optimization model. Generate the initial current parent population in a random manner according to the above encoding rules. In each iteration, the non-dominated sorting and crowding degree of the current parent population are calculated. By calculating the crowding distance, individuals located in sparse areas can be preferentially retained during the selection process, thereby maintaining the diversity of the population, and at least one solution in the first-level non-dominated layer corresponding to the current parent population in the current iteration is saved as the current pending optimal solution. When the maximum number of iterations is reached, the current pending optimal solution can be output as the optimal scheduling solution. The current parent population is split into a breeding sub-population and an ordinary sub-population by the reproductive capacity determination method, wherein the offspring solution bred by the breeding sub-population has better results on the target model than the ordinary sub-population. The embodiment of the present invention splits the parent population into two sub-populations, which are evolved separately and then merged, instead of using the parent population to generate a sub-population and then merging the parent population with the sub-population. By splitting into two sub-populations and evolving them separately, the global advantage of the population can be maintained, while enhancing the convergence of the algorithm. After the non-dominated sorting genetic algorithm completes the population evolution, it further evolves each solution in the population through the D3QN deep reinforcement learning network, dynamically selects the optimal local search operator for each solution, and improves the search efficiency of the algorithm. The idea of ​​the fast non-dominated sorting algorithm is to compare the dominance and non-domination relationship between other individuals and the current individual for each individual. If no individual is better than the current individual, the current individual is marked as a non-dominated individual. Traverse the population until all non-dominated individuals are found. In this way, the first-level non-dominated layer is obtained, and then these marked non-dominated individuals are ignored (that is, these individuals are no longer compared in the next round). Repeat the above steps to obtain the second-level non-dominated layer. And so on, until the entire population is stratified. Crowding distance is a key concept in the NSGA-II algorithm, which is used to measure the sparseness of individuals in the non-dominated frontier. By calculating the crowding distance, individuals in sparse areas can be preferentially retained during the selection process, thereby maintaining the diversity of the population.The steps of the algorithm are to initialize the population first and set its crowding distance to 0; then sort by target, for each target function, sort the individuals in the frontier according to the target function value; then calculate the crowding distance, and treat the border individuals specially. For each target function, the crowding distance of the minimum and maximum individuals is set to a large value (this experiment assigns it to 10^9) to ensure that they are not ignored in the selection process (because the border individuals need to be protected). For the intermediate individuals, calculate the crowding distance increment on the target; finally, add the crowding distance increment on all target functions to the crowding distance of the individual to get the crowding distance of the individual. After performing non-dominated sorting and calculating the crowding distance, the quality of the individuals can be compared. First, the smaller the number of layers obtained by non-dominated sorting, the better the individual; for individuals in the same layer, the larger the crowding distance, the better the individual. This ensures that the Pareto frontier is evenly distributed, making the population more diverse.

[0041] Furthermore, the method of using the reproductive capacity determination method to split the current parent population into reproductive sub-populations and common sub-populations includes:

[0042] Randomly generate a test population;

[0043] For each solution in the current parent population, use the solution and all test solutions in the test population to perform crossover operations to generate multiple offspring solutions;

[0044] Performing non-dominated sorting on the solution and the multiple offspring solutions, and determining the number of offspring solutions among the multiple offspring solutions whose non-dominated sorting is better than that of the solution according to the sorting result, and using the obtained number as the reproductive capacity of the solution;

[0045] Calculate the average reproductive capacity of all solutions in the current parent population, take the solutions in the current parent population whose reproductive capacity is greater than the average reproductive capacity as the pending reproductive sub-population, and take the solutions in the current parent population whose reproductive capacity is less than or equal to the average reproductive capacity as the pending common sub-population;

[0046] Determine whether the number of solutions in the pending breeding subpopulation is greater than or equal to N*β;

[0047] If it is determined that the number of solutions in the pending breeding subpopulation is greater than or equal to N*β, the N*β solutions with the largest reproductive capacity in the pending breeding subpopulation are used as the breeding subpopulation, and the pending common subpopulation is used as the common subpopulation;

[0048] If it is determined that the number of solutions in the pending breeding subpopulation is less than N*β, then a first preset number of solutions with the largest reproductive capacity are selected from the pending common subpopulation and added to the pending breeding subpopulation, and the obtained pending breeding subpopulation is used as the breeding subpopulation; the pending common subpopulation with the first preset number of solutions with the largest reproductive capacity removed is used as the common subpopulation;

[0049] Among them, the first preset number is equal to N*β minus the number of solutions in the pending breeding sub-population; N is the number of solutions of the current parent population; β is a positive number less than 1, which is a preset population ratio parameter.

[0050] In some embodiments, when performing multi-objective optimization, traditional evolutionary algorithms often rely on a single evolutionary path, which may limit their exploration capabilities and potential for accelerated convergence, while increasing the uncertainty of the results. In order to overcome these limitations, a dual-population evolutionary strategy is introduced to enhance the directional search capability and environmental adaptability of the NSGA-Ⅱ algorithm, thereby broadening the search range and reducing the risk of falling into local optimality. During the iteration of the algorithm, the main population is divided into two sub-populations according to specific population ratio parameters and reproductive capacity determination mechanisms, and differentiated genetic strategies are implemented for the two sub-populations. For multi-objective optimization tasks, the pseudo code of the specific process of the reproductive capacity determination mechanism in the population division process is shown in Table 1.

[0051]

[0052] Table 1 Pseudo code of reproductive capacity determination method

[0053] The reproductive capacity determination method first randomly generates a test population, and then for each individual (solution) in the current parent population, a crossover operation is performed with the individual (solution) in the test population to generate offspring individuals (offspring solutions). If the offspring individual is better than the parent individual (solution) of the offspring individual in the current parent population in the non-dominated sorting, the reproductive capacity of the parent individual is increased by 1. According to the comparison between the reproductive capacity of the parent individuals and the average reproductive capacity, the parent individuals with higher reproductive capacity than the average can be divided into reproductive sub-populations, and the rest are ordinary sub-populations; in order to ensure the number of reproductive sub-populations, N×β individuals with stronger reproductive capacity among the reproductive individuals are selected to form the reproductive sub-population. If the number is insufficient, individuals with stronger reproductive capacity among the ordinary individuals are selected to supplement; the remaining individuals form the ordinary sub-population.

[0054] Furthermore, the breeding sub-population and the common sub-population are respectively subjected to evolutionary updating using different evolutionary strategies to obtain the breeding sub-population and the common sub-population after evolutionary updating, including:

[0055] The solutions in the breeding sub-population are randomly paired into groups of two solutions, the fitness values ​​of the two solutions in each group are compared, and the solution with the best fitness value in each group is used to form a competitive breeding sub-population;

[0056] The process selection information in the solution of the competitive breeding sub-population is evolved by ordinary POX crossover and random two-point exchange mutation, and the machine selection information in the solution of the competitive breeding sub-population is evolved by UX crossover and random replacement machine mutation, to obtain the competitive breeding sub-population after mutation evolution;

[0057] For each solution in the competitive breeding sub-population after mutation and evolution, while keeping the process sequence corresponding to each workpiece in the solution unchanged, rearrange the production sequence between the processes of different workpieces, and continuously arrange the operation time of the machines to obtain the breeding sub-population after evolution and update;

[0058] The solutions in the common sub-population are randomly paired into groups of two solutions, the fitness values ​​of the two solutions in each group are compared, and the solution with the best fitness value in each group is used to form a competitive common sub-population;

[0059] For the process selection information in the solutions of the competitive ordinary sub-population, the improved POX crossover method and the mutation strategy evolution caused by the gene sequence are adopted. For the machine selection information in the solutions of the competitive ordinary sub-population, the uniform crossover and the machine mutation strategy of randomly selecting the machine with the shorter processing time are adopted to obtain the competitive ordinary sub-population after mutation evolution.

[0060] For each solution in the competitive ordinary sub-population after mutation and evolution, while keeping the process sequence corresponding to each workpiece in the solution unchanged, the production sequence between the processes of different workpieces is rearranged, and the operation time of the machines is continuously arranged to obtain the ordinary sub-population after evolution and update.

[0061] In some embodiments, individuals in the breeding subpopulation (population 1) have strong reproductive capacity, so the process sorting part adopts ordinary POX crossover and random two-point exchange mutation, the machine selection adopts UX crossover and random replacement machine mutation, and the factory allocation part adopts ordinary UX crossover, so as to maintain the global advantage of the population. For the ordinary subpopulation (population 2), the process sorting part adopts the improved POX crossover method and the mutation strategy of gene sequence inversion; in machine selection, the mutation strategy of uniform crossover and random selection of machines with shorter processing time is adopted to enhance the convergence of the algorithm; the factory allocation part adopts ordinary UX crossover with population 1. By evolving the breeding subpopulation and the ordinary subpopulation separately through the above method, the limitations of the traditional method in generating new solutions (individuals) are overcome. The traditional method may cause the algorithm to be inefficient in local search and it is difficult to explore new and possibly better solution space areas. The traditional algorithm will fall into the local optimal solution and cannot effectively jump out and explore the global solution space. In the embodiment of the present invention, the binary competition method is to select two individuals from the population (each individual has the same probability of being selected), and according to the fitness value of each individual, select the individual with the best fitness value to enter the next generation population. Figure 3 This is a POX crossover example, with 5 artifacts and 12 processes. Let the parent process chromosomes be P1 and P2, and the child chromosomes generated by the crossover be C1 and C2. First, randomly divide all artifact sets into two non-empty sets A and B, where A = {J1, J3}, B = {J2, J4, J5}. Copy artifacts with the same sequence number in P1 and A to C1, and copy artifacts with the same sequence number in P2 and B to C2, and ensure that their positions do not change before and after copying. Copy artifacts with the same sequence number in P2 and B to C1, and artifacts with the same sequence number in P1 and A to C2, and ensure that their order does not change before and after copying. Figure 4 This is an example of UX crossover. Let the parent machine chromosomes be M1 and M2, and the child chromosomes generated after crossover are C3 and C4. First, randomly generate an array C with the same length as the parent machine chromosome, and randomly fill C with numbers 0 and 1. Traverse the C array, record the positions where the number 1 appears in the C array, copy the workpiece at the same position where M1 and C appear with number 1 to C4, and copy the rest of the positions with the same position of M2. Copy the workpiece at the same position where M2 and C appear with number 1 to C3, and copy the rest of the positions with the same position of M1. In the scheduling problem considering energy consumption, total energy consumption is a key indicator and has a decisive influence on the realization of green scheduling. In the production process of workpieces, reducing the machine's operating time and idle time is a key method to achieve energy reduction. Based on this, an energy-saving strategy is designed to effectively shorten the machine's idle time by locally optimizing and adjusting the production process, thereby reducing energy consumption in the entire production process. As Figure 5As shown, the common subpopulation (pop2) uses the improved POX crossover method and gene sequence inversion mutation strategy for process selection. Figure 5 This is an example of a mutation strategy using modified POX crossover and gene sequence inversion. The basic information of modified POX crossover is as follows: Figure 3 For example, in the ordinary POX crossover, a circular right shift operation is added, and two crossover points are randomly selected in the parent chromosome. Starting from one of the crossover points, a section of the process sequence is circularly shifted to the right until the other crossover point is reached. This means that the moved process sequence segment will be copied and inserted into another position of the chromosome. In this way, two new daughter chromosomes are generated. The process sequence in the daughter chromosome is a rearrangement of the process sequence in the parent chromosome; gene sequence inversion is a mutation operation that generates a new chromosome by randomly selecting a section of the gene sequence in the chromosome and inverting it (i.e. reversing the order). . As Figure 6 As shown in the figure, an energy-saving scheduling strategy is presented, which can significantly reduce the idle period of the machine by rearranging the production order of different workpieces while keeping the process sequence unchanged. Without changing the critical path, it helps to reduce the overall energy consumption and improve production efficiency.

[0062] Furthermore, for each solution in the new population, the D3QN deep reinforcement learning network is used to select a local search operator corresponding to the solution in a preset action space, and the solution is evolutionarily updated according to the local search operator corresponding to the solution to obtain a solution after D3QN deep reinforcement learning, and all solutions after D3QN deep reinforcement learning are combined to obtain a new population after D3QN deep reinforcement learning, and the new population after D3QN deep reinforcement learning is used as the current parent population, including:

[0063] For each solution in the new population, a local search operator corresponding to the solution is selected from a preset action space according to a greedy strategy;

[0064] Evolutionarily updating the solution according to the local search operator corresponding to the solution to obtain a solution corresponding to the solution after deep reinforcement learning by D3QN;

[0065] Determine the total processing time and total energy consumption of the solution and the corresponding solution after D3QN deep reinforcement learning respectively;

[0066] Determine a reward value corresponding to the solution after D3QN deep reinforcement learning according to the total processing time and total energy consumption of the solution and the corresponding solution after D3QN deep reinforcement learning;

[0067] The solution, the solution after D3QN deep reinforcement learning corresponding to the solution, the total processing time and total energy consumption of the solution, the total processing time, total energy consumption and reward value of the solution after D3QN deep reinforcement learning corresponding to the solution are put into the preset memory library as this experience;

[0068] Randomly extract a second preset number of samples in a small batch from a preset memory bank, and calculate a target difference of the extracted samples;

[0069] The evaluation network parameters and the target network parameters in the D3QN deep reinforcement learning network are updated according to the target difference; wherein the D3QN deep reinforcement learning network includes an evaluation network and a target network, the evaluation network is used to generate Q values ​​of all possible actions in the current state, and the target network is used to stabilize the training process, provide a relatively stable target Q value, and reduce the instability in the learning process.

[0070] In some embodiments, the greedy strategy uses a deep neural network to evaluate the expected returns of all possible actions in the current state. This network outputs a value distribution, which represents the predicted future benefits of executing each action in the current state. The greedy strategy simply selects the action with the highest value in this distribution for execution. The DQN algorithm is an efficient unsupervised deep reinforcement learning algorithm that has the ability to autonomously learn the data distribution in the solution space and can construct a probability mapping related to operator selection based on these distribution characteristics. The original DQN has the problem of overestimating the Q value in some cases, which may lead to instability and performance degradation during the learning process. Therefore, an improved version of DQN, DDQN (Double DQN), was proposed. DDQN introduces two neural networks: one for selecting actions and calculating target Q values ​​(called the prediction network), and the other for evaluating target Q values ​​(called the target network). During the training process, the target Q value is generated by regularly updating the target network instead of directly using the Q value generated by the prediction network, thereby reducing the risk of overestimation.

[0071] The objective function of DDQN is:

[0072]

[0073] in, Represents the sum of the immediate reward and the expected future return obtained using the Double DQN algorithm R t+1 represents the immediate reward obtained at time step t+1; γ represents the discount factor; max a Q() represents the maximum Q value of all possible actions in a given state; S t+1 represents the state of the environment at time step t+1; argmax aQ() represents the action a that maximizes the Q value; a represents the action selected by the agent in a specific state; θ t represents the parameters of the network evaluated at time step t; represents the parameters of the target network at time step t.

[0074] The Q value Q(s,a) implies the value of being in state s and the advantage of choosing action a, so the Dueling network was proposed to clearly distinguish the state value and action advantage in a state. In the Dueling network, two independent networks are used to estimate the state value function V(s) and the action advantage function A(s,a) in the state respectively.

[0075] The action value function Q(s,a) calculation formula is:

[0076]

[0077] Among them, Q() represents the action-value function; s represents the current state; a represents the action; θ represents the parameters of the evaluation network; α represents the learning rate; V() represents the state value function; A() represents the action advantage function; |A| represents the absolute value of the action advantage function; a' represents the optimal action selected by the evaluation network under a specific state s.

[0078] D3QN (Dueling Double DQN) combines the advantages of Dueling DQN and Double DQN. Figure 7 The D3QN network consists of an input layer, three fully connected layers (in some embodiments, with 128, 64, and 32 neurons, respectively, all using ReLU activation functions), a branched fully connected layer (divided into value and advantage functions), and an output layer.

[0079] When training D3QN, random initialization is used to obtain greater diversity. Randomly sort the workpieces and process sequences to generate the sort code OS segment; in the process code, select the processable machines from the optional machine set according to the process to generate the machine sequence MS segment; finally, according to the process code, randomly assign to the processable factory to generate the factory sequence FS segment. State design: In each iteration, the local search operator corresponding to each solution is recommended based on the current workshop state expressed by the solution. Therefore, it is very important to accurately extract and represent the current workshop state. Considering that the constrained objectives of the embodiment of the present invention are related to both time and energy consumption, three indicators related to the overall attributes of the scheduling problem (the number of workpieces, the number of machines, and the total number of processes), four indicators related to the workshop processing time (total machine completion time, machine idle / processing time, total completion time of each factory, machine idle / processing time of each factory), and two indicators related to energy consumption attributes (total machine energy consumption, energy consumption of each factory) are proposed to accurately characterize the current workshop state. The action space includes multiple local search operators. Reward design: The designed reward is based on the current state (i.e., the maximum completion time C corresponding to the action to be executed or the local search operator corresponding to a solution learned in this iteration). max and total energy consumption TEC), and the previous state (i.e., the maximum completion time C' corresponding to the solution before executing the pending action or local search operator corresponding to a solution learned in this iteration) max and total energy consumption TEC'), the reward mechanism in some embodiments is if C max Less than C' max , and TEC is less than TEC', then the reward value is set to the first reward value; if C max Less than C' max and TEC is greater than TEC', or, C max Greater than C' max And if TEC is less than TEC', the reward value is set to the second reward value. If neither of the above two judgments is satisfied, the reward value is set to the third reward value, wherein the third reward value is less than the first reward value, and the first reward value is less than the second reward value; preferably, the third reward value is 0, the second reward value is 10, and the first reward value is 5; the specific process is shown in Table 2.

[0080]

[0081] Table 2 Reward mechanism

[0082] The process of combining NSGA-II with deep reinforcement learning is mainly reflected in the use of the NSGA-II algorithm to initialize and evolve the population to generate and update the population of scheduling solutions. At the same time, the local search operator is learned and selected through the D3QN network. The D3QN network is trained based on multiple indicators of the current workshop state, the selected operator action, and the reward value based on the degree of state improvement to learn to select the most appropriate operator in a specific state to optimize the scheduling solution. This combination allows the algorithm to not only maintain the global search advantage of NSGA-II, but also dynamically adjust the search strategy through the reinforcement learning ability of D3QN to improve the search efficiency and the quality of the solution.

[0083] Furthermore, the action space includes three types of local search operators: classical operator class, destruction and reconstruction operator class, and critical path operator class;

[0084] The classical operator class includes: randomly selecting two processes and exchanging their positions, randomly selecting a process and inserting it into a position, randomly selecting two processes and inserting one process before the other process, and / or randomly selecting two processes and inserting one process after the other process;

[0085] The destruction and reconstruction operator class includes: randomly generating a 0-1 array, retaining values ​​of 1 and randomly assigning values ​​of 0 to the remaining positions; randomly selecting a point, keeping the right side of the point unchanged and randomly shuffling the left side; randomly selecting a point, keeping the left side of the point unchanged and randomly shuffling the right side; and / or randomly selecting two points, shuffling the process between the two points; the critical path operator class includes: randomly selecting a process on the critical path and randomly moving all processes of the workpiece to which it belongs to another factory; randomly selecting a process on the critical path, selecting a processing factory through a roulette wheel selection system, and moving all processes of the workpiece to which the process belongs to the selected factory; randomly selecting a process on the critical path, randomly reselecting a machine tool for processing within the range of its optional machines, and / or randomly selecting a process on the critical path, selecting a processing machine through a roulette wheel selection system, and moving the current process to the selected machine for processing.

[0086] In some embodiments, the embodiments of the present invention include 12 local search operators of three types to accelerate the convergence of the algorithm; in the search process, the embodiments of the present invention utilize operators related to the critical path to enable the algorithm to obtain more significant solution quality improvements in each iteration; when the algorithm encounters a local optimal solution, the destructive reconstruction operator is used to significantly increase the probability of the algorithm jumping out of the local optimal solution. The three types of local search operators with different attributes in the embodiments of the present invention are the classic operator class, the destruction and reconstruction operator class, and the critical path operator class. The following are the functions and effects of each type of operator: The role of the classic operator is to explore the impact of different process sequences on the scheduling results. In the destruction and reconstruction operator class, a 0-1 array is randomly generated and the processes are redistributed to break the current local optimal structure and increase the diversity of the population; the processes to the left or right of a point are randomly disrupted to explore new process arrangements and combinations. In the critical path operator class, operations are performed on the processes on the critical path because the critical path directly affects the total completion time of the job, so a targeted search can be conducted on the total completion time; the processes on the critical path are randomly moved to another factory or machine to reduce machine idle time and energy consumption. The specific local search operators are shown in Table 3:

[0087]

[0088]

[0089] Table 3 Local search operator

[0090] Further, the state space includes: indicators related to the overall attributes of the scheduling problem, indicators related to the workshop processing time, and indicators related to the energy consumption attributes;

[0091] The indicators related to the overall properties of the scheduling problem include: the number of workpieces, the number of machines, and / or the total number of processes;

[0092] The indicators related to workshop processing time include: total machine completion time, machine idle / processing time, total completion time of each factory, and / or machine idle and processing time of each factory;

[0093] The indicators related to energy consumption attributes include: total energy consumption of machines and / or energy consumption of each factory.

[0094] Furthermore, the multi-objective optimization model is specifically the following formula:

[0095] MIN{C max ,TEC} (3)

[0096] Among them, MIN means minimization, C max represents the maximum completion time, TEC represents the total energy consumption;

[0097]

[0098] Among them, MAX represents the time consumed by the workpiece that takes the longest time to complete all processes among all workpieces; The nth i The processing completion time of a process in factory f;

[0099] TEC=E w +E I (5)

[0100]

[0101] Among them, E w Indicates the energy consumption of machine processing; E I represents the idle energy consumption of the machine; i represents the workpiece index; I represents the workpiece set; j represents the process index, J i represents the process set of workpiece i; f represents the factory index; F represents the factory set; k represents the machine index; M represents the machine set; t represents the processing position index number on the machine; P k represents the set of processing locations of machine k in factory f; W p Indicates the machine operating power; T i,j,f,k represents the processing time of the jth process of the i-th workpiece on the machine k in factory f; X i,j,f,k,t The value is 0 or 1, indicating whether the jth process of the i-th workpiece is processed on the tth position of the machine k in the factory f. If it is processed, it is 1, otherwise it is 0; P' k represents the front P of machine k in factory f k-1 A set of processing positions; W s Indicates the idle power of the machine; B f,k,t+1 represents the start time of processing at position t+1 of machine k in factory f; B f,k,t It represents the start time of processing at the tth position of machine k in factory f.

[0102] Furthermore, the preset scheduling constraints include:

[0103] The same workpiece can only be assigned to one factory, and all processes of the workpiece are completed in the assigned factory;

[0104] The same process can be processed on the same machine, but the processing time on the machines in different factories is different;

[0105] The same process of the same workpiece can only be processed by one machine at the same time;

[0106] Once the processing of each step of each workpiece begins, the processing cannot be interrupted;

[0107] Different artifacts have the same priority;

[0108] There is no order constraint between the processes of different workpieces, but there is order constraint between the processes of the same workpiece;

[0109] All workpieces and machines are ready for processing at time zero.

[0110] In some embodiments, each workpiece can be assigned to only one factory for processing, that is, the following formula is satisfied:

[0111]

[0112] Once a workpiece is assigned to a factory, all its processes are completed in that factory, which satisfies the following formula:

[0113]

[0114] The relationship between the maximum completion time of each factory and the total maximum completion time satisfies the following formula:

[0115]

[0116] The start time of each process must be after the previous process is completed, that is, it satisfies the following formula:

[0117]

[0118] Each machine can only process one process at the same time, that is, it satisfies the following formula:

[0119]

[0120] The relationship between the start processing time and the completion time of each machine satisfies the following formula:

[0121]

[0122] Among them, Y i,f The value is 0 or 1, indicating whether job i is assigned to factory f for processing. If yes, the value is 1, otherwise it is 0; S f,i,ni The nth i The start time of the process in factory f; S f,i,j represents the start time of the jth process of job i in factory f; S f,i,j+1 represents the start time of the j+1th process of workpiece i in factory f; i represents the workpiece index; I represents the workpiece set; j represents the process index, J i represents the process set of workpiece i; f represents the factory index; F represents the factory set; k represents the machine index; M represents the machine set; t represents the processing position index number on the machine; P' f,krepresents the front P of machine k in factory f k-1 A set of processing positions; T i,j,f,k represents the processing time of the jth process of the i-th workpiece on the machine k in factory f; X i,j,f,k,t The value is 0 or 1, indicating whether the jth process of the i-th workpiece is processed on the tth position of the machine k in the factory f. If it is processed, it is 1, otherwise it is 0; B f,k,t+1 represents the start time of processing at position t+1 of machine k in factory f; B f,k,t It represents the start time of processing at the tth position of machine k in factory f.

[0123] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with specific application examples. For technical details not introduced during the implementation process, please refer to the relevant description in the previous text.

[0124] The embodiment of the present invention discloses a D3QN-NSGA-II algorithm for energy-saving scheduling problems in distributed heterogeneous flexible job shops, which belongs to the field of manufacturing system scheduling and deep reinforcement learning technology. The algorithm combines the D3QN network on the basis of NSGA-II, splits the population into two sub-populations through the reproductive capacity determination method, and designs different crossover mutation operators for each sub-population. In addition, the algorithm designs 12 local search operators of three types and an energy-saving strategy, which effectively shortens the idle time of the machine and reduces the energy consumption in the entire production process by optimizing and adjusting the production process locally. The construction of the D3QN network includes state design, action design and reward design to dynamically select the optimal operator and improve the search efficiency of the algorithm. The global search capability and local search efficiency of the algorithm are improved through the dual population strategy and the D3QN operator selection mechanism. The designed energy-saving strategy effectively reduces the energy consumption in the production process while ensuring production efficiency. This provides an efficient solution to the energy-saving scheduling problem of distributed heterogeneous flexible job shops, which has important theoretical and practical significance.

[0125] In view of the above problems, the purpose of the present invention is to provide a D3QN-NSGA-Ⅱ algorithm for distributed heterogeneous flexible job shop energy-saving scheduling problem, comprising the following steps:

[0126] S1: Establish a multi-objective optimization model that combines energy-saving scheduling objectives and production efficiency of the job shop;

[0127] S2: Initialize the population and split it into two sub-populations using the gender determination method. Each sub-population uses a different crossover mutation operator.

[0128] S3: Design 12 local search operators of three types to accelerate the convergence of the algorithm;

[0129] S4: Use D3QN to dynamically select the optimal operator for each solution to improve the search efficiency of the algorithm.

[0130] The construction of the D3QN network includes the following components:

[0131] State design: extract multiple indicators related to scheduling problems to accurately represent the current workshop state;

[0132] Action design: Select one of 12 local search operators to optimize the current solution;

[0133] Reward design: Determine the reward value based on the degree of improvement between the current state and the previous state.

[0134] The dual-population evolution strategy includes the following steps: dividing the main population into two sub-populations according to the population ratio parameter and the gender determination mechanism, and implementing differentiated genetic strategies for the two sub-populations to enhance the global and local search capabilities of the algorithm.

[0135] The energy-saving strategy effectively shortens the idle time of machines and reduces energy consumption in the entire production process by optimizing and adjusting local production processes.

[0136] The D3QN-NSGA-Ⅱ algorithm process includes non-dominated sorting, crowding distance calculation, D3QN-based operator selection mechanism, and updating and training of the experience pool.

[0137] The embodiments of the present invention have the following technical effects: through the dual population strategy and D3QN operator selection mechanism, the global search capability and local search efficiency of the algorithm are improved. The designed energy-saving strategy effectively reduces the energy consumption in the production process while ensuring production efficiency. The effectiveness and superiority of the algorithm are verified through actual cases, especially in large-scale problems.

[0138] The following is a detailed description of an embodiment of the present invention. The embodiment of the present invention provides a D3QN-NSGA-II algorithm for solving the distributed heterogeneous flexible job shop energy-saving scheduling problem, including the following steps:

[0139] S1: Establish a multi-objective optimization model that combines energy-saving scheduling objectives and production efficiency of the job shop;

[0140] Distributed Heterogeneous Flexible Job Shop (DHFJS) is an advanced production system consisting of multiple geographically dispersed factories, each with different types and quantities of machines. This workshop model allows workpieces to be dynamically allocated between different factories based on process requirements and resource availability to optimize production efficiency and energy consumption. It is characterized by the ability to handle a variety of different production tasks while maintaining a high degree of flexibility and production efficiency, and is particularly suitable for manufacturing environments with changing demands and customized production.

[0141] DHFJS needs to meet the following constraints during processing:

[0142] (1) A workpiece can only be assigned to one factory, and all processes are completed in the assigned factory.

[0143] (2) The same processing machine is available for selection, but the processing time is different on machines in different factories.

[0144] (3) The same process of the same workpiece can only be processed by one machine at the same time.

[0145] (4) Once each process of each workpiece begins, the processing cannot be interrupted.

[0146] (5) Different artifacts have the same priority.

[0147] (6) There are no sequence constraints between the processes of different workpieces, but there are sequence constraints between the processes of the same workpiece.

[0148] The following table 4 gives an explanation of the parameters involved in the embodiment of the present invention:

[0149]

[0150]

[0151] Table 4 Parameter meaning

[0152] (7) All workpieces and machines can start processing at time zero.

[0153] As shown in formula (1), the objective function of DHFJS includes minimizing the maximum completion time C max And total energy consumption TEC two goals.

[0154] a. Minimize the maximum completion time C max , as shown in formula (4);

[0155] b. The formulas for total energy consumption TEC are formula (5), formula (6) and formula (7);

[0156] The constraints are stated as follows:

[0157] Ensure that each workpiece can only be assigned to one factory for processing, that is, formula (8);

[0158] Ensure that once a workpiece is assigned to a factory, all its processes are completed in that factory, that is, formula (9);

[0159] The relationship between the maximum completion time of each factory and the total maximum completion time is formula (10);

[0160] The start time of each process must be after the previous process is completed, that is, formula (11);

[0161] Each machine can only process one process at the same time, that is, formula (12);

[0162] The relationship between the start processing time and completion time of each machine is formula (13);

[0163] S2: Initialize the population and split it into two sub-populations using the gender determination method. Each sub-population uses a different crossover mutation operator.

[0164] Gender determination mechanism: The algorithm first randomly generates a test population, and then for each individual in the main population, a crossover operation is performed with an individual in the test population to produce offspring. If the offspring individual is better than the parent individual in the non-dominated sorting, the parent individual's reproductive capacity is increased by 1. According to the comparison between the individual's reproductive capacity and the average reproductive capacity, the individuals above the average are divided into the reproductive sub-population (pop1), and the rest are the ordinary sub-population (pop2).

[0165] Differentiated genetic strategy: Since individuals in the breeding subpopulation (pop1) have strong reproductive ability, ordinary POX crossover and random two-point exchange mutation methods, as well as UX crossover and random replacement machine mutation are used to maintain the global advantage of the population. The ordinary subpopulation (pop2) uses the improved POX crossover method and gene sequence inversion mutation strategy, as well as uniform crossover and random selection of machines with shorter processing time mutation strategies to enhance the convergence of the algorithm.

[0166] As the instruction manual Figure 2 As shown in Figure 1, this is a solution encoding method. OS represents the process selection, MS represents the machine selection for each process, and FS represents the factory sequence to which the workpiece is assigned. The length of OS and MS is the total number of processes, and the length of FS is the total number of workpieces. During decoding, all workpieces are assigned to each heterogeneous factory according to the FS vector. Then, according to the MS vector, each process selects an optional machine, and O is obtained. i，jFinally, the start time and completion time of each process are obtained, thereby obtaining the maximum completion time and total energy consumption of the entire workshop.

[0167] When training D3QN, random initialization is used to obtain greater diversity. The workpieces and process sequences are randomly sorted to generate a sort code OS; in the process code, the processable machines are selected from the optional machine set according to the process to generate a machine sequence MS; finally, according to the process code, they are randomly assigned to the processable factories to generate a factory sequence FS.

[0168] When performing multi-objective optimization, traditional evolutionary algorithms often rely on a single evolutionary path, which may limit their exploration capabilities and potential for accelerated convergence, while increasing the uncertainty of the results. In order to overcome these limitations, a dual-population evolutionary strategy is introduced to enhance the directional search capability and environmental adaptability of the NSGA-Ⅱ algorithm, thereby broadening the search range and reducing the risk of falling into local optimality. During the algorithm's iteration process, the main population is divided into two sub-populations based on specific population ratio parameters and gender determination mechanisms, and differentiated genetic strategies are implemented for these two sub-populations. For multi-objective optimization tasks, the specific process of the gender determination mechanism in the population division process is shown in the pseudo code in Table 1.

[0169] Select N×β individuals with strong reproductive capacity from the breeding individuals to form subpopulation 1. If the number is insufficient, select individuals with strong reproductive capacity from ordinary individuals to supplement; the remaining individuals form subpopulation 2. Since the individuals in population 1 have strong reproductive capacity, the process sorting part adopts the ordinary POX crossover and random two-point exchange mutation method, the machine selection adopts UX crossover and random replacement machine mutation, and the factory allocation part adopts ordinary UX crossover, so as to maintain the global advantage of the population. For population 2, since it is difficult for traditional genetic algorithms to generate new individuals, this limits the local search efficiency of the algorithm. Instruction manual attached Figure 5 It demonstrates that in the process sorting part, an improved POX crossover method and a mutation strategy of gene sequence inversion are used; in machine selection, a mutation strategy of uniform crossover and random selection of machines with shorter processing time is used to enhance the convergence of the algorithm.

[0170] S3: Design 12 local search operators of three types to accelerate the convergence of the algorithm;

[0171] When solving the energy-efficient scheduling problem of DHFJS, there are many efficient and widely accepted local search strategies, such as position exchange, post-insertion and pre-insertion, etc. In the search process, the use of operators related to the critical path can prompt the algorithm to obtain more significant improvement in solution quality in each iteration; when the algorithm encounters a local optimal solution, the use of destructive reconstruction operators can significantly increase the probability of the algorithm jumping out of the local optimal solution.

[0172] The embodiment of the present invention designs three types of local search operators with different properties, namely, a classic operator class, a destruction and reconstruction operator class, and a critical path operator class. The following are the functions and effects of each type of operator:

[0173] Classical operators:

[0174] (1) The positions of two processes are exchanged to explore the impact of different process sequences on the scheduling results.

[0175] (2) Insert one process into the position of another process to explore the optimization potential of the new process combination for scheduling.

[0176] (3) Insert one process before or after another process to optimize the order of the processes.

[0177] Destruction and reconstruction operators:

[0178] (1) Randomly generate a 0-1 array and redistribute the processes to break the current local optimal structure and increase the diversity of the population.

[0179] (2) Randomly disrupt the processes to the left or right of a point to explore new process arrangements and combinations.

[0180] Critical Path Operator:

[0181] (1) Operate on the processes on the critical path, because the critical path directly affects the total completion time of the job.

[0182] (2) Randomly move the processes on the critical path to another factory or machine to reduce machine idle time and reduce energy consumption.

[0183] The detailed settings are shown in Table 3:

[0184] In the scheduling problem considering energy consumption, total energy consumption is a key indicator and has a decisive influence on the realization of green scheduling. In the production process of workpieces, reducing the machine's operating time and idle time is a key method to achieve energy reduction. Based on this, an energy-saving strategy is designed to effectively shorten the machine's idle time through local optimization and adjustment of production processes, thereby reducing energy consumption in the entire production process.

[0185] Instructions attached Figure 6 The energy-saving scheduling strategy is demonstrated, which can significantly reduce the idle period of the machine by rearranging the production order of different workpieces while keeping the process sequence unchanged. It helps to reduce the overall energy consumption and improve production efficiency without changing the critical path.

[0186] S4: Use D3QN to dynamically select the optimal operator for each solution to improve the search efficiency of the algorithm.

[0187] The DQN algorithm is an efficient unsupervised deep reinforcement learning algorithm that has the ability to autonomously learn the data distribution in the solution space and construct a probability mapping related to operator selection based on these distribution characteristics. The original DQN has the problem of overestimating the Q value in some cases, which may lead to instability and performance degradation during the learning process. Therefore, an improved version of DQN, DDQN (Double DQN), was proposed. DDQN introduces two neural networks: one for selecting actions and calculating target Q values ​​(called the prediction network), and the other for evaluating the target Q value (called the target network). During the training process, the target Q value is generated by regularly updating the target network instead of directly using the Q value generated by the prediction network, thereby reducing the risk of overestimation. The objective function of DDQN is shown in formula (1).

[0188] The Q value Q(s, a) implies the value of being in state s and the advantage of choosing action a. Therefore, the Dueling network[2] was proposed to clearly distinguish the state value and action advantage in a state. In the Dueling network, two independent networks are used to estimate the state value function V(s) and the action advantage function A(s, a) in the state respectively. The action value function Q(s, a) is calculated as formula (2).

[0189] D3QN (Dueling Double DQN) combines the advantages of Dueling DQN and Double DQN. Figure 7 Its network structure is shown.

[0190] State design: The local search operator is recommended based on the current state of the workshop. Therefore, it is very important to accurately extract and represent the current state of the workshop. Considering that the constraints of this study are related to both time and energy consumption, three indicators related to the overall attributes of the scheduling problem (number of workpieces, number of machines, total number of processes), four indicators related to the workshop processing time (total completion time of the machine, machine idle / processing time, total completion time of each factory, machine idle / processing time of each factory), and two indicators related to energy consumption attributes (total energy consumption of the machine, energy consumption of each factory) are proposed to accurately represent the current state of the workshop.

[0191] Action design: The relevant action is to select one from the 12 local search operators shown in Table 1.

[0192] Reward design: The designed reward is based on the current state C max , TEC and the previous state C' max , TEC', the specific process is shown in Table 2.

[0193] The process of combining NSGA-II with deep reinforcement learning is mainly reflected in the use of the NSGA-II algorithm to initialize and evolve the population to generate and update the population of scheduling solutions. At the same time, the local search operator is learned and selected through the D3QN network. The D3QN network is trained based on multiple indicators of the current workshop state, the selected operator actions, and the reward value based on the degree of state improvement to learn to select the most appropriate operator in a specific state to optimize the scheduling solution. This combination allows the algorithm to not only maintain the global search advantage of NSGA-II, but also dynamically adjust the search strategy through the reinforcement learning ability of D3QN to improve the search efficiency and the quality of the solution.

[0194] Figure 8 A detailed flow chart of an embodiment of the present invention is given, which reads workpiece information, where multiple workpieces may exist; initializes algorithm parameters, such as the current number of iterations; initializes the population, initializes the population according to the workpiece information, and each solution in the population corresponds to a scheduling scheme to obtain the current parent population; calculates the fitness of the population, so that when the binary competition method is selected, the solution with good fitness is selected from the sub-population (population 1 and population 2) according to the fitness to continue crossover mutation; non-dominated sorting and crowding calculation are performed on the current parent population, and the current first-level non-dominated layer is saved as the current undetermined optimal solution; the reproductive capacity determination method is used to split the population to obtain population 1 (reproduction sub-population) and population 2 (ordinary sub-population), and population 1 and population 2 are respectively subjected to their respective binary competition method selection and crossover mutation to obtain offspring population 1 (competition reproduction sub-population) and offspring population 2 (competition ordinary sub-population), and then each is subjected to an energy-saving strategy to obtain an updated and evolved reproduction sub-population and ordinary sub-population. , merge the updated and evolved breeding sub-population and the common sub-population to obtain a new population; if the current number of iterations is greater than the preset maximum number of iterations G, the current undetermined optimal solution obtained in this iteration is used as the optimal scheduling solution; otherwise, execute D3QN deep reinforcement learning to further evolve the solutions in the new population and then update the current parent population. Specifically, select a local search operator from the action space according to the greedy strategy, and determine the local search operator of the solution for each solution in the new population; execute the selection action, obtain the reward and the next state, apply the local search operator of the solution to the solution, so that the solution is evolved and updated, and determine the reward based on the solution and the target model after the evolutionary update; put this experience into the memory bank, randomly extract a small batch of b samples from the memory bank, and calculate the target difference of the extracted samples; update the evaluation network parameters, and update the target network parameters every c steps, so that the D3QN deep reinforcement learning network can learn new experience and give more accurate action selection in subsequent iterations.

[0195] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0196] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0197] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0198] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including". In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0199] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0200] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0201] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0202] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0203] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed heterogeneous flexible job shop energy-saving scheduling method, characterized in that: include: Acquire shop information and at least one workpiece information of a distributed heterogeneous flexible job shop, take minimizing maximum completion time and minimizing total energy consumption as scheduling objectives, and establish a multi-objective optimization model according to the shop information and at least one workpiece information; A population is established and updated based on a non-dominated sorting genetic algorithm, and in the process of updating the population, a D3QN deep reinforcement learning network is used to evolve the population, so as to solve the multi-objective optimization model and obtain a scheduling scheme that can achieve the scheduling goal; Among them, the population includes multiple solutions, each solution corresponds to a scheduling plan, and each solution includes process selection information, machine selection information and workshop selection information; the process selection information records the processing sequence of all processes of all workpieces; the machine selection information records the machine corresponding to each process; and the workshop selection information records the workshop selected for each workpiece.

2. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 1, characterized in that: The non-dominated sorting genetic algorithm is used to establish and update the population, and in the process of updating the population, the D3QN deep reinforcement learning network is used to evolve the population, so as to solve the multi-objective optimization model and obtain a scheduling scheme that can achieve the scheduling goal, including: Initialize the current number of iterations; Initialize a population according to the workshop information, the at least one workpiece information and a preset encoding method, and use the initialized population as the current parent population; the preset encoding method satisfies a preset scheduling constraint condition; Calculate the fitness value of each solution in the current parent population; Calculate the non-dominated sorting and crowding degree for the current parent population, obtain the multi-level non-dominated layers corresponding to the current parent population and the crowding degree of each level of non-dominated layers, and take at least one solution in the first level of non-dominated layers corresponding to the current parent population as the current optimal solution to be determined; wherein each level of non-dominated layers includes at least one solution; Use the reproductive capacity determination method to split the current parent population into reproductive sub-populations and common sub-populations; Different evolutionary strategies are used to perform evolutionary updates on the reproductive sub-population and the common sub-population, respectively, to obtain the reproductive sub-population and the common sub-population after evolutionary updates; Merge the evolutionarily updated breeding sub-population and the common sub-population into a new population; Determine whether the current number of iterations is greater than the preset maximum number of iterations; If it is determined that the current number of iterations is less than or equal to the preset maximum number of iterations, the current number of iterations is incremented by 1, and for each solution in the new population, the D3QN deep reinforcement learning network is used to select the local search operator corresponding to the solution in the preset action space, and the solution is evolutionarily updated according to the local search operator corresponding to the solution to obtain a new population after D3QN deep reinforcement learning, and the new population after D3QN deep reinforcement learning is used as the current parent population, and the process of calculating the non-dominated sorting and crowding degree for the current parent population is returned to obtain the multi-level non-dominated layers corresponding to the current parent population and the crowding degree of each level of the non-dominated layers, and the solution in the first level non-dominated layer corresponding to the current parent population is used as the current undetermined optimal solution. If it is determined that the current number of iterations is greater than the preset maximum number of iterations, the current undetermined optimal solution is used as a scheduling solution that can achieve the scheduling target.

3. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 2, characterized in that: The method of using the reproductive capacity determination method to split the current parent population into a reproductive sub-population and a common sub-population includes: Randomly generate a test population; For each solution in the current parent population, use the solution and all test solutions in the test population to perform crossover operations to generate multiple offspring solutions; Performing non-dominated sorting on the solution and the multiple offspring solutions, and determining the number of offspring solutions among the multiple offspring solutions whose non-dominated sorting is better than that of the solution according to the sorting result, and using the obtained number as the reproductive capacity of the solution; Calculate the average reproductive capacity of all solutions in the current parent population, take the solutions in the current parent population whose reproductive capacity is greater than the average reproductive capacity as the pending reproductive sub-population, and take the solutions in the current parent population whose reproductive capacity is less than or equal to the average reproductive capacity as the pending common sub-population; Determine whether the number of solutions in the pending breeding subpopulation is greater than or equal to N*β; If it is determined that the number of solutions in the pending breeding subpopulation is greater than or equal to N*β, the N*β solutions with the largest reproductive capacity in the pending breeding subpopulation are used as the breeding subpopulation, and the pending common subpopulation is used as the common subpopulation; If it is determined that the number of solutions in the pending breeding subpopulation is less than N*β, then a first preset number of solutions with the largest reproductive capacity are selected from the pending common subpopulation and added to the pending breeding subpopulation, and the obtained pending breeding subpopulation is used as the breeding subpopulation; the pending common subpopulation with the first preset number of solutions with the largest reproductive capacity removed is used as the common subpopulation; Among them, the first preset number is equal to N*β minus the number of solutions in the pending breeding sub-population; N is the number of solutions of the current parent population; β is a positive number less than 1, which is a preset population ratio parameter.

4. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 2, characterized in that: The method of using different evolutionary strategies to perform evolutionary updates on the breeding sub-population and the common sub-population to obtain the breeding sub-population and the common sub-population after evolutionary updates includes: The solutions in the breeding sub-population are randomly paired into groups of two solutions, the fitness values ​​of the two solutions in each group are compared, and the solution with the best fitness value in each group is used to form a competitive breeding sub-population; The process selection information in the solution of the competitive breeding sub-population is evolved by ordinary POX crossover and random two-point exchange mutation, and the machine selection information in the solution of the competitive breeding sub-population is evolved by UX crossover and random replacement machine mutation, to obtain the competitive breeding sub-population after mutation evolution; For each solution in the competitive breeding sub-population after mutation and evolution, while keeping the process sequence corresponding to each workpiece in the solution unchanged, rearrange the production sequence between the processes of different workpieces, and continuously arrange the operation time of the machines to obtain the breeding sub-population after evolution and update; The solutions in the common sub-population are randomly paired into groups of two solutions, the fitness values ​​of the two solutions in each group are compared, and the solution with the best fitness value in each group is used to form a competitive common sub-population; For the process selection information in the solutions of the competitive ordinary sub-population, the improved POX crossover method and the mutation strategy evolution caused by the gene sequence are adopted. For the machine selection information in the solutions of the competitive ordinary sub-population, the uniform crossover and the machine mutation strategy of randomly selecting the machine with the shorter processing time are adopted to obtain the competitive ordinary sub-population after mutation evolution. For each solution in the competitive ordinary sub-population after mutation and evolution, while keeping the process sequence corresponding to each workpiece in the solution unchanged, the production sequence between the processes of different workpieces is rearranged, and the operation time of the machines is continuously arranged to obtain the ordinary sub-population after evolution and update.

5. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 2, characterized in that: For each solution in the new population, the D3QN deep reinforcement learning network is used to select a local search operator corresponding to the solution in a preset action space, and the solution is evolutionarily updated according to the local search operator corresponding to the solution to obtain a solution after D3QN deep reinforcement learning, all solutions after D3QN deep reinforcement learning are combined to obtain a new population after D3QN deep reinforcement learning, and the new population after D3QN deep reinforcement learning is used as the current parent population, including: For each solution in the new population, a local search operator corresponding to the solution is selected from a preset action space according to a greedy strategy; Evolutionarily updating the solution according to the local search operator corresponding to the solution to obtain a solution corresponding to the solution after deep reinforcement learning by D3QN; Determine the total processing time and total energy consumption of the solution and the corresponding solution after D3QN deep reinforcement learning respectively; Determine a reward value corresponding to the solution after D3QN deep reinforcement learning according to the total processing time and total energy consumption of the solution and the corresponding solution after D3QN deep reinforcement learning; The solution, the solution after D3QN deep reinforcement learning corresponding to the solution, the total processing time and total energy consumption of the solution, the total processing time, total energy consumption and reward value of the solution after D3QN deep reinforcement learning corresponding to the solution are put into the preset memory library as this experience; Randomly extract a second preset number of samples in a small batch from a preset memory bank, and calculate a target difference of the extracted samples; The evaluation network parameters and the target network parameters in the D3QN deep reinforcement learning network are updated according to the target difference; wherein the D3QN deep reinforcement learning network includes an evaluation network and a target network, the evaluation network is used to generate Q values ​​of all possible actions in the current state, and the target network is used to stabilize the training process, provide a relatively stable target Q value, and reduce the instability in the learning process.

6. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 5, characterized in that: The action space includes three types of local search operators: classic operator class, destruction and reconstruction operator class and critical path operator class; The classical operator class includes: randomly selecting two processes and exchanging their positions, randomly selecting a process and inserting it into a position, randomly selecting two processes and inserting one process before the other process, and / or randomly selecting two processes and inserting one process after the other process; The destruction and reconstruction operator class includes: randomly generating a 0-1 array, retaining values ​​of 1 and randomly assigning values ​​of 0 to other positions, randomly selecting a point, keeping the right side of the point unchanged and randomly shuffling the left side, randomly selecting a point, keeping the left side of the point unchanged and randomly shuffling the right side, and / or randomly selecting two points, shuffling the process between the two points; The critical path operator class includes: randomly selecting a process on the critical path and randomly moving all processes of the workpiece to which it belongs to another factory, randomly selecting a process on the critical path, selecting a processing factory through a roulette wheel selection system, and moving all processes of the workpiece to which the process belongs to the selected factory, randomly selecting a process on the critical path, randomly reselecting a machine tool for processing within its optional machine range, and / or randomly selecting a process on the critical path, selecting a processing machine using a roulette wheel, and moving the current process to the selected machine for processing.

7. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 5, characterized in that: The state space includes: indicators related to the overall attributes of the scheduling problem, indicators related to the workshop processing time, and indicators related to the energy consumption attributes; The indicators related to the overall properties of the scheduling problem include: the number of workpieces, the number of machines, and / or the total number of processes; The indicators related to workshop processing time include: total machine completion time, machine idle / processing time, total completion time of each factory, and / or machine idle and processing time of each factory; The indicators related to energy consumption attributes include: total energy consumption of machines and / or energy consumption of each factory.

8. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 1, characterized in that: The multi-objective optimization model is specifically the following formula: MIN{Cmax,TEC} Among them, MIN means minimization, C max represents the maximum completion time, TEC represents the total energy consumption; Among them, MAX represents the time consumed by the workpiece that takes the longest time to complete all processes among all workpieces; The nth i The processing completion time of a process in factory f; TEC=E w +E I Among them, E w Indicates the energy consumption of machine processing; E I represents the idle energy consumption of the machine; i represents the workpiece index; I represents the workpiece set; j represents the process index, J i represents the process set of workpiece i; f represents the factory index; F represents the factory set; k represents the machine index; M represents the machine set; t represents the processing position index number on the machine; P k represents the set of processing locations of machine k in factory f; W p Indicates the machine operating power; T i,j,f,k represents the processing time of the jth process of the i-th workpiece on the machine k in factory f; X i,j,f,k,t The value is 0 or 1, indicating whether the jth process of the i-th workpiece is processed on the tth position of the machine k in the factory f. If it is processed, it is 1, otherwise it is 0; P' k represents the front P of machine k in factory f k-1 A set of processing positions; W s Indicates the idle power of the machine; B f,k,t+1 represents the start time of processing at position t+1 of machine k in factory f; B f,k,t It represents the start time of processing at the tth position of machine k in factory f.

9. The distributed heterogeneous flexible job shop energy-saving scheduling method according to claim 2, characterized in that: The preset scheduling constraints include: The same workpiece can only be assigned to one factory, and all processes of the workpiece are completed in the assigned factory; The same process can be processed on the same machine, but the processing time on the machines in different factories is different; The same process of the same workpiece can only be processed by one machine at the same time; Once the processing of each step of each workpiece begins, the processing cannot be interrupted; Different artifacts have the same priority; There is no order constraint between the processes of different workpieces, but there is order constraint between the processes of the same workpiece; All workpieces and machines are ready for processing at time zero.

Images