Coevolution method for distributed heterogeneous flexible job shop scheduling

Abstract

The invention discloses a co-evolution method for distributed heterogeneous flexible job shop scheduling, which designs a three-section type coding and full-active decoding method based on three coupling sub-problems, and constructs a search space for the distributed heterogeneous flexible job shop scheduling problem. On the basis of a key path and key block theory, various neighborhood structures are designed for three-section type coding and serve as individual search directions; a deep reinforcement learning technology is utilized to perform feature extraction on a key factory, and a training algorithm is designed to train an intelligent agent, so that customized neighborhood structures can be selected for different individuals, and the diversity of a population is increased; a probability matrix is calculated in the training process, a high-quality population is generated in the co-evolution process, and convergence of the algorithm is accelerated through crossover and mutation operators among sub-populations. The method can also adapt to different scales and complexity problems through learning and optimization of the intelligent agent, and the overall performance and the resource utilization rate of a production system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a knowledge-driven distributed estimation co-evolution method, which is an optimization method for distributed heterogeneous flexible job shop scheduling problems and belongs to the field of intelligent manufacturing and optimized scheduling in complex production systems. Background Art

[0002] As more and more production tasks with heterogeneous characteristics are modeled as distributed heterogeneous flexible job shop problems and widely used in complex discrete manufacturing systems, research in this field has become increasingly important. Generally speaking, with the expansion of production scale, large-scale production tasks need to be assigned to multiple production centers for processing to ensure the rational use of resources and maximize production efficiency. However, in the actual production process, the heterogeneous characteristics between production centers are difficult to ignore, such as the differences in the technical levels of different workers, the processing capabilities of production equipment, and the production environment. These heterogeneities increase the complexity of task allocation, and ultimately lead to different processing times for the same production task in different production centers. In addition, due to the heterogeneous resources, flexible equipment, and complex constraints of the distributed heterogeneous workshop environment, the scheduling problem of distributed heterogeneous flexible job shops still faces huge challenges.

[0003] Under the paradigms of classical computing theory and combinatorial optimization, the job shop scheduling problem has been proven to be an NP-hard problem. As an extension of this problem, the distributed heterogeneous flexible job shop scheduling problem consists of multiple strongly coupled sub-problems, so it is a more complex strongly NP-hard problem, which means that it is computationally very difficult to find its optimal scheduling solution. Due to the huge difficulty in solving this problem and the difficulty in decoupling, previous methods often only optimize for homogeneous environments. In order to take into account heterogeneity and make the design and evaluation of algorithms more intuitive and direct, while effectively utilizing computing resources and speeding up the optimization process, the distributed heterogeneous flexible job shop scheduling problem is modeled as a single-objective optimization problem to provide the optimal scheduling solution for decision-making. By optimizing the production cycle, the optimization solution has good robustness and can more flexibly respond to changes and uncertainties that may occur in the production process, so that customers can obtain efficient scheduling solutions.

[0004] The goal of the distributed heterogeneous flexible job shop scheduling problem is to establish an effective match between production tasks and production resources, which is also a complex discrete combinatorial optimization problem. As the scale of production tasks increases, the problem may evolve into a large-scale optimization problem. Metaheuristic algorithms, as a random search method, are often used to solve complex combinatorial optimization problems. However, metaheuristic algorithms still have the shortcomings of traditional evolutionary algorithms. For example, during the iteration process, the diversity of the algorithm population may decrease, and the experience of the solution space that has been searched cannot be effectively utilized. In addition, the lack of knowledge for specific problems and the blind randomized search method increase the solution time; the lack of targeted neighborhood structure may cause the search to converge or stagnate prematurely; and it is impossible to effectively balance exploration and development, global and local search. In order to improve these shortcomings, the present invention uses deep reinforcement learning technology to train an intelligent agent through a data-driven offline pre-training method to guide the search direction of individuals in the population. At the same time, a probabilistic model is constructed and the model is used to co-evolve the population. This method can effectively combine knowledge for specific problems and has a faster convergence speed. Therefore, the knowledge-driven distribution estimation co-evolutionary algorithm is beneficial for solving the distributed heterogeneous flexible job shop scheduling problem. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a single-objective distributed heterogeneous flexible job shop scheduling method. The method combines deep reinforcement learning technology and specific domain knowledge to optimize the production cycle of heterogeneous workshops. In addition, the method utilizes the scheduling characteristics of flexible job shops and simultaneously builds a probabilistic model during the training process, so that the agent and the model can be applied to problems of any scale; and by using problem-specific knowledge, an efficient neighborhood structure is designed, so that users can obtain high-quality scheduling solutions in a short time, thereby achieving a balance between user needs for efficient production and optimal resource utilization.

[0006] Technical solution: To achieve the above purpose, the present invention proposes a single-objective distributed heterogeneous flexible job shop scheduling method. This method is mainly based on three coupled sub-problems, designs a three-stage coding and fully active decoding method, and constructs the search space of the distributed heterogeneous flexible job shop scheduling problem; based on the critical path and critical block theory, a variety of neighborhood structures are designed for the three-stage coding as the search direction of individuals; using deep reinforcement learning technology, feature extraction of key factories, designing training algorithms to train intelligent agents, so that they can select customized neighborhood structures for different individuals and increase the diversity of the population; calculating the probability matrix during the training process, generating high-quality populations in the co-evolution process, and accelerating the convergence of the algorithm through crossover and mutation operators between sub-populations.

[0007] A distributed heterogeneous flexible job shop scheduling method comprises the following steps:

[0008] Step 1: For the three coupled sub-problems of distributed heterogeneous flexible job shop scheduling, a three-segment encoding strategy is used to encode different sub-problems to represent a scheduling solution, and a fully active decoding strategy is used to solve the target value of the scheduling solution, thereby constructing the problem space;

[0009] Step 2: Randomly generate scheduling data for training, including the number of workpieces, the number of processes contained in each workpiece, the set of optional machines for each process, the processing time of each process on the optional machines, the number of factories, and the number of machines owned by each factory;

[0010] Step 3: Based on the scheduling data generated in step 2, randomly initialize a scheduling solution S represented by a three-segment code 0 , as the starting point of a training round;

[0011] Step 4: Extract state features of the scheduling solution S currently represented by the three-segment encoding, and use the extracted state features s as the input of the intelligent agent;

[0012] Step 5: Design the neighborhood structure based on the three-segment encoding strategy as the action a output by the agent;

[0013] Step 6: Apply the action a currently output by the agent to the current scheduling solution S, calculate the reward value r according to the change of the total production cycle makespan, update the current scheduling solution and extract its feature s′, and save the four-tuple (s, a, r, s′) to the experience pool;

[0014] Step 7: Randomly select several quadruplets from the experience pool to train the agent and optimize the agent's decision-making ability;

[0015] Step 8: Determine whether the current training round has ended. If a negative reward value is obtained for a set number of consecutive times, the current training round ends. optimal The three-segment encoding is sampled, the statistical model is updated, and step 9 is entered; otherwise, the current training round is not completed, and step 4 is returned;

[0016] Step 9: Determine whether the maximum number of training rounds has been reached. If not, return to step 2. If reached, form a probability model based on the statistical model and output the trained agent and probability model.

[0017] Step 10: Initialize the population. All three-stage encodings use a random initialization strategy and calculate the fitness value of each individual.

[0018] Step 11: Generate a competing population using the probability model obtained in step 9, merge it with the current population, and perform crossover within the merged population to select the top 50% of the elite population as the next generation population;

[0019] Step 12: Use the intelligent agent obtained in step 9 to select the optimal neighborhood structure for each individual in the elite population and improve each individual;

[0020] Step 13: Determine whether the maximum number of iterations has been reached. If not, return to step 11; if reached, output the individual with the best fitness in the current population as the optimal solution.

[0021] Furthermore, the three sub-questions in step 1 are:

[0022] (1) Factory allocation: a set of jobs is allocated to multiple distributed heterogeneous factories, and each job needs to select a suitable factory for processing;

[0023] (2) Machine selection: Each heterogeneous factory has multiple processing machines that can process the job. Each job consists of multiple processes, and each process needs to select an appropriate machine from the set of optional machines.

[0024] (3) Process sequencing: All processes of a job must be processed in the same factory.

[0025] Furthermore, the scheduling features in step 4 include the average SF of the completion time of each workpiece 1 and standard deviation SF 2 , the utilization rate of each machine SF 3 , the average completion time of each machine SF 4 and standard deviation SF 5 , the average waiting time of each process SF 6 and standard deviation SF 7 , the average processing time of each process SF 8 and standard deviation SF 9 , and load balancing between different plants SF 10 .

[0026] Furthermore, the neighborhood structure in step 5 includes:

[0027] (1) Move the key process in a key block to another key block;

[0028] (2) Randomly select key processes and assign them to other machines according to probability;

[0029] (3) randomly select a key job and randomly assign it to other plants;

[0030] (4) Exchange key operations between factories.

[0031] Furthermore, in step 6, the reward value r is calculated according to the change of the total production cycle makespan as follows:

[0032]

[0033] Furthermore, the statistical model initialization formula in step 8 is as follows:

[0034] C[i,k]=0fori=1,…,Landk=1,…K

[0035] Where: C[i,k] is the i-th row and k-th column element of the frequency matrix C used to count the occurrence of integers at each position of the three-segment code, L = N + M, representing the number of N workpieces, M represents the number of machines, and K represents the range of integers in different codes;

[0036] The update formula of the statistical model is as follows:

[0037] C[i,code[i]]=C[i,code[i]]+1

[0038] Where: code[i] represents the integer value at position i in different codes.

[0039] Furthermore, the expression of the probability model in step 9 is as follows:

[0040]

[0041] Furthermore, in step 11, the probability model obtained in step 9 is used to generate a competitive population, which specifically includes:

[0042] For each position i of the three-segment encoding, the cumulative probability distribution C is constructed using the probability model P[i,k] i The formula is as follows:

[0043]

[0044] Where j represents the cumulative probability distribution C i The index value in ;

[0045] Generate a random number r∈[0,1] and select i [j-1] <r≤C i The integer [j] is used as the value at position i in the three-segment encoding, and finally an individual solution generated based on statistics is obtained.

[0046] Furthermore, the crossover method in step 11 includes:

[0047] For OS vectors, the POX crossover operator is used to exchange gene sequences between parent individuals through partial mapping to generate new offspring individuals. The specific steps are as follows:

[0048] Sub-step 1: After inputting the parent individuals p1 and p2, all artifacts are randomly divided into two non-empty subsets J1 and J2;

[0049] Sub-step 2: Create two empty offspring individuals c1 and c2, whose lengths are the same as p1 and p2 respectively;

[0050] Sub-step 3: Traverse p1 and p2, and copy the processes in p1 that belong to J1 to c1 according to their original positions; and copy the processes in p2 that belong to J2 to c2 according to their original positions;

[0051] Sub-step 4: traverse the parent chromosomes p1 and p2, and copy the processes in p1 that belong to J2 to the unallocated positions in c1 in sequence; and copy the processes in p2 that belong to J1 to the unallocated positions in c2 in sequence;

[0052] Sub-step 5: Complete the generation of two offspring individuals.

[0053] Furthermore, the crossover method in step 11 includes:

[0054] Use sequential crossover for MS vectors and FA vectors. The specific steps are as follows:

[0055] Randomly select two intersection points, and the part between the two intersection points is called a "substring". At the same time, the two intersection points divide the parent vector into three parts: front, middle, and back.

[0056] Directly copy the substring in the first parent individual p1 to the same position in the first offspring individual c1. From the second parent individual p2, select the genes that are not in the substring in the order from the right crossover point to the starting point, and fill them in the remaining vacancies in the first offspring individual c1 in sequence. For the second offspring c2, repeat the above steps, and the individuals use the substring of the second parent p2 and the order of the first parent p1 to generate two offspring c1 and c2 after the crossover.

[0057] Beneficial effects: Starting from the characteristics of the heterogeneous job shop scheduling environment, the present invention aims at the dependency between the three coupled sub-problems in the scheduling problem, combines the problem-specific domain knowledge, adopts deep reinforcement learning technology, distribution estimation algorithm and co-evolution algorithm, and proposes a novel single-objective distributed heterogeneous flexible job shop scheduling method, which effectively solves the shortcomings of traditional methods in dealing with complex heterogeneous environments and improves the quality and efficiency of scheduling solutions. This method can not only accelerate the convergence speed, but also adapt to problems of different scales and complexities through the learning and optimization of intelligent agents, and significantly improve the overall performance and resource utilization of the production system. This method can provide users with high-quality scheduling solutions in a short time, and is suitable for complex manufacturing systems, batch production, and various industrial environments that require efficient resource allocation. The present invention can also be expanded by adding constraints, such as worker scheduling, transportation and assembly processes, to cope with more complex needs in actual production. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the processing flow of distributed heterogeneous flexible job shops.

[0059] Figure 2 It is the overall flow chart of the present invention.

[0060] Figure 3 It is a Gantt chart of the distributed heterogeneous flexible job shop scheduling model.

[0061] Figure 4 It is a schematic diagram of the neighborhood structure based on the process sequence vector OS.

[0062] Figure 5 It is a schematic diagram of the neighborhood structure based on the machine selection vector MS.

[0063] Figure 6 It is a schematic diagram of the neighborhood structure based on the factory allocation vector FA.

[0064] Figure 7 It is a flowchart of agent training.

[0065] Figure 8 It is a schematic diagram of the POX crossover operator of the present invention.

[0066] Fig. 9 It is a schematic diagram of the sequential crossover operator of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described below in conjunction with the accompanying drawings.

[0068] The processing flow of the distributed heterogeneous flexible workshop involved in the present invention is as shown in the attached figure. Figure 1As shown. The workpieces to be processed need to be distributed to distributed heterogeneous factories, and each factory has a flexible job shop scheduling problem, forming three coupled sub-problems of the problem. The present invention first needs to train the neural network and the probability model, and then use the neural network as an intelligent agent, and use the output of the probability model to initialize a population composed of random scheduling schemes, and calculate the fitness value of each scheduling scheme. Then use the constructed probability model for co-evolution, and the intelligent agent selects a customized neighborhood structure for each individual. The overall process is shown in the attached figure. Figure 2 , the specific technical solution includes the following steps:

[0069] Step 1: For the three coupled sub-problems of distributed heterogeneous flexible job shop scheduling, a three-segment (OS, MS, FA) encoding strategy is used to encode different sub-problems to represent a scheduling solution, and a fully active decoding strategy is used to solve the target value of the scheduling solution, which is the maximum completion time (Makespan), thereby constructing the problem space.

[0070] Step 2: Randomly generate scheduling data for training, including the number of workpieces n, the number of processes contained in each workpiece o, and the optional machine set {M 1 ,M 2 ,…,M k}, the processing time Ptime of each process on the optional machines, the number of factories f, and the number of machines m owned by each factory.

[0071] Step 3: Based on the data generated in step 2, randomly initialize a three-segment coded scheduling solution S 0 , as the starting point of a training round.

[0072] Step 4: Extract state features of the current scheduling solution represented by the three-segment encoding, including the average SF of the completion time of each workpiece 1 and standard deviation SF 2 , the utilization rate of each machine SF 3 , the average completion time of each machine SF 4 and standard deviation SF 5 , the average waiting time of each process SF 6 and standard deviation SF 7 , the average processing time of each process SF 8 and standard deviation SF 9 , load balancing between different factories SF 10 , and the extracted state feature s is used as the input of the agent.

[0073] Step 5: Design the neighborhood structure based on the three-segment coding strategy as the action a output by the agent, including: based on the process sequence vector OS: move the key process in the key block to another key block NSO; based on the machine selection vector MS: randomly select the key process and assign it to other machines according to probability NSM; based on the plant allocation vector FA: randomly select a key job and randomly assign it to other factories NSF1; exchange key jobs between factories NSF2.

[0074] Step 6: Apply the action a currently output by the agent to the current scheduling solution S, calculate the reward value r according to the change of the total production cycle makespan, update the current scheduling solution and extract its feature s′, and save the four-tuple (s, a, r, s′) to the experience pool.

[0075] Step 7: Randomly select several quadruplets from the experience pool to train the agent. By using the data of these quadruplets, the agent can learn the value of different actions in different states, and then use the gradient descent algorithm to adjust the parameters of the agent network to minimize the loss function and optimize the agent's decision-making ability.

[0076] Step 8: Determine whether the current training round has ended. If negative reward values ​​are obtained for 5 consecutive times, the round ends and a better scheduling solution S will be obtained. optimal , sample the three-segment encoding of the solution, and update the statistical model, including recording the frequency of occurrence of each value at each position in each encoding of the three-segment encoding, so as to further optimize and improve the accuracy and performance of the model, and then enter step 9; otherwise, the round is not over, and return to step 4.

[0077] Step 9: Determine whether the maximum number of training rounds has been reached. If not, return to step 2; if reached, convert the frequency in the statistical model into probability to form a probability model, and output the trained agent and probability model.

[0078] Step 10: Initialize the population. All three-stage encodings use a random initialization strategy and calculate the fitness value of each individual.

[0079] Step 11: Use the probability model obtained in step 9 to generate a high-quality competitive population, merge it with the current population, and perform crossover within the merged population to select the top 50% of the elite population as the next generation population.

[0080] Step 12: Use the neural network agent obtained in step 9 to select the optimal neighborhood structure for each individual in the elite population and improve the individual.

[0081] Step 13: Determine whether the maximum number of iterations has been reached. If not, return to step 11; if reached, output the individual with the best fitness in the population as the optimal solution found.

[0082] 1. Constructing the problem space based on three-stage encoding and fully active decoding strategy

[0083] In the distributed heterogeneous flexible job shop scheduling problem, the complex problem space mainly includes three coupled sub-problems: (1) factory allocation: a set of jobs are allocated to multiple distributed heterogeneous factories, and each job needs to select a suitable factory for processing; (2) machine selection: each heterogeneous factory has multiple processing machines that can process the job, and each job consists of multiple processes. Each process needs to select an appropriate machine from the set of optional machines; (3) process sorting: all processes of a job must be processed in the same factory. However, the processing time of the same process in different factories is different, and the processing order of the processes on each machine needs to be determined. These three sub-problems have a close dependency relationship and together constitute the complexity and challenge of the distributed heterogeneous flexible job shop scheduling problem. Therefore, a feasible solution is represented by three vectors to collaboratively represent the complete problem space of the heterogeneous job shop scheduling problem. First, the OS vector is used to represent the processing order of the process. Its length is the total number of processes. Each element of the vector is an integer between 1 and |J| (the maximum job number), and the number of repetitions of the element represents the process number that needs to be processed. The MS vector represents the mapping relationship between jobs and machines. The length of the MS vector is the same as that of the OS vector. Each integer in the vector represents the machine selected by the process of each job. The FA vector represents the distribution of jobs. Its length is equal to the number of jobs. Each subscript represents a job, and its corresponding value is an integer from 1 to |F| (number of factories), indicating the factory to which the job is assigned. Table 1 is an example of a distributed heterogeneous flexible job shop scheduling problem.

[0084] Table 1

[0085]

[0086] Fully active decoding is a decoding strategy for scheduling problems and other combinatorial optimization problems. It can flexibly cope with various constraints and changes in complex scheduling problems. It ensures the feasibility and optimization of the solution by decoding the task to the first position in the topological relationship. In the vector FA, when the i-th element is set to f, it means that the i-th job is assigned to the f-th factory. Therefore, the distribution of the job can be quickly obtained. Based on this, the OS vector is decomposed into multiple sub-vectors OS', and the production cycle (makespan) is calculated through the following steps. The example of fully active decoding is attached. Figure 3 shown.

[0087] (1) Initialize the scheduling schedule of each factory: According to the job allocation, initialize the scheduling schedule of each factory and record the idle time of each machine.

[0088] (2) Task Scheduling: Extract processes from the OS' vector step by step and assign machines to each process according to the MS vector. Record the start and end time of each process to ensure the sequence of processes and the availability of machines.

[0089] (3) Calculate processing time: Based on the selected machine and processing time for each process, update the scheduling table of the corresponding machine and record the completion time of each process.

[0090] (4) Update factory status: After each process is assigned, the status of the corresponding factory and machine is updated, including the next available time of the machine and the completion time of the current process.

[0091] (5) Determine the completion time of each factory: After all processes are assigned and completed, determine the completion time of the last process in each factory.

[0092] (6) Calculate makespan: Select the maximum value from the completion time of all factories as the overall production cycle (makespan), and this factory is the key factory.

[0093] 2. Neighborhood structure based on critical path theory

[0094] In the job shop scheduling problem, the neighborhood structure refers to a method of generating a new solution by making local adjustments to the current solution. The design of the neighborhood structure directly affects the search efficiency of the scheduling optimization algorithm and the quality of the final solution. Based on the conclusion that "moving non-critical processes cannot reduce the length of the critical path" and the three-stage coding strategy, this paper designs different neighborhood structures NSO, NSM, and NSF for OS, MS, and FA in the three-stage coding:

[0095] (1) Move the key process in the key block to another key block NSO: On the critical path CP = {O 1 ,O 2 ,…,O n}

[0096] In the process, a key process is randomly selected and moved into other key blocks. The new key path after the exchange is CP′. Figure 4 shown.

[0097] (2) Randomly select the key process and assign it to other machines according to probability. NSM: Randomly select the critical path CP = {O 1 ,O 2 ,…,O n A key process in i, the optional machine set of the process is {M 1 ,M 2 ,…,M k}, use proportional selection to reselect O i The probability of each machine being selected is calculated as follows:

[0098]

[0099] Process O i On machine M j The processing time on is p ij , P j Indicates that machine M is selected j The probability of Figure 5 shown.

[0100] (3) Randomly select a key task and randomly assign it to another factory NSF1: Randomly select the critical path CP = {O 1 ,O 2 ,…,O n A key process in i , randomly assign the job of the process to other factories for processing, and obtain a new FA vector; exchange the key job NSF2 between factories. Figure 6 shown.

[0101] 3. Data-driven offline pre-training method for intelligent agents

[0102] The process transformation in a distributed heterogeneous flexible job shop can be modeled as a Markov decision process. The agent training process is as follows: Figure 7 The training steps of the deep Q network as the agent are as follows:

[0103] Sub-step 1: Randomly generate scheduling data for training, including the number of workpieces n, the number of processes contained in each workpiece o, and the optional machine set {M 1 ,M 2 ,…,M k}, the processing time Ptime of each process on the optional machines, the number of factories f, and the number of machines m owned by each factory.

[0104] Sub-step 2: Based on the data generated in sub-step 1, randomly initialize a scheduling solution S represented by a three-segment encoding 0 , as the starting point of a training round.

[0105] Sub-step 3: Extract state features of the current scheduling solution represented by the three-segment encoding, and use the extracted state features s as the input of the agent. The state features are shown in Table 2.

[0106] Table 2

[0107] Serial number Status feature name describe 1 <![CDATA[SF 1 ]]> The average completion time of each workpiece 2 <![CDATA[SF 2 ]]> The standard deviation of the completion time of each workpiece 3 <![CDATA[SF 3 ]]> Utilization rate of each machine 4 <![CDATA[SF 4 ]]> The average completion time of each machine 5 <![CDATA[SF 5 ]]> The standard deviation of the completion time of each machine 6 <![CDATA[SF 6 ]]> The average waiting time of each process 7 <![CDATA[SF 7 ]]> The standard deviation of waiting time for each process 8 <![CDATA[SF 8 ]]> The average processing time of each process 9 <![CDATA[SF 9 ]]> Standard deviation of processing time for each process 10 <![CDATA[SF 10 ]]> Load balancing between different factories

[0108] The neighborhood structure is designed based on the three-segment encoding strategy as the action a output by the agent. The actions are shown in Table 3.

[0109] Table 3

[0110]

[0111]

[0112] Sub-step 4: Apply the action a currently output by the agent to the scheduling solution S represented by the three-segment encoding, calculate the reward value r according to the change of the total production cycle makespan, and update the current solution. Then extract its feature s′ and save the four-tuple (s, a, r, s′) to the experience pool.

[0113]

[0114] Sub-step 5: Randomly select several quadruplets from the experience pool to train the agent. By using the data of these quadruplets, the agent can learn the value of different actions in different states, and then use the gradient descent algorithm to adjust the parameters of the agent network to minimize the loss function. The loss function formula is as follows, where Indicates that at time step t, based on the Q network parameters The loss function of s t represents the state at time step t; a t represents the action taken at time step t; Q(s t ,a t θ Q ) means in state s t Next, select action a t When Q network parameter θ Q Q value function; N represents the number of four-tuple samples; y t Represents the target Q value calculated based on the Bellman equation.

[0115]

[0116] Sub-step 6: Determine whether the current training round has ended. If the round has ended, a better scheduling solution S will be obtained. optimal , sample the three-segment encoding of the solution, update the statistical model, including recording the frequency of occurrence of each value at each position in the encoding to further optimize and improve the accuracy and performance of the statistical model, and then enter the next round; if the round is not over, return to sub-step 3.

[0117] Sub-step 7: Determine whether the maximum number of training rounds has been reached. If not, return to sub-step 1; if reached, convert the frequency in the statistical model into probability, thereby forming a probability model, and output the trained agent and probability model. 4. Method for constructing individual solutions based on probability models

[0118] Establishing a probability model while training the agent can make more efficient use of existing training samples. This method can learn from historical experience and continuously optimize the generated individual solutions, thereby improving the sample efficiency of the algorithm. As the training progresses, the probability model will be continuously updated, gradually reflecting the structure and distribution characteristics of the problem space. This adaptability enables the generated individual solutions to better adapt to the current problem environment and improve the adaptability and robustness of the algorithm. Assume that the three-segment code length is L, including N workpieces and M machines. The integer at each position can be represented as a process number, workpiece number, or machine number. The method of establishing a probability model and constructing an individual solution is as follows:

[0119] (1) Initialize the statistical model.

[0120] Define a matrix C to count the frequency of integers at each position of the three-segment encoding. The size is L×K, where K is the range of integers in different encodings. The initialization formula is as follows.

[0121] C[i,k]=0fori=1,…,Landk=1,…K

[0122] (2) Update the statistical model.

[0123] After each training round, a high-quality solution is obtained, and the statistical model C is updated using the solution. The update formula is as follows, where code[i] is the integer value at index position i in different codes.

[0124] C[i, code[i]]=C[i, code[i]]+1,i=1,…L

[0125] (3) Calculate the probability model.

[0126] Define a probability matrix P, where each element P[i,k] represents the probability of number k appearing at position i. The formula for calculating matrix P using matrix C is as follows.

[0127]

[0128] (4) Use probabilistic models to construct individual solutions to form a high-quality competitive population.

[0129] For each position i of the three-segment encoding, the cumulative probability distribution C is constructed using the probability model P[i,k] i The formula is as follows, where j represents the cumulative probability distribution Ci The index value in .

[0130]

[0131] Generate a random number r∈[0,1] and select i [j-1] <r≤C i [j], which corresponds to the specific value of position i in the three-segment code. This process will traverse all positions i (from 1 to L) of the code, select an integer for each position, and gradually build a complete high-quality individual solution. Use this statistical method to generate individual solutions with the same number as the current population to form a competitive population, and then merge with the current population.

[0132] 5. Co-evolution strategy based on competitive relationship

[0133] In the present invention, the cooperative evolution strategy based on competition relationship is to use the obtained probability model to generate a high-quality competitive population, merge it with the initialization population, and crossover within the merged population to select the top 50% elite population as the next generation population. Then use the obtained neural network agent to select the optimal neighborhood structure for each individual in the elite population to improve the individual. The crossover operator is as follows:

[0134] For OS vectors, the present invention uses the POX (precedence operation crossover) crossover operator, which exchanges gene sequences between parent individuals through partial mapping to generate new offspring individuals, which can retain the excellent characteristics of the parent generation and introduce new gene combinations to increase the diversity of the population and improve the search efficiency and optimization effect of the genetic algorithm. The schematic diagram of POX is shown in the attached figure. Figure 8 As shown, the sub-steps are as follows:

[0135] Sub-step 1: Randomly divide the artifact set. After inputting the parent individuals p1 and p2, first randomly divide all artifacts into two non-empty subsets J1 and J2. The purpose of this step is to mix the characteristics of the parent in the offspring individuals.

[0136] Sub-step 2: Initialize the offspring individuals. Create two empty offspring individuals c1 and c2, which have the same length as the parent individuals p1 and p2, but all positions are empty. This step prepares for the subsequent process replication.

[0137] Sub-step 3: Traverse the parent individuals p1 and p2, and copy the processes in p1 that belong to J1 to the child individual c1 in their original positions. Similarly, copy the processes in p2 that belong to J2 to the child individual c2 in their original positions.

[0138] Sub-step 4: Traverse the parent individuals p1 and p2 again, and copy the steps in p1 that belong to J2 to the unassigned positions in the child individual c1. Similarly, copy the steps in p2 that belong to J1 to the unassigned positions in the child individual c2.

[0139] Sub-step 5: Output offspring individuals. Finally, two offspring individuals c1 and c2 are generated, which are a mixture of the characteristics of the parent chromosomes and have potential excellent characteristics.

[0140] Sequential crossover is used for MS vector and FA vector. The schematic diagram of sequential crossover is shown in the attached figure. Fig. 9 As shown, the specific steps are as follows:

[0141] First, two crossover points are randomly selected, and the part between these two points is called a "substring". These crossover points divide the parent vector into three parts: front, middle, and back. Then, the substring in the first parent individual p1 is directly copied to the same position in the first offspring individual c1. From the second parent individual p2, in the order of appearance (starting from the right crossover point and going around to the starting point), select those genes that are not in the substring, and fill in the remaining vacancies in the first offspring individual c1 in sequence. For the second offspring c2, repeat the above steps, using the substring of the second parent p2 and the order of the first parent p1. Thus, two offspring c1 and c2 are generated after the crossover. This method strictly follows the topological structure of the tasks in the workflow, ensuring that the generated solution is feasible and conforms to the sequence and dependencies in the actual problem.

Claims

1. A collaborative evolution method for distributed heterogeneous flexible job shop scheduling, characterized in that: The steps include: Step 1: For the three coupled sub-problems of distributed heterogeneous flexible job shop scheduling, a three-segment encoding strategy is used to encode different sub-problems to represent a scheduling solution, and a fully active decoding strategy is used to solve the target value of the scheduling solution, thereby constructing the problem space; Step 2: Randomly generate scheduling data for training, including the number of workpieces, the number of processes contained in each workpiece, the set of optional machines for each process, the processing time of each process on the optional machines, the number of factories, and the number of machines owned by each factory; Step 3: Based on the scheduling data generated in step 2, randomly initialize a scheduling solution S0 represented by a three-segment code as the starting point of a training round; Step 4: Extract state features of the scheduling solution S currently represented by the three-segment encoding, and use the extracted state features s as the input of the intelligent agent; Step 5: Design the neighborhood structure based on the three-segment encoding strategy as the action a output by the agent; Step 6: Apply the action a currently output by the agent to the current scheduling solution S, calculate the reward value r according to the change of the total production cycle makespan, update the current scheduling solution and extract its feature s′, and save the four-tuple (s, a, r, s′) to the experience pool; Step 7: Randomly select several quadruplets from the experience pool to train the agent and optimize the agent's decision-making ability; Step 8: Determine whether the current training round has ended. If a negative reward value is obtained for a set number of consecutive times, the current training round ends. optimal The three-segment encoding is sampled, the statistical model is updated, and step 9 is entered; otherwise, the current training round is not completed, and step 4 is returned; Step 9: Determine whether the maximum number of training rounds has been reached. If not, return to step 2. If reached, form a probability model based on the statistical model and output the trained agent and probability model. Step 10: Initialize the population. All three-stage encodings use a random initialization strategy and calculate the fitness value of each individual. Step 11: Generate a competing population using the probability model obtained in step 9, merge it with the current population, and perform crossover within the merged population to select the top 50% of the elite population as the next generation population; Step 12: Use the intelligent agent obtained in step 9 to select the optimal neighborhood structure for each individual in the elite population and improve each individual; Step 13: Determine whether the maximum number of iterations has been reached. If not, return to step 11; if reached, output the individual with the best fitness in the current population as the optimal solution.

2. A collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 1, characterized in that: The three sub-questions in step 1 are: (1) Factory allocation: a set of jobs is allocated to multiple distributed heterogeneous factories, and each job needs to select a suitable factory for processing; (2) Machine selection: Each heterogeneous factory has multiple processing machines that can process the job. Each job consists of multiple processes, and each process needs to select an appropriate machine from the set of optional machines. (3) Process sequencing: All processes of a job must be processed in the same factory.

3. The collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 1, characterized in that: The scheduling features in step 4 include the average SF1 and standard deviation SF2 of the completion time of each workpiece, the utilization rate SF3 of each machine, the average SF4 and standard deviation SF5 of the completion time of each machine, the average SF6 and standard deviation SF7 of the waiting time of each process, the average SF8 and standard deviation SF9 of the processing time of each process, and the load balance SF 10 .

4. The collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 1, characterized in that: The neighborhood structure in step 5 includes: (1) Move the key process in a key block to another key block; (2) Randomly select key processes and assign them to other machines according to probability; (3) randomly select a key job and randomly assign it to other plants; (4) Exchange key operations between factories.

5. The collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 1, characterized in that: In step 6, the reward value r is calculated according to the change of the total production cycle makespan as follows:

6. The collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 1, characterized in that: The statistical model initialization formula in step 8 is as follows: C[i,k]=0for i=1,…,L andk=1,…K Where: C[i, k] is the i-th row and k-th column element of the frequency matrix C used to count the occurrence of integers at each position of the three-segment code, L = N + M, representing the number of N workpieces, M representing the number of machines, and K representing the range of integers in different codes; The update formula of the statistical model is as follows: C[i, code[i]] = C[i, code[i]] + 1 Where: code[i] represents the integer value at position i in different codes.

7. A collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 6, characterized in that: The expression of the probability model in step 9 is as follows:

8. The collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 7, characterized in that: In step 11, the probability model obtained in step 9 is used to generate a competitive population, which specifically includes: For each position i of the three-segment encoding, the cumulative probability distribution C is constructed using the probability model P[i, k] i The formula is as follows: Where j represents the cumulative probability distribution C i The index value in ; Generate a random number r∈[0,1] and select i [j-1] <r≤C i The integer [j] is used as the value at position i in the three-segment encoding, and finally an individual solution generated based on statistics is obtained.

9. The collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 1, characterized in that: The crossover method in step 11 includes: For OS vectors, the POX crossover operator is used to exchange gene sequences between parent individuals through partial mapping to generate new offspring individuals. The specific steps are as follows: Sub-step 1: After inputting the parent individuals p1 and p2, all artifacts are randomly divided into two non-empty subsets J1 and J2; Sub-step 2: Create two empty offspring individuals c1 and c2, whose lengths are the same as pl and p2 respectively; Sub-step 3: Traverse p1 and p2, and copy the processes in p1 that belong to J1 to c1 according to their original positions; and copy the processes in p2 that belong to J2 to c2 according to their original positions; Sub-step 4: traverse the parent chromosomes p1 and p2, and copy the processes in p1 that belong to J2 to the unallocated positions in c1 in sequence; and copy the processes in p2 that belong to J1 to the unallocated positions in c2 in sequence; Sub-step 5: Complete the generation of two offspring individuals.

10. The collaborative evolution method for distributed heterogeneous flexible job shop scheduling according to claim 1, characterized in that: The crossover method in step 11 includes: Use sequential crossover for MS vectors and FA vectors. The specific steps are as follows: Randomly select two intersection points, and the part between the two intersection points is called "substring". At the same time, the two intersection points divide the parent vector into three parts: front, middle and back. Create two empty offspring individuals c1 and c2; For c1, directly copy the substring in p1 to the same position in c1; from p2, in the order from the right crossover point to the starting point, select the genes that are not in the substring and fill the remaining vacancies in c1 in sequence; For c2, directly copy the substring in p2 to the same position in c2; from p1, in the order from the right crossover point to the starting point, select the genes that are not in the substring and fill the remaining vacancies in c2 in sequence; Complete the generation of two offspring individuals.

Images