A workshop production scheduling method and system based on deep reinforcement learning and multimodal multi-objective computation

Abstract

This invention discloses a workshop production scheduling method and system based on deep reinforcement learning and multimodal multi-objective computation. The method includes the following steps: obtaining a population size N, an objective function, and a maximum number of iterations; randomly generating an initial population based on the population size and the objective function, defining the state of the initial population, and constructing a deep Q-network; setting a counter and comparing the counter with a stagnation check period; adaptively selecting operators for the initial population using the deep Q-network to generate a child population; generating three temporary files based on the child population and merging them to obtain a merged file; performing environment selection on the merged file to obtain a new population and calculating state-reward pairs; updating the deep Q-network every 50 iterations based on the state-reward pairs; re-comparing the counter with the stagnation check period, and outputting the scheduling result if the counter reaches the stagnation check period.

Description

Technical Field

[0001] This invention belongs to the field of production scheduling technology, specifically relating to a workshop production scheduling method and system based on deep reinforcement learning and multimodal multi-objective computation. Background Technology

[0002] In recent years, the application of deep reinforcement learning (DRL) technology in multi-objective optimization problems has received widespread attention and in-depth research. Researchers mainly use DRL to solve complex multi-objective optimization problems in the real world, or to improve the performance of multi-objective algorithms. Luca et al. proposed an optimization method based on the DRL framework for virtual machine layout problems in cloud data centers. This method optimizes data center resource allocation by learning the optimal layout strategy for each virtual machine, significantly improving the system's resource utilization and operating efficiency. Stefan et al. designed a DRL-based Deep Coord method for a three-objective service coordination problem. In this method, the agent, through adaptive learning, can automatically coordinate the allocation of service tasks without prior knowledge or expert guidance, thus achieving optimization in complex dynamic environments. Li et al. proposed a multi-objective evolutionary algorithm based on the DRL framework. First, through a multi-objective optimization problem transformation method based on a decomposition strategy, the original multi-objective problem is decomposed into a set of structured scalar sub-optimization problems. Second, an adaptive neural network cluster is constructed, where each sub-problem is modeled as an independent network module. A parameter transfer mechanism based on neighborhood co-evolution and the DRL algorithm are introduced to achieve collaborative optimization of the sub-problem model parameters. Zhang et al. proposed a meta-learning-based DRL method to accelerate the solution process of multi-objective optimization problems. This method first trains a fine-tuned meta-model to derive corresponding sub-models for different sub-problems, effectively shortening the training time. Based on this, the technique is applied to multi-objective traveling salesman problems with time windows and multi-objective vehicle routing problems, significantly improving the efficiency and accuracy of optimization solutions.

[0003] In the field of multimodal multi-objective optimization, although scholars have proposed various improved algorithms and achieved significant progress, these methods still have several key shortcomings. First, regarding diversity maintenance, existing algorithms typically employ random reinitialization or simple perturbation strategies to maintain population diversity. This approach has two fundamental problems: on the one hand, it cannot accurately identify the critical stages when the population truly needs diversification; on the other hand, it may perform unnecessary diversity operations before the Pareto front has fully converged, potentially damaging the already obtained high-quality solution set. Second, in terms of decision space processing, current algorithms handle high-dimensional decision variables rather crudely. Although random adjustment of decision variables can generate diversity, the lack of a guiding mechanism for the global optimization direction can easily cause the population to deviate from the true multimodal Pareto optimal solution set. More importantly, during dynamic optimization, existing methods lack sufficient accuracy in evaluating the optimization state and fail to fully consider two important characteristics: first, different decision variable dimensions have asynchronous convergence characteristics; and second, the diversity requirements differ significantly across dimensions. This crude approach leads to the algorithm exhibiting excessive sensitivity to problem characteristics. To address these issues, designing innovative solutions that integrate the following key technologies within the framework of multimodal and multi-objective optimization algorithms is of significant research value: a dynamic policy selection mechanism based on reinforcement learning, a diversity preservation method considering dimensionality correlation, and an adaptive search space adjustment strategy oriented towards multimodal characteristics. This systematic improvement is expected to significantly enhance the overall performance of algorithms in complex multimodal and multi-objective optimization problems.

[0004] Currently, no literature has been found that uses deep reinforcement learning to guide a multimodal, multi-objective evolutionary algorithm with a serial-parallel mechanism. The deep reinforcement learning-guided multimodal, multi-objective evolutionary algorithm with a serial-parallel mechanism used in this patent should be relatively novel. Summary of the Invention

[0005] This invention aims to address the shortcomings of existing technologies and provides the following solutions:

[0006] A workshop production scheduling method based on deep reinforcement learning and multimodal multi-objective computation includes the following steps:

[0007] S1. Obtain the population size N, objective function, and maximum number of iterations. The objective function includes: job completion time, machine load, and total energy consumption.

[0008] S2. Based on the population size and the objective function, an initial population is randomly generated, the state of the initial population is defined, and a deep Q-network is constructed;

[0009] S3. Set a counter and compare the counter with the stall check period. If the counter has not reached the stall check period, execute S4-S7. If the counter has reached the stall check period, output the scheduling result.

[0010] S4. Use the deep Q-network adaptive selection operator to generate a sub-population from the initial population;

[0011] S5. For the offspring population, a first temporary archive is generated using a grid-based optimization method, a second temporary archive is generated using clustering-based crowding distance, and a third temporary archive is generated using a neighborhood dominance-based update method. The first temporary archive, the second temporary archive, and the third temporary archive are then merged to obtain a merged archive.

[0012] S6. Perform environmental selection on the merged files to obtain a new population, calculate the state of the new population, and calculate the reward value based on the difference between the state of the new population and the state of the offspring population to obtain a state-reward value pair;

[0013] S7. Based on the state-reward pairs, update the deep Q-network once every 50 iterations;

[0014] S8. Recompare the counter with the stall check cycle. If the counter has not reached the stall check cycle, re-execute S4-S7. If the counter has reached the stall check cycle, output the scheduling result. The scheduling result includes: the latest time for all jobs to be completed, the standard deviation of machine load, the total energy consumption of all machine processing processes, and the machine number assigned to each process.

[0015] Preferably, in step S2, the formula for calculating the state of the initial population is:

[0016]

[0017] in, Con Indicates the convergence of the population in the target space. Div This represents the diversity of a population in the decision-making space;

[0018] The convergence of the population in the target space is:

[0019]

[0020] in, N Indicates population size, Represents individuals within a neighborhood. B ( i , j ) represents an individual i Dominate j , iIndicates the first in the population i Individual, j Indicates the first in the population j Individual;

[0021] The diversity of the population in the decision space is:

[0022]

[0023]

[0024] in, dis Indicates crowded distance, t Indicates the number of targets to be optimized. k This indicates the target being optimized. Indicates the first k Individuals in the target i , Indicates the first k Individuals in the target j .

[0025] Preferably, in step S2, the method for constructing the deep Q-network includes:

[0026] Estimating the action-value function using a deep neural network:

[0027]

[0028] in, r Indicates taking action a The reward received at that time Indicates the discount factor. Indicates the next state Of all possible actions, choose the one that makes Q The action of maximizing value Q value;

[0029] Construct a loss function based on the action value function:

[0030]

[0031] in, T This represents training data randomly sampled during experience replay. Indicates the network output. Indicates the network input. express Take action in a state of Q value;

[0032] The deep Q-network is obtained by training the Q-Network based on the loss function.

[0033] Preferably, in step S6, the method for obtaining the difference between the new population and the offspring population state includes:

[0034]

[0035] in, This indicates the convergence of the offspring population in the target space. This indicates the diversity of the offspring population in the decision space.

[0036] Preferably, in S8:

[0037] The latest time to complete all tasks is:

[0038]

[0039] in, Indicates the first v The completion time of each task, max represents v The longest time taken for any given task is the latest time to complete it.

[0040] The standard deviation of machine load is:

[0041]

[0042]

[0043]

[0044] in, Z Indicates the total number of machines. Indicates the first z Total load of each machine Indicates average load. Indicates the first b The selected machine number is manipulated. Indicates the processing time after selecting a machine;

[0045] The total energy consumption of all machine processing processes is:

[0046]

[0047] in, D Indicates the total number of operands. α Indicates the energy consumption coefficient. Indicates the first b The processing time of the selected machine for each operation.

[0048] The present invention also provides a workshop production scheduling system based on deep reinforcement learning and multimodal multi-objective computation. The scheduling system applies the above-mentioned scheduling method and includes: a data acquisition module, a population state calculation module, a periodic comparison module, a offspring population generation module, an archive merging module, a new population calculation module, a network update module, and a comparison output module.

[0049] The data acquisition module is used to acquire the population size N, the objective function, and the maximum number of iterations. The objective function includes: job completion time, machine load, and total energy consumption.

[0050] The population state calculation module randomly generates an initial population based on the population size and the objective function, defines the state of the initial population, and constructs a deep Q-network.

[0051] The cycle comparison module is used to set a counter and compare the counter with the stall check cycle. If the counter reaches the stall check cycle, the scheduling result is output. If the counter does not reach the stall check cycle, the subsequent process continues.

[0052] The offspring population generation module is used to adaptively select operators from the initial population using the deep Q-network to generate an offspring population;

[0053] The file merging module is used to generate a first temporary file for the offspring population using a grid-based optimization method, a second temporary file using a clustering-based crowding distance method, and a third temporary file using a neighborhood dominance-based update method. The first temporary file, the second temporary file, and the third temporary file are then merged to obtain a merged file.

[0054] The new population calculation module is used to perform environmental selection on the merged files to obtain a new population, calculate the state of the new population, and calculate the reward value based on the difference between the state of the new population and the state of the offspring population to obtain a state-reward value pair.

[0055] The network update module updates the deep Q-network every 50 iterations based on the state-reward value pair.

[0056] The comparison output module is used to recompare the counter with the stagnation check period. If the counter has not reached the stagnation check period, it re-enters the offspring population generation module and executes the subsequent process. If the counter reaches the stagnation check period, it outputs the scheduling result, which includes: the latest time for all jobs to be completed, the standard deviation of machine load, the total energy consumption of all machine processing processes, and the machine number assigned to each process.

[0057] Preferably, in the population state calculation module, the formula for calculating the state of the initial population is:

[0058]

[0059] in, Con Indicates the convergence of the population in the target space. Div This represents the diversity of a population in the decision-making space;

[0060] The convergence of the population in the target space is:

[0061]

[0062] in, N Indicates population size, Represents individuals within a neighborhood. B ( i , j ) represents an individual i Dominate j , i Indicates the first in the population i Individual, j Indicates the first in the population j Individual;

[0063] The diversity of the population in the decision space is:

[0064]

[0065]

[0066] in, dis Indicates crowded distance, t Indicates the number of targets to be optimized. k This indicates the target being optimized. Indicates the first k Individuals in the target i , Indicates the first k Individuals in the target j .

[0067] Preferably, in the population state calculation module, the process of constructing the deep Q-network includes:

[0068] Estimating the action-value function using a deep neural network:

[0069]

[0070] in, r Indicates taking action a The reward received at that time Indicates the discount factor. Indicates the next state Of all possible actions, choose the one that makes Q The action of maximizing value Q value;

[0071] Construct a loss function based on the action value function:

[0072]

[0073] in, T This represents training data randomly sampled during experience replay. Indicates the network output. Indicates the network input. express Take action in a state of Q value;

[0074] The deep Q-network is obtained by training the Q-Network based on the loss function.

[0075] Preferably, in the new population calculation module, the process for obtaining the difference between the new population and the offspring population state includes:

[0076]

[0077] in, This indicates the convergence of the offspring population in the target space. This indicates the diversity of the offspring population in the decision space.

[0078] Preferably, in the comparison output module:

[0079] The latest time to complete all tasks is:

[0080]

[0081] in, Indicates the first v The completion time of each task, max represents v The longest time taken for any given task is the latest time to complete it.

[0082] The standard deviation of machine load is:

[0083]

[0084]

[0085]

[0086] in, Z Indicates the total number of machines. Indicates the first zTotal load of each machine Indicates average load. Indicates the first b The selected machine number is manipulated. Indicates the processing time after selecting a machine;

[0087] The total energy consumption of all machine processing processes is:

[0088]

[0089] in, D Indicates the total number of operands. α Indicates the energy consumption coefficient. Indicates the first b The processing time of the selected machine for each operation.

[0090] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0091] (1) This invention proposes an improved deep reinforcement learning model to solve the operator selection problem in multimodal optimization problems (MMOPs). In this framework, a deep Q-network is used to evaluate the Q-value of the action in the current state, and the operator is adaptively selected based on the Q-value. This approach significantly enhances the population's ability to balance exploration and exploitation.

[0092] (2) This invention proposes an improved sorting method for population update. The convergence performance is evaluated by calculating the number of dominant individuals in the neighborhood of each individual. The convergence quality is inversely proportional to the number of dominant individuals. The fewer dominant individuals, the higher the convergence quality. The algorithm prioritizes retaining the top N individuals with high convergence quality, thereby improving the convergence of the population in the target space.

[0093] (3) This invention employs a serial-parallel mechanism to enhance population diversity, where CSCD and ISM run serially during evolution and are executed in parallel with GrEA. This mechanism effectively improves the computational efficiency of the algorithm. By generating more candidate solutions, this strategy can effectively prevent the algorithm from converging prematurely into suboptimal regions, thereby significantly improving population diversity. Attached Figure Description

[0094] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0095] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0096] Figure 2 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0097] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0098] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0099] Example 1

[0100] In this embodiment, as Figure 1 , Figure 2 As shown, a workshop production scheduling method based on deep reinforcement learning and multimodal multi-objective computation includes the following steps:

[0101] S1. Obtain the population size N, objective function, and maximum number of iterations. The objective function includes: job completion time, machine load, and total energy consumption. In this embodiment, the maximum number of iterations is the algorithm termination condition. When the algorithm reaches the maximum number of iterations, it automatically terminates; otherwise, it continues to run.

[0102] S2. Randomly generate an initial population based on the population size and objective function, define the state of the initial population, and construct a deep Q-network.

[0103] In S2, the formula for calculating the initial population state is:

[0104]

[0105] in, Con Indicates the convergence of the population in the target space. Div This represents the diversity of a population in the decision-making space;

[0106] The convergence of the population in the target space is:

[0107]

[0108] in, N Indicates population size, Represents individuals within a neighborhood. B ( i , j ) represents an individual i Dominate jWhen individuals within a population i Dominant Individual j If the value is 1, then the value is 1; otherwise, the value is 0. i Indicates the first in the population i Individual, j Indicates the first in the population j Individual; the formula for calculating the neighborhood is as follows:

[0109]

[0110] in, M Indicates the number of decision variables. and They represent the first m The coordinates of the smallest and largest individual variables among the decision variables. η This represents an empirical constant used to control the average length of the neighborhood.

[0111] The diversity of a population in the target space is based on the crowding distance, which is calculated using the following formula:

[0112]

[0113] in, dis Indicates crowded distance, t Indicates the number of targets to be optimized. k This indicates the target being optimized. Indicates the first k Individuals in the target i , Indicates the first k Individuals in the target j The diversity of a population in the decision space is:

[0114] .

[0115] Before building a deep Q-network, it is essential to ensure that the Experience Playback (EP) meets the required size threshold, as sufficient training data is crucial for deep Q-network optimization. If the EP fails to reach the specified size, operators will be randomly selected for use by the current population to ensure a comprehensive exploration of the solution space. Conversely, when the EP meets the training criteria but the deep Q-network is not initialized, the deep Q-network must be built and trained first to enable intelligent operator selection.

[0116] In S2, the methods for constructing deep Q-networks include:

[0117] Deep neural networks are used to estimate action-value functions, thus enabling more efficient handling of complex high-dimensional state spaces. The action-value function, also often called the Q-value function, is used to evaluate the expected cumulative reward obtained by taking an action in a given state. The formula for calculating the Q-value function is as follows:

[0118]

[0119] in, r Indicates taking action a The reward received at that time This represents a discount factor used to adjust the current value of future rewards; it typically takes a value between 0 and 1. Indicates the next state Of all possible actions, choose the one that makes Q The action of maximizing value Q The value represents the expected value of the future reward. The Q-Table is a traditional tool in reinforcement learning that records the reward for each possible action in each state. However, the Q-Table performs poorly with large state and action spaces and struggles to handle the challenges of high-dimensional data. In this embodiment, a loss function is constructed based on the action-value function:

[0120]

[0121] in, T Indicates experience replay ( EP The training data is randomly sampled from the dataset, where EP represents the dataset that records the historical data of the training network. Indicates the network output. Indicates the network input. express Take action in a state of Q Value; a deep Q-network is obtained by training a Q-Network based on the loss function.

[0122] S3. Set a counter and compare it with the stall check period. If the counter has not reached the stall check period, execute S4-S7. If the counter has reached the stall check period, output the scheduling result.

[0123] S4. Use a deep Q-network to adaptively select operators for the initial population to generate the offspring population.

[0124] S5. For the offspring population, a first temporary archive is generated using a grid-based optimization method, a second temporary archive is generated using a clustering-based crowding distance method, and a third temporary archive is generated using a neighborhood dominance-based update method. The first, second, and third temporary archives are then merged to obtain the merged archive.

[0125] In this embodiment, the process begins by generating a duplicate population (offspring population) from the original population. The main population is generated using CSCD-based progeny generation, and the resulting solution is stored in an external temporary archive (first temporary archive) and subsequently merged into the newly formed population. This population is further processed by ISM to generate a third temporary archive, which is also merged into the new population. Simultaneously, the offspring population generates progeny using GrEA, stores it in a second temporary archive, and is ultimately transferred to the new population.

[0126] When using a neighborhood-dominance-based ranking method to generate the third temporary archive, the evaluation of local convergence quality is the core of this ranking method. In the process of calculating local convergence quality, it is first necessary to evaluate the dominance relationship between an individual and other individuals in its neighborhood. This process helps to determine the individual's competitiveness and fitness in the current population. The method for calculating local convergence quality can be described as follows:

[0127]

[0128]

[0129] in, C ( i ) represents the local convergence quality. This represents the number of disposable individuals within the neighborhood. h Represents the first in the neighborhood h Individual, and This represents the weight value.

[0130] S6. Perform environmental selection on the merged files to obtain a new population, calculate the state of the new population, and calculate the reward value based on the difference between the state of the new population and the state of the offspring population to obtain a state-reward value pair.

[0131] After merging the files, an environment selection process is performed, retaining the top N high-quality solutions from the new population. Subsequently, the algorithm quantifies the system state of the new population and calculates the reward value based on the state difference between the original and new populations. These state-reward pairs (including the state of the original population, the state of the new population, and the calculated reward) are recorded as empirical data and stored in the EP buffer.

[0132] In S6, the methods for obtaining the difference between the state of the new population and the offspring population include:

[0133]

[0134] in, This indicates the convergence of the offspring population in the target space. This indicates the diversity of the offspring population in the decision space.

[0135] S7. Based on state-reward pairs, the deep Q-network is updated once every 50 iterations.

[0136] In this embodiment, the deep Q-network performs a plan update every 50 iterations. After the update, the termination condition is evaluated—if it is met, the optimization process ends; otherwise, subsequent iterations are started until the convergence criterion is reached. This cyclical process ensures that the solution quality is continuously optimized while maintaining population diversity, supported by the parallel archiving strategy.

[0137] S8. Recompare the counter with the stall check cycle. If the counter has not reached the stall check cycle, re-execute S4-S7. If the counter has reached the stall check cycle, output the scheduling result, which includes: the latest time for all jobs to complete, the standard deviation of machine load, the total energy consumption of all machine processing processes, and the machine number assigned to each process.

[0138] In S8: The latest time for all tasks to complete is:

[0139]

[0140] in, Indicates the first v The completion time of each task, max represents v The longest time taken for any given task is the latest completion time; the standard deviation of machine load is:

[0141]

[0142]

[0143]

[0144] in, Z Indicates the total number of machines. Indicates the first z Total load of each machine Indicates average load. Indicates the first b The selected machine number is manipulated. This indicates the processing time after selecting a machine; the total energy consumption for all machine processing processes is:

[0145]

[0146] in, D Indicates the total number of operands. α Indicates the energy consumption coefficient. Indicates the first b The processing time of the selected machine for each operation.

[0147] Example 2

[0148] In this embodiment, to verify the effectiveness of the proposed algorithm, DRLMMEA is compared with MO_Ring_PSO_SCD, TriMOEATAR, MMEAWI, CMMO, and MMEAPSL on the CEC2019 benchmark set. The population size of both the proposed algorithm and the comparison algorithms is 100, and the maximum number of evaluations is D·5000, where D is the dimension of the problem. All other parameters of the comparison algorithms are the original parameters. The experimental environment is: Windows 10 64-bit, Intel i5 processor, 32GB RAM. The simulation software version is Matlab 2022b.

[0149] To reduce experimental error, six algorithms were run independently, and their performance was evaluated 21 times. The average of the results from these 21 runs was used for comparison. The Wilcoxon rank-sum test was used to determine the number of dominant functions of the proposed algorithm on the test set compared to the comparison algorithms, thus quantifying the overall solution accuracy of the proposed algorithm.

[0150] The statistical results of the experiments show that this invention successfully obtained 15 optimal IGD values. This is mainly attributed to the algorithm's ability to generate solution sets with good coverage, uniform distribution, and diversity in the target space, thereby effectively approximating the true Pareto front and achieving an optimal balance among multiple objectives. This invention also achieved the most optimal PSP values ​​(14) in the PSP performance index. This is primarily due to the synergistic effect of deep reinforcement learning and the neighborhood dominance ranking mechanism: deep reinforcement learning enhances the algorithm's exploration ability, enabling it to discover more effective solutions; while the neighborhood dominance ranking mechanism ensures that these solutions maintain better distribution diversity in the decision space. The combination of these two mechanisms allows DRLMMEA to simultaneously optimize the convergence and distribution of solutions.

[0151] To further analyze the convergence of this invention, this section discusses the convergence curves of the algorithm on several problem functions. This invention exhibits a significant decision space convergence advantage on the four MMF test functions, an advantage reflected in the convergence curves of the indices. Convergence analysis based on the IGD index shows that this invention demonstrates excellent convergence performance on the four MMF test functions. Specifically, the IGD value shows a continuous decreasing trend with increasing evolutionary iterations, confirming the algorithm's excellent convergence characteristics in the target space. The convergence curves of the PSP index show that the convergence curves of this invention on the four MMF test functions gradually rise with the evolutionary process, exhibiting excellent convergence performance.

[0152] This invention also conducted a Friedman test, ensuring the statistical reliability of the conclusions through statistical analysis of multiple experimental results. It should be noted that a higher PSP value results in a lower Friedman test ranking, while a lower IGD value results in a higher ranking. To ensure consistency, this embodiment uses IGD and 1 / PSP for the Friedman test; a lower ranking value indicates better algorithm performance. The experimental results show that the proposed DRLMMEA achieves the best ranking across all metrics: IGD (1.82) and 1 / PSP (2.15), with an average ranking (1.99) significantly better than all compared algorithms. Furthermore, the p-values ​​for each metric are all below the significance level, indicating significant differences in performance among different algorithms on these metrics. In conclusion, DRLMMEA significantly outperforms the compared algorithms in both IGD and 1 / PSP metrics.

[0153] Example 3

[0154] In this embodiment, a workshop production scheduling system based on deep reinforcement learning and multimodal multi-objective computation includes: a data acquisition module, a population state calculation module, a periodic comparison module, a offspring population generation module, an archive merging module, a new population calculation module, a network update module, and a comparison output module.

[0155] The data acquisition module is used to obtain the population size N, the objective function, and the maximum number of iterations. The objective function includes: job completion time, machine load, and total energy consumption.

[0156] The population state calculation module randomly generates an initial population based on the population size and objective function, defines the state of the initial population, and constructs a deep Q-network.

[0157] In the population state calculation module, the formula for calculating the initial population state is:

[0158]

[0159] in, Con Indicates the convergence of the population in the target space. Div Let represent the diversity of the population in the decision space; the convergence of the population in the goal space is:

[0160]

[0161] in, N Indicates population size, Represents individuals within a neighborhood. B ( i , j ) represents an individual i Dominate j , i Indicates the first in the populationi Individual, j Indicates the first in the population j Individual; the diversity of a population in the decision space is:

[0162]

[0163]

[0164] in, dis Indicates crowded distance, t Indicates the number of targets to be optimized. k This indicates the target being optimized. Indicates the first k Individuals in the target i , Indicates the first k Individuals in the target j .

[0165] In the population state calculation module, the process of constructing a deep Q-network includes: estimating the action-value function using a deep neural network.

[0166]

[0167] in, r Indicates taking action a The reward received at that time Indicates the discount factor. Indicates the next state Of all possible actions, choose the one that makes Q The action of maximizing value Q Value; constructing a loss function based on the action-value function:

[0168]

[0169] in, T This represents training data randomly sampled during experience replay. Indicates the network output. Indicates the network input. express Take action in a state of Q Value; a deep Q-network is obtained by training a Q-Network based on the loss function.

[0170] The period comparison module is used to set a counter and compare the counter with the stall check period. If the counter reaches the stall check period, the scheduling result is output; if the counter does not reach the stall check period, the subsequent process continues.

[0171] The offspring population generation module is used to adaptively select operators from the initial population using a deep Q-network to generate the offspring population.

[0172] The file merging module is used to generate a first temporary file for the offspring population using a grid-based optimization method, a second temporary file using a clustering-based crowding distance method, and a third temporary file using a neighborhood dominance-based update method. The first, second, and third temporary files are then merged to obtain the merged file.

[0173] The new population calculation module is used to select an environment for the merged files, obtain a new population, calculate the state of the new population, and calculate the reward value based on the difference between the state of the new population and the offspring population, thus obtaining a state-reward value pair.

[0174] In the new population calculation module, the process for obtaining the difference between the state of the new population and the offspring population includes:

[0175]

[0176] in, This indicates the convergence of the offspring population in the target space. This indicates the diversity of the offspring population in the decision space.

[0177] The network update module updates the deep Q network every 50 iterations based on state-reward pairs.

[0178] The comparison output module is used to recompare the counter with the stall check period. If the counter has not reached the stall check period, it will re-enter the offspring population generation module and execute the subsequent process. If the counter has reached the stall check period, it will output the scheduling result, which includes: the latest time for all jobs to be completed, the standard deviation of machine load, the total energy consumption of all machine processing processes, and the machine number assigned to each process.

[0179] In the comparison output module: the latest time for all jobs to complete is:

[0180]

[0181] in, Indicates the first v The completion time of each task, max represents v The longest time taken for any given task is the latest completion time; the standard deviation of machine load is:

[0182]

[0183]

[0184]

[0185] in, Z Indicates the total number of machines. Indicates the first z Total load of each machine Indicates average load. Indicates the first b The selected machine number is manipulated. This indicates the processing time after selecting a machine; the total energy consumption for all machine processing processes is:

[0186]

[0187] in, D Indicates the total number of operands. α Indicates the energy consumption coefficient. Indicates the first b The processing time of the selected machine for each operation.

[0188] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A workshop production scheduling method based on deep reinforcement learning and multimodal multi-objective computation, characterized in that, Includes the following steps: S1. Obtain the population size N, objective function, and maximum number of iterations. The objective function includes: job completion time, machine load, and total energy consumption. S2. Based on the population size and the objective function, an initial population is randomly generated, the state of the initial population is defined, and a deep Q-network is constructed; S3. Set a counter and compare the counter with the stall check period. If the counter has not reached the stall check period, execute S4-S7. If the counter has reached the stall check period, output the scheduling result. S4. Use the deep Q-network adaptive selection operator to generate a sub-population from the initial population; S5. For the offspring population, a first temporary archive is generated using a grid-based optimization method, a second temporary archive is generated using clustering-based crowding distance, and a third temporary archive is generated using a neighborhood dominance-based update method. The first, second, and third temporary archives are then merged to obtain a merged archive. Specifically, the offspring population is generated from the initial population, and the main population is generated using CSCD-based progeny generation. The resulting solution is stored in the external archive, the first temporary archive, and subsequently merged into the newly formed population. This population is further processed by ISM to generate a third temporary archive, which is also merged into the new population. Simultaneously, the offspring population generates progeny using GrEA, stores it in the second temporary archive, and finally transfers it to the new population. S6. Perform environmental selection on the merged files to obtain a new population, calculate the state of the new population, and calculate the reward value based on the difference between the state of the new population and the state of the offspring population to obtain a state-reward value pair; S7. Based on the state-reward pairs, update the deep Q-network once every 50 iterations; S8. Recompare the counter with the stall check cycle. If the counter has not reached the stall check cycle, re-execute S4-S7. If the counter has reached the stall check cycle, output the scheduling result. The scheduling result includes: the latest time for all jobs to be completed, the standard deviation of machine load, the total energy consumption of all machine processing processes, and the machine number assigned to each process.

2. The workshop production scheduling method based on deep reinforcement learning and multimodal multi-objective computation according to claim 1, characterized in that, In step S2, the formula for calculating the state of the initial population is: in, Con Indicates the convergence of the population in the target space. Div This represents the diversity of a population in the decision-making space; The convergence of the population in the target space is: in, N Indicates population size, Represents individuals within a neighborhood. B ( i , j ) represents an individual i Dominate j When individuals within a population i Dominant Individual j If the value is 1, then the value is 1; otherwise, the value is 0. i Indicates the first in the population i Individual, j Indicates the first in the population j Individual; The diversity of the population in the decision space is: in, dis Indicates crowded distance, t Indicates the number of targets to be optimized. k This indicates the target being optimized. Indicates the first k Individuals in the target i , Indicates the first k Individuals in the target j .

3. The workshop production scheduling method based on deep reinforcement learning and multimodal multi-objective computation according to claim 1, characterized in that, In step S2, the method for constructing the deep Q-network includes: Estimating the action-value function using a deep neural network: in, r Indicates taking an action a The reward received at that time Indicates the discount factor. Indicates the next state Of all possible actions, choose the one that makes Q The action of maximizing value Q value; Construct a loss function based on the action value function: in, T This represents training data randomly sampled during experience replay. Indicates the network output. Indicates the network input. express Take action in a state of Q value; The deep Q-network is obtained by training the Q-Network based on the loss function.

4. The workshop production scheduling method based on deep reinforcement learning and multimodal multi-objective computation according to claim 2, characterized in that, In step S6, the method for obtaining the difference between the new population and the offspring population state includes: in, This indicates the convergence of the offspring population in the target space. This indicates the diversity of the offspring population in the decision space.

5. The workshop production scheduling method based on deep reinforcement learning and multimodal multi-objective computation according to claim 1, characterized in that, In S8: The latest time to complete all tasks is: in, Indicates the first v The completion time of each task, max represents v The longest time in a single task; The standard deviation of machine load is: in, Z Indicates the total number of machines. Indicates the first z Total load of each machine Indicates average load. Indicates the first b The selected machine number is manipulated. Indicates the processing time after selecting a machine; The total energy consumption of all machine processing processes is: in, D Indicates the total number of operands. α Indicates the energy consumption coefficient. Indicates the first b The processing time of the selected machine for each operation.

6. A workshop production scheduling system based on deep reinforcement learning and multimodal multi-objective computation, wherein the scheduling system applies the scheduling method according to any one of claims 1-5, characterized in that, include: The system includes a data acquisition module, a population state calculation module, a periodic comparison module, a offspring population generation module, a file merging module, a new population calculation module, a network update module, and a comparison output module. The data acquisition module is used to acquire the population size N, the objective function, and the maximum number of iterations. The objective function includes: job completion time, machine load, and total energy consumption. The population state calculation module randomly generates an initial population based on the population size and the objective function, defines the state of the initial population, and constructs a deep Q-network. The cycle comparison module is used to set a counter and compare the counter with the stall check cycle. If the counter reaches the stall check cycle, the scheduling result is output. If the counter does not reach the stall check cycle, the subsequent process continues. The offspring population generation module is used to adaptively select operators from the initial population using the deep Q-network to generate an offspring population; The file merging module is used to generate a first temporary file from the offspring population using a grid-based optimization method, a second temporary file using clustering-based crowding distance, and a third temporary file using a neighborhood dominance-based update method. The first, second, and third temporary files are then merged to obtain a merged file. Specifically, the method involves generating an offspring population from the initial population, generating a primary population using CSCD-based progeny generation, storing the resulting solution in the external first temporary file, and subsequently merging it into the newly formed population. This population is further processed by ISM to generate a third temporary file, which is also merged into the new population. Simultaneously, the offspring population generates progeny using GrEA, stores it in the second temporary file, and finally transfers it to the new population. The new population calculation module is used to perform environmental selection on the merged files to obtain a new population, calculate the state of the new population, and calculate the reward value based on the difference between the state of the new population and the state of the offspring population to obtain a state-reward value pair. The network update module updates the deep Q-network every 50 iterations based on the state-reward value pair. The comparison output module is used to recompare the counter with the stagnation check period. If the counter has not reached the stagnation check period, it re-enters the offspring population generation module and executes the subsequent process. If the counter reaches the stagnation check period, it outputs the scheduling result, which includes: the latest time for all jobs to be completed, the standard deviation of machine load, the total energy consumption of all machine processing processes, and the machine number assigned to each process.

7. The workshop production scheduling system based on deep reinforcement learning and multimodal multi-objective computation according to claim 6, characterized in that, In the population state calculation module, the formula for calculating the state of the initial population is: in, Con Indicates the convergence of the population in the target space. Div This represents the diversity of a population in the decision-making space; The convergence of the population in the target space is: in, N Indicates population size, Represents individuals within a neighborhood. B ( i , j ) represents an individual i Dominate j When individuals within a population i Dominant Individual j If the value is 1, then the value is 1; otherwise, the value is 0. i Indicates the first in the population i Individual, j Indicates the first in the population j Individual; The diversity of the population in the decision space is: in, dis Indicates crowded distance, t Indicates the number of targets to be optimized. k This indicates the target being optimized. Indicates the first k Individuals in the target i , Indicates the first k Individuals in the target j .

8. The workshop production scheduling system based on deep reinforcement learning and multimodal multi-objective computation according to claim 6, characterized in that, The process of constructing the deep Q-network in the population state calculation module includes: Estimating the action-value function using a deep neural network: in, r Indicates taking an action a The reward received at that time Indicates the discount factor. Indicates the next state Of all possible actions, choose the one that makes Q The action of maximizing value Q value; Construct a loss function based on the action value function: in, T This represents training data randomly sampled during experience replay. Indicates the network output. Indicates the network input. express Take action in a state of Q value; The deep Q-network is obtained by training the Q-Network based on the loss function.

9. The workshop production scheduling system based on deep reinforcement learning and multimodal multi-objective computation according to claim 7, characterized in that, The process for obtaining the difference between the new population and the offspring population state in the new population calculation module includes: in, This indicates the convergence of the offspring population in the target space. This indicates the diversity of the offspring population in the decision space.

10. The workshop production scheduling system based on deep reinforcement learning and multimodal multi-objective computation according to claim 6, characterized in that, In the comparison output module: The latest time to complete all tasks is: in, Indicates the first v The completion time of each task, max represents v The longest time in a single task; The standard deviation of machine load is: in, Z Indicates the total number of machines. Indicates the first z Total load of each machine Indicates average load. Indicates the first b The selected machine number is manipulated. Indicates the processing time after selecting a machine; The total energy consumption of all machine processing processes is: in, D Indicates the total number of operands. α Indicates the energy consumption coefficient. Indicates the first b The processing time of the selected machine for each operation.

Images