Production scheduling optimization method based on deep reinforcement learning and genetic algorithm fusion

Abstract

The invention provides a production scheduling optimization method based on deep reinforcement learning and genetic algorithm fusion. The method comprises a state sensing module, an initial scheduling generation module, a genetic algorithm optimization engine and a hierarchical DRL model. The initial scheduling generation module constructs an initial population, inputs the initial population into a genetic algorithm optimization engine to carry out multi-target fitness evaluation, executes selection and crossover operations, and generates an optimized population; and the state sensing module extracts scheduling state features of the optimized population in real time, inputs the scheduling state features into the layered DRL model, and outputs high-value variation actions. According to the method, a hierarchical DRL model is constructed, and an intelligent variation strategy of data driving and environment perception is provided in a GA iteration process; the search efficiency and the solution quality are obviously improved, and efficient and high-quality production scheduling solution is realized; according to the method, the intellectualization and autonomous evolution capability of scheduling optimization are realized, the dependence on expert experience and rules is reduced, the universality is high, and the method is suitable for different production line configurations and task fluctuations and can be migrated to other evolutionary algorithms.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of production scheduling, and particularly relates to a production scheduling optimization method based on deep reinforcement learning and genetic algorithm fusion. BACKGROUND

[0002] In industrial manufacturing, production scheduling refers to the process of allocating production tasks to production resources. The goal is to reasonably arrange the processing order of tasks and resource allocation under the conditions of meeting process constraints, resource capacity, delivery period and other multiple restrictions, to minimize the production cycle, maximize equipment utilization and the like, so as to optimize the production capacity and improve the production efficiency. With the development of manufacturing towards flexibility, customization and intelligence, the complexity of scheduling problems has risen sharply. Multi-machine, multi-task, multi-material and other multi-dimensional constraints make the scheduling problem have high combinatorial complexity. Traditional methods based on rules, mathematical programming or single heuristic algorithm have been difficult to cope with real challenges such as frequent dynamic disturbance, multi-objective trade-off and large-scale combinatorial optimization, and it is difficult to obtain high-quality solutions within a limited time.

[0003] As a classic meta-heuristic search method, genetic algorithm has been widely used in production scheduling due to its strong global search ability, easy handling of discrete variables and nonlinear constraints. However, the mutation and crossover operations of traditional GA rely on random mechanism or fixed rules, and the mutation is highly blind, lacking effective guidance to the search direction, which easily leads to low search efficiency, slow convergence and even falling into local optimal solution.

[0004] On the other hand, deep reinforcement learning performs well in sequence decision and policy learning, and can learn optimal behavior policy through interaction with the environment. In recent years, DRL has been tried to be used to directly solve scheduling problems, but when the state space or action space is large, the training stability is poor, the generalization ability is weak, and the feasibility of the solution is difficult to guarantee, especially in large-scale industrial scenarios, which is limited in performance and difficult to handle large-scale combinatorial optimization problems.

[0005] Therefore, combining deep reinforcement learning with genetic algorithms has become an important technical path to improve the performance of intelligent scheduling systems. The patent document "Production line scheduling method based on deep reinforcement learning and multi-objective optimization" (CN117540990A) proposes to establish a low-carbon multi-objective model containing customer satisfaction, energy consumption, and bottleneck machine load, and uses a multi-objective genetic algorithm to solve it. In the solving process, a deep reinforcement learning method based on Q-learning is used to adaptively optimize parameters such as the crossover probability (Pc) and mutation probability (Pm) of the genetic algorithm to improve the performance of the algorithm. However, this technology only uses DRL to dynamically adjust the control parameters (such as Pc and Pm) of the genetic algorithm, and does not directly participate in the exploration and development decisions of the solution space. The patent document "Non-related parallel machine scheduling method based on deep reinforcement learning" (CN120450376A) discloses a deep reinforcement learning algorithm for solving similar problems in the field of manufacturing production scheduling by analyzing the data model of the non-related parallel machine scheduling problem. The mutation operation of these two schemes is still based on traditional operators (such as exchange and insertion) random or rule-based operations, which cannot guarantee the effectiveness and directionality of the mutation operation, and the degree of intelligence is limited.

[0006] The patent document "Batch scheduling optimization method and process based on deep reinforcement learning and genetic algorithm" (CN112488315A) discloses using a pointer network combined with an Actor-Critic algorithm to train a deep reinforcement learning model; then, using the solution generated by the DRL model to optimize the initial population of the genetic algorithm (i.e. combining the DRL solution with randomly generated individuals and selecting them according to fitness); finally, using an improved genetic algorithm (i.e. an improved multi-point crossover method) to further solve the problem. However, using a pointer network as the Actor network of DRL forms a sequence-to-sequence end-to-end model that directly maps a sequence of job information (input) to a complete batch scheduling scheme sequence (output). This "black box" decision-making process is difficult to train, has poor interpretability, and has limited generalization ability. It only solves the single-machine single-process batch problem, and its DRL model is independent of the search process of GA. Similarly, the patent document "Chip production logistics optimization scheduling method and system based on deep reinforcement learning" (CN119378953A) uses an end-to-end deep reinforcement learning model to directly solve the scheduling problem, which has a complex model structure and is not easy to train.

[0007] In summary, existing hybrid methods lack dynamic strategy learning capabilities, and most methods are only used to speed up RL training speed or optimize GA parameters, without deep intervention in the search process.

[0008] Based on this idea: using the "policy learning" ability of DRL to enhance the "intelligent search" ability of GA, a production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm is provided, which not only retains the robustness of GA in global optimization, but also introduces the intelligence of DRL in local search guidance, realizes the efficient cooperation of the two, and significantly improves the solution quality and response speed of the scheduling scheme under complex production environment. SUMMARY

[0009] In view of the defects in the prior art, the purpose of the present application is to provide a production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm.

[0010] According to the production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm provided by the present application, the method comprises a training step and an inference optimization step. The training step comprises: Step S1, based on actual production line information, constructing a scheduling simulation environment and pre-training a hierarchical DRL model; Step S2, further training the pre-trained hierarchical DRL model to output a final hierarchical DRL model; The inference optimization step comprises: Step S3, constructing an initial population by an initial scheduling generation module; Step S4, performing multi-objective fitness evaluation on the current population, executing selection and crossover operations, and generating an optimized population; Step S5, a state perception module real-time extracts scheduling state features of the optimized population inputting the hierarchical DRL model, outputs high-value mutation actions and executes them; Step S6, generating a next generation population and judging the termination condition.

[0011] Preferably, the scheduling simulation environment simulates the execution of mutation operations and evaluates the fitness changes before and after mutation based on the process constraints, resource capacity, task path and task characteristics of the actual production line.

[0012] In the step S1, a traditional genetic algorithm or heuristic algorithm is used to solve a plurality of scheduling problem instances, and the state of each generation population and the effective mutation action leading to a significant improvement in fitness in the subsequent search process are recorded as pre-training data, and the model is pre-trained according to the pre-training data.

[0013] In the step S2, the current scheduling state is input into the pre-trained hierarchical DRL model to output a mutation action, the mutation action is applied to the scheduling simulation environment to obtain the fitness change, and the network parameters of the hierarchical DRL model are updated according to the fitness change.

[0014] The hierarchical DRL model includes a high-level policy network and a low-level action network.

[0015] Input the global scheduling status features into the high-level policy network to determine whether to perform mutation and the type of mutation to be performed; The underlying action network generates specific parameters for the corresponding mutation action based on the mutation type.

[0016] Preferably, the hierarchical DRL model sets different mutation types and corresponding specific mutation actions.

[0017] The high-level policy network determines whether to perform mutation. If mutation is not performed, the underlying action network is not activated and the result is output directly. If mutation is performed, the mutation type to be performed is selected.

[0018] The underlying action network determines the corresponding mutation action based on the selected mutation type, decomposes it into multiple ordered sub-actions in a structured manner, generates the corresponding specific parameters, and outputs them level by level.

[0019] The sub-action accepts the sampling result of the previous sub-action as the input of the next network. Combined with the set constraint checking mechanism, if the action is invalid, the resampling mechanism is triggered or the default action is returned. If the action is valid, the next sub-action is judged, and all sub-actions are traversed.

[0020] Preferably, the initial population is a set of feasible initial solutions based on expert rules or randomly generated.

[0021] In step S5, the state perception module extracts the scheduling state features of individuals in the optimized population in real time and inputs them into the hierarchical DRL model. The hierarchical DRL model outputs high-value mutation actions and performs feasibility checks. If the actions are valid, they are executed. If they are invalid, local adjustments are made or the samples are resampled and then input into the hierarchical DRL model for mutation.

[0022] The termination condition is set according to the actual needs of real-time production status, and the judgment includes: If the set termination condition is met, the iteration is terminated and mutation is stopped; if the set termination condition is not met, step S4 is executed.

[0023] Preferably, the method further includes step S7: We continuously collect data on the effects of actual scheduling results and the mutation actions output by the hierarchical DRL model, evaluate the performance of each generation of scheduling schemes, and feed the results back to the hierarchical DRL model for online fine-tuning or periodic retraining.

[0024] The production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm provided by the present invention includes: a state-aware module, an initial schedule generation module, a genetic algorithm optimization engine, and a hierarchical DRL model; The initial scheduling generation module constructs the initial population, inputs it into the genetic algorithm optimization engine for multi-objective fitness evaluation, performs selection and crossover operations, and generates an optimized population; The state awareness module extracts the scheduling state features of the optimized population in real time and inputs them into the hierarchical DRL model. It outputs high-value mutation actions, generates the next generation of the population, and determines the termination conditions.

[0025] Preferably, the hierarchical DRL model includes at least one sub-network, which is trained using a proximal policy optimization algorithm, and makes hierarchical decisions based on the current scheduling status, outputting specific mutation actions.

[0026] The sub-network adopts an independent parameter network structure and shares the underlying state encoder.

[0027] The reward function of the hierarchical DRL model is a percentage function of the fitness improvement before and after mutation relative to the fitness of the initial solution.

[0028] Preferably, the hierarchical DRL model includes a high-level policy network and a low-level action network, with different mutation types and corresponding specific mutation actions.

[0029] Input the global scheduling state features into the high-level policy network to determine whether to perform mutation and the type of mutation to be performed. If mutation is not performed, the underlying action network is not activated and the result is output directly. If mutation is performed, the type of mutation to be performed is selected.

[0030] The underlying action network determines the corresponding mutated action based on the mutation type, breaks it down into multiple ordered sub-actions in a structured manner, generates the corresponding specific parameters, and outputs them level by level.

[0031] The sub-action accepts the sampling result of the previous sub-action as the input of the next network. Combined with the set constraint checking mechanism, if the action is invalid, the resampling mechanism is triggered or the default action is returned. If the action is valid, the next sub-action is judged, and all sub-actions are traversed.

[0032] Preferably, the initial population is a set of feasible initial solutions based on expert rules or randomly generated.

[0033] The state awareness module extracts the scheduling state features of individuals in the optimized population in real time and inputs them into the hierarchical DRL model.

[0034] The hierarchical DRL model outputs high-value mutation actions and performs feasibility checks. If the actions are valid, they are executed. If they are invalid, local adjustments are made or the samples are resampled and then input into the hierarchical DRL model for mutation.

[0035] The termination condition is set according to the actual needs of real-time production status, and the judgment includes: If the set termination condition is met, the iteration terminates and mutation stops. If the set termination condition is not met, the genetic algorithm optimization engine performs a multi-objective fitness evaluation on the newly generated population.

[0036] Preferably, it also includes an evaluation and feedback module; The evaluation and feedback module continuously collects the actual scheduling results and the effect data of the variant actions output by the hierarchical DRL model, evaluates the performance of each generation of scheduling schemes, and feeds the results back to the hierarchical DRL model for online fine-tuning or periodic retraining.

[0037] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention uses a deep reinforcement learning model to intelligently guide the mutation process of the genetic algorithm, which significantly improves the search efficiency and solution quality, and achieves efficient and high-quality production scheduling solutions, with a 90% increase in convergence speed and a 10% improvement in solution performance.

[0038] 2. This invention constructs a hierarchical DRL model, which learns the mapping relationship from the current scheduling state to "effective mutation actions", thereby providing a data-driven and environment-aware intelligent mutation strategy during the GA iteration process.

[0039] 3. The mutation strategy learned by DRL in this invention has transferability and adaptability, realizes intelligent scheduling optimization and autonomous evolution capability, reduces dependence on expert experience and rules, has strong universality, is applicable to different production line configurations and task fluctuations, and can be transferred to other evolutionary algorithms. Attached Figure Description

[0040] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithms. Detailed Implementation

[0041] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0042] To address the complex scheduling problem involving multiple machines and processes, a collaborative optimization framework integrating Deep Reinforcement Learning (DRL) and Genetic Algorithm (GA) is proposed to solve production scheduling problems in complex industrial scenarios.

[0043] This invention provides a production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithms. By constructing a hierarchical DRL model, it learns the mapping relationship from the current scheduling state to "effective mutation actions," thereby providing a data-driven, environment-aware intelligent mutation strategy during the GA iteration process. Figure 1 For example, it includes training steps and inference optimization steps. It uses a deep reinforcement learning model to intelligently guide the mutation process of the genetic algorithm, thereby achieving efficient and high-quality production scheduling solutions.

[0044] The training steps utilize a deep reinforcement learning model to learn an efficient "mutation strategy" from scheduling data and environmental conditions (tasks, crews, materials, etc.), and dynamically guide the mutation operation of the solution during the genetic algorithm iteration process, thereby improving the intelligence and effectiveness of the search direction.

[0045] Specifically, deep reinforcement learning models are trained using actual production data from the manufacturing process or by generating a large number of typical scheduling scenario samples through simulation environments. This includes: Step S1: Build a scheduling simulation environment: Based on information such as process constraints, resource capabilities, and task characteristics of the actual production line, a simulation environment is constructed to simulate the scheduling execution process.

[0046] The simulation environment supports rapid evaluation of scheduling schemes and serves as an interaction platform for the DRL agent. Based on actual process constraints, resource capabilities, task paths, and other rules, it simulates the mutation operation and quickly evaluates the fitness changes before and after mutation. These fitness changes are also fed back to the DRL model as reward signals to update its network parameters.

[0047] The network parameters are the parameters of all relevant networks in the hierarchical DRL model.

[0048] In more preferred examples, if a deviation is found between the simulation environment and the real production line (e.g., some constraints are not modeled or parameters are inaccurate), the simulation environment is calibrated or enhanced based on real operating data, and then the model is retrained or fine-tuned.

[0049] The model training process is highly dependent on the state, action verification, reward and dynamic simulation capabilities provided by the simulation environment. Therefore, a high-fidelity simulation environment is one of the key prerequisites.

[0050] Step S2, Pre-training: To accelerate the training process, we first use traditional genetic algorithms or heuristic algorithms to solve multiple scheduling problem instances. During the search process, we record the state of each generation of the population (including individual scheduling schemes, task-resource mapping relationships, equipment load, etc.) and the effective "mutation actions" that lead to a significant improvement in fitness. Using this data, we pre-train the model so that it has better prior knowledge.

[0051] Whether or not pre-training is performed mainly affects the training efficiency and stability of the model. Supervised or behavioral cloning pre-training of the DRL model can enable the model to have certain prior knowledge or reasonable behavior patterns before the formal reinforcement learning training begins, thereby reducing the blindness in the early stage of training and the sensitivity to hyperparameters, improving the stability and success rate of the training process. Under the same training resources, it is easier to achieve better final performance after pre-training.

[0052] Skipping pre-training often requires investing more computational resources and effort in extensive model tuning to achieve better performance.

[0053] Step S3, Hierarchical DRL Model Training: During the training phase, the current scheduling state is input into the DRL model, which outputs a mutation action. This action is then applied to the scheduling scheme in the simulation environment.

[0054] To improve the decision-making efficiency and action controllability of deep reinforcement learning in complex scheduling optimization, a hierarchical DRL model is adopted, which decomposes the learning process of intelligent mutation strategy into two levels: high-level policy decision-making and low-level action execution, thereby achieving precise guidance of the mutation operation of genetic algorithm.

[0055] Specifically, the high-level policy network takes global scheduling state features as input and determines whether to perform mutation and which type of mutation to perform (mutation operation types can be "swap_two_tasks" to exchange the order of two tasks, "move_a_batch" to move a batch of consecutive tasks, etc.). The mutation type and the corresponding specific mutation action are pre-set in the hierarchical DRL model.

[0056] If the high-level policy network determines that mutation should not be performed, it will not activate the low-level action network (equivalent to the DRL smart mutation module being temporarily ineffective and ultimately not generating new individuals), and will directly output the result; if it determines that mutation should be performed, it will select the mutation type to be performed.

[0057] The underlying action network provides the corresponding mutation action based on the selected mutation type (the DRL module generates new individuals based on the mutation action). Regardless of the mutation type chosen by the higher-level network, the underlying action network is directly activated, but the instructions issued by the higher-level policy network to the underlying action network differ. For example, if the "move_a_batch" mutation type is selected, the underlying action network is activated to generate the parameters corresponding to "move_a_batch".

[0058] The underlying action network is responsible for generating the specific parameters corresponding to the mutation type after the higher-level network selects the mutation type.

[0059] The model decomposes complex combined actions into a series of ordered sub-decision steps, which are output step by step through a hierarchical policy network, improving training efficiency and action feasibility. Taking the "move_a_batch" mutation as an example, its action is structurally decomposed into the following four ordered parameters: head_id: Select a starting task in the current scheduling scheme as the head task of the batch to be moved; batch_choice: Based on head_id, determine the number of tasks to be selected consecutively (e.g., 1 to 3) to form a transferable task block (batch). target_mac_id: Select the target processing unit, i.e., the machine number to which this batch of tasks will be migrated; insert_pos: Determines the insertion position (e.g., the kth slot) in the current process sequence of the target unit.

[0060] The generation of all sub-actions will accept the sampling results of the previous action as the input of the next network, forming a chain decision-making process. Combined with the constraint checking mechanism, the feasibility of the action is ensured. If the action is invalid, the resampling mechanism is triggered or the default action is returned.

[0061] The hierarchical DRL model does not directly generate complete scheduling solutions. Instead, it learns to execute "intelligent mutation" strategies during the iteration of the genetic algorithm, which makes the decision-making process hierarchically decoupled, with high interpretability and training efficiency. When used, it achieves deep collaboration with GA. The overall model combines the advantages of reinforcement learning and GA, resulting in better solution performance and stronger generalization.

[0062] Each subnetwork adopts an independent parameter network structure, while sharing the underlying state encoder, achieving a balance between parameter efficiency and decision-making flexibility.

[0063] The model is trained using the Proximal Policy Optimization (PPO) algorithm. Based on the current scheduling status, the model makes hierarchical decisions and finally outputs specific mutation actions such as "insert the three consecutive tasks from in to in+2 on unit m1 into position x of unit m2", which serve as the intelligent mutation strategy of GA.

[0064] This approach makes mutations more purposeful, avoids blind searches, significantly improves optimization efficiency and solution quality, and achieves a deeper integration of DRL and GA.

[0065] The reward function is designed as a percentage function of the fitness improvement before and after mutation relative to the fitness of the initial solution, in order to encourage the model to learn efficient search directions.

[0066] In the inference step, the optimization suggestions output by DRL serve as enhanced mutation operators for GA, improving convergence speed and the quality of the final scheduling scheme. They specify the concrete operations for mutation. For example, "migrate task A from machine M1 to M2 and swap its order with task B" is an action oriented towards a specific solution structure, and DRL provides precise suggestions for each mutation.

[0067] DRL learns the most efficient search path in the current state, enabling intelligent guidance of mutation operations. It obtains high-quality solutions without relying on manually designed complex mutation rules, avoiding a large number of invalid or destructive random mutations, thus improving both search efficiency and solution quality. Specifically, when the system is put into actual operation, the genetic algorithm optimization process is initiated based on real-time production status, and the trained DRL model is introduced for dynamic guidance, including: Step S1: Initialize the scheduling population: The initial scheduling generation module generates a set of feasible initial solutions based on expert rules or randomness, which constitute the initial population of the genetic algorithm.

[0068] Step S2, Fitness Assessment: A multi-objective fitness assessment is conducted on all individuals in the current population, taking into account indicators such as completion time, delivery time fulfillment rate, equipment utilization rate, and cost.

[0069] Step S3, Select: Perform a selection process (such as tournament selection, without being limited to a specific algorithm) on the current population to retain elite individuals.

[0070] Step S4, Cross: During the crossover phase, a crossover operation is performed (not limited to a specific algorithm) to generate offspring individuals to form an optimized population.

[0071] Step S5, Mutation: During the mutation phase, the user can choose whether to use traditional random mutation or heuristic mutation methods.

[0072] Simultaneously, a pre-trained hierarchical DRL model (with multiple different mutation methods that can be executed in parallel) is activated: the state-aware module extracts the current population in real time, that is, optimizes the scheduling state features of individuals in the population, and inputs them into the DRL model; the DRL model outputs suggested "high-value mutation actions"; the feasibility of the action is verified (such as process sequence, resource conflicts, etc.), and if it is legal, the mutation is executed; otherwise, local adjustments or resampling are performed.

[0073] Step S6: Perform mutation based on high-value mutation actions to generate the next generation population, and determine the termination condition: Determine if the termination condition is met (such as the maximum number of iterations or the convergence threshold, which is set at the beginning based on the real-time production status and actual generation needs). If not met, return to S2 to continue the next round of evolution.

[0074] More preferred examples also include step S7, closed-loop feedback, and continuous learning: During system operation, the system continuously collects actual scheduling results and DRL recommended action effect data for online fine-tuning or periodic retraining of the model, enabling the autonomous evolution and continuous optimization of the scheduling strategy.

[0075] Within the mature framework of genetic algorithms, a hybrid intelligent optimization structure is adopted, using deep reinforcement learning to guide the mutation operation of the genetic algorithm. This structure uses the deep reinforcement learning model as the intelligent mutation operator within the genetic algorithm, enabling it to learn the mapping relationship from the current scheduling state to effective local adjustment actions. This hybrid paradigm retains the robustness of genetic algorithms in global search while introducing the intelligence of deep reinforcement learning in policy learning. It addresses the challenges of high mutation blindness in traditional genetic algorithms, the difficulty of pure reinforcement learning in handling large-scale combinatorial optimization problems, and the lack of dynamic policy learning capabilities in existing hybrid methods. This achieves efficient, stable, and interpretable solutions to complex combinatorial optimization problems.

[0076] This invention also provides a production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm. The production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm can be implemented by executing the process steps of the production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm. That is, those skilled in the art can understand the production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm as a preferred embodiment of the production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm.

[0077] The present invention provides a production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm, comprising: a state awareness module, an initial schedule generation module, a genetic algorithm optimization engine, a hierarchical DRL model, and an evaluation and feedback module; The state perception and feature extraction module, also known as the state perception module, collects multi-dimensional state information from the production system in real time, including a list of tasks to be processed (duration, priority, process path), unit status (availability, processing capacity, maintenance plan), and material inventory and circulation status. An encoder (such as a Transformer) maps the original state into feature vectors, which are then used as input to the subsequent model.

[0078] The initial scheduling generation module generates one or more feasible initial scheduling schemes based on heuristic rules (such as the shortest processing time) or simple scheduling algorithms (such as greedy algorithms), which serve as the initial population for the genetic algorithm.

[0079] The genetic algorithm optimization engine is responsible for the overall search process, including standard operations such as selection, crossover, mutation, and fitness evaluation. Its key improvement lies in the introduction of a DRL model to provide guidance during the mutation operation. The fitness function comprehensively considers indicators such as completion time, delivery time fulfillment rate, equipment utilization rate, and cost.

[0080] The hierarchical DRL model adopts a hierarchical structure design, including at least one sub-network. It is trained using a proximal policy optimization algorithm, and makes hierarchical decisions based on the current scheduling status, outputting specific mutation actions.

[0081] The sub-network adopts an independent parameter network structure and shares the underlying state encoder.

[0082] The reward function of the hierarchical DRL model is a percentage function of the fitness improvement before and after mutation relative to the fitness of the initial solution.

[0083] The hierarchical DRL model comprises a high-level policy network and a low-level action network, with different mutation types and corresponding specific mutation actions. The high-level policy network determines whether mutation is needed and which type of mutation to execute, while the low-level action network outputs the specific "mutation action".

[0084] The global scheduling state features are input into the high-level policy network to determine whether to perform mutation and the type of mutation to be performed. If mutation is not performed, the lower-level action network is not activated, and the result is output directly. If mutation is performed, the mutation type to be performed is selected. The lower-level action network determines the corresponding mutation action based on the mutation type, decomposes it into multiple ordered sub-actions in a structured manner, generates the corresponding specific parameters, and outputs them level by level.

[0085] The sub-action accepts the sampling result of the previous sub-action as the input of the next network. Combined with the set constraint checking mechanism, if the action is invalid, the resampling mechanism is triggered or the default action is returned. If the action is valid, the next sub-action is judged, and all sub-actions are traversed.

[0086] The hierarchical DRL model uses the scheduling state of individuals in the genetic algorithm population as the state space and aims to efficiently improve fitness as the reward objective. During the training phase, it learns the optimal mutation strategy by interacting with the simulation environment.

[0087] The initial scheduling generation module constructs the initial population, inputs it into the genetic algorithm optimization engine for multi-objective fitness evaluation, performs selection and crossover operations, and generates an optimized population; The state awareness module extracts the scheduling state features of the optimized population in real time and inputs them into the hierarchical DRL model. It outputs high-value mutation actions, generates the next generation of the population, and determines the termination conditions.

[0088] The initial population is a set of feasible initial solutions based on expert rules or randomly generated.

[0089] The state awareness module extracts the scheduling state features of individuals in the optimized population in real time and inputs them into the hierarchical DRL model. The hierarchical DRL model outputs high-value mutation actions and performs feasibility checks. If the actions are valid, they are executed. If they are invalid, local adjustments are made or the samples are resampled and then input into the hierarchical DRL model for mutation.

[0090] The termination condition is set according to the actual needs of real-time production status, and the judgment includes: If the set termination condition is met, the iteration terminates and mutation stops. If the set termination condition is not met, the genetic algorithm optimization engine performs a multi-objective fitness evaluation on the newly generated population.

[0091] In practical use, the evaluation and feedback module performs performance evaluations (such as total completion time, equipment utilization, etc.) on each generation of scheduling schemes generated by the genetic algorithm, and feeds the results back to the DRL model for subsequent strategy adjustments, forming a closed-loop learning mechanism. It continuously collects the effect data of actual scheduling results and the mutation actions output by the hierarchical DRL model for online fine-tuning or periodic retraining of the model.

[0092] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0093] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm, characterized in that, The application relates to a method for optimizing production scheduling based on a hierarchical deep reinforcement learning (DRL) model. The method comprises a training step and an inference optimization step. The training step comprises: Step S1: constructing a scheduling simulation environment based on actual production line information and pre-training a hierarchical DRL model; Step S2: further training the pre-trained hierarchical DRL model to output a final hierarchical DRL model; The inference optimization step comprises: Step S3: constructing an initial population by an initial scheduling generation module; Step S4: performing multi-objective fitness evaluation on the current population, executing selection and crossover operations, and generating an optimized population; Step S5: a state perception module extracts scheduling state features of the optimized population in real time, inputs the scheduling state features into the hierarchical DRL model, outputs high-value mutation actions, and executes the high-value mutation actions; Step S6: generating a next generation population and performing termination condition judgment. 2.The production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm according to claim 1, characterized in that, The scheduling simulation environment simulates the execution of mutation operations and evaluates the fitness changes before and after the mutation operations based on the process constraints, resource capabilities, task paths and task characteristics of the actual production line. In step S1, a traditional genetic algorithm or a heuristic algorithm is used to solve multiple scheduling problem instances, the state of each generation population and the effective mutation actions that subsequently lead to a significant improvement in fitness are recorded as pre-training data, and the model is pre-trained according to the pre-training data. In step S2, the current scheduling state is input into the pre-trained hierarchical DRL model, a mutation action is output, the mutation action is applied to the scheduling simulation environment to obtain the fitness change, and the network parameters of the hierarchical DRL model are updated according to the fitness change. The hierarchical DRL model comprises a high-level strategy network and a low-level action network. The global scheduling state features are input into the high-level strategy network to determine whether to execute a mutation and the type of mutation to be executed. The low-level action network generates specific parameters of the corresponding mutation action according to the type of mutation. 3.The production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm according to claim 2, characterized in that, Different mutation types and corresponding specific mutation actions are set in the hierarchical DRL model. The high-level strategy network determines whether to execute a mutation, if no mutation is executed, the low-level action network is not activated, and the result is directly output, if a mutation is executed, the type of mutation to be executed is selected; The low-level action network determines the corresponding mutation action according to the selected mutation type, structures and disassembles the mutation action into an ordered plurality of sub-actions, generates corresponding specific parameters and outputs the specific parameters step by step; The sub-action accepts the sampling result of the previous sub-action as the input of the next network, judges in combination with the set constraint checking mechanism, if the action is illegal, a resampling mechanism is triggered or a default action is returned, if the action is legal, the next sub-action is continuously judged, and all sub-actions are traversed. 4.The production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm according to claim 1, characterized in that, The initial population is a group of feasible initial solutions generated based on expert rules or randomly. In step S5, the state perception module extracts scheduling state features of individuals in the optimized population in real time and inputs the scheduling state features into the hierarchical DRL model; The hierarchical DRL model outputs high-value mutation actions and performs feasibility verification, if the mutation actions are legal, the mutation actions are executed, if the mutation actions are illegal, local adjustment or resampling is performed, and the hierarchical DRL model is input again to perform mutation; The termination condition is set according to the actual production state, and the judgment comprises: If the set termination condition is met, the iteration is terminated and the mutation is stopped, and if the set termination condition is not met, step S4 is executed.

5. The production scheduling optimization method based on the fusion of deep reinforcement learning and genetic algorithm according to any one of claims 1-4, characterized in that, Further comprising step S7: The effect data of the actual scheduling result and the mutation action output by the hierarchical DRL model is continuously collected, the performance of each generated scheduling scheme is evaluated, and the results are fed back to the hierarchical DRL model for online fine-tuning or periodic retraining of the model.

6. A production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm, characterized in that, Comprise: The state perception module, the initial scheduling generation module, the genetic algorithm optimization engine and the hierarchical DRL model; The initial scheduling generation module constructs an initial population, inputs the genetic algorithm optimization engine for multi-objective fitness evaluation, performs selection and crossover operations, and generates an optimized population; The state perception module extracts the scheduling state features of the optimized population in real time and inputs them into the hierarchical DRL model to output high-value mutation actions, generate the next generation population, and perform termination condition judgment. 7.The production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm according to claim 6, characterized in that, The hierarchical DRL model comprises at least one subnetwork, is trained using a proximal policy optimization algorithm, makes hierarchical decisions based on the current scheduling state, and outputs specific mutation actions; The subnetworks use a network structure with independent parameters and share a bottom state encoder; The reward function of the hierarchical DRL model is a percentage function of the fitness improvement before and after mutation compared to the fitness of the initial solution. 8.The production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm of claim 7, wherein, The hierarchical DRL model comprises a high-level policy network and a bottom-level action network, and different mutation types and corresponding specific mutation actions are set; The global scheduling state features are input into the high-level policy network to determine whether to perform mutation and the type of mutation to be performed, if no mutation is performed, the bottom-level action network is not activated and the result is directly output, and if mutation is performed, the type of mutation to be performed is selected; The bottom-level action network determines the corresponding mutation action based on the mutation type, structurally decomposes it into an ordered plurality of sub-actions, generates corresponding specific parameters and outputs them step by step; The sub-actions accept the sampling results of the previous sub-action as the input of the next network, and judge in combination with the set constraint checking mechanism, if the action is illegal, the resampling mechanism is triggered or the default action is returned, if the action is legal, the next sub-action is judged, and all sub-actions are traversed. 9.The production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm of claim 6, wherein, The initial population is a set of feasible initial solutions generated based on expert rules or randomly; The state perception module extracts the scheduling state features of the individuals in the optimized population in real time and inputs them into the hierarchical DRL model; The hierarchical DRL model outputs high-value mutation actions and performs feasibility verification, if legal, the mutation action is executed, if not legal, local adjustment or resampling is performed and the hierarchical DRL model is input again for mutation; The termination condition is set according to the actual production state generation requirement, and the judgment includes: If the set termination condition is met, the iteration is terminated and the mutation is stopped, and if the set termination condition is not met, the genetic algorithm optimization engine performs multi-objective fitness evaluation on the newly generated population.

10. The production scheduling optimization system based on the fusion of deep reinforcement learning and genetic algorithm according to any one of claims 6-9, characterized in that, Further comprising an evaluation and feedback module; The evaluation and feedback module continuously collects the effect data of the actual scheduling result and the mutation action output by the hierarchical DRL model, evaluates the performance of each generated scheduling scheme, and feeds the results back to the hierarchical DRL model for online fine-tuning or periodic retraining of the model.

Images