Workshop scheduling scheme generation method and system based on deep reinforcement learning

Abstract

The invention provides a workshop scheduling scheme generation method and system based on deep reinforcement learning, and the method comprises the following steps: firstly, obtaining a data set, and dividing the data set into a training set and a verification set; constructing an overall framework of the intelligent manufacturing system; then, carrying out workshop scheduling environment compiling based on a Markov chain, improving a deep reinforcement learning algorithm, and constructing an improved deep reinforcement learning model; next, based on the training set and the verification set, training the improved deep reinforcement learning model, and adjusting model parameters to obtain a trained deep reinforcement learning model; and finally, inputting workshop system parameters and a scheduling target into the trained deep reinforcement learning model to obtain a visual recommendation scheme. According to the method and the system provided by the invention, the existing progress of the workshop can be redistributed, and a more intelligent recommendation scheme is provided for the management level by researching the construction method of the integrated scheduling environment of the production workshop.

Description

Technical Field

[0001] The present application relates to the technical field of workshop integrated scheduling, and in particular to a method and system for generating a workshop scheduling plan based on deep reinforcement learning. Background Art

[0002] Shop scheduling is an important task in production management, which aims to reasonably arrange production resources and processes to improve production efficiency and reduce production costs. With the rapid development of the manufacturing industry, the production environment is becoming increasingly complex, and shop scheduling faces many challenges, including multi-variety, small batch production, equipment failure, and differences in worker skills.

[0003] Traditional scheduling methods mainly rely on experience and manual judgment, and are difficult to adapt to rapidly changing production needs. In recent years, with the advancement of artificial intelligence technology and intelligent manufacturing, deep reinforcement learning algorithms and optimization methods have gradually been introduced into workshop scheduling. In addition, the application of real-time data acquisition and analysis technology enables the scheduling system to monitor the production status in real time, respond quickly to emergencies, and improve production flexibility and adaptability. Modern workshop scheduling not only focuses on production efficiency, but also emphasizes response speed and flexibility, becoming an important means to enhance corporate competitiveness. Therefore, developing a workshop scheduling solution generation system based on deep reinforcement learning is of great significance to improving the response speed and flexibility of workshop scheduling. Summary of the invention

[0004] The embodiment of the present application provides a workshop scheduling plan generation system based on deep reinforcement learning, which can reallocate the existing work progress and actual progress of the workshop, and provide a more intelligent recommendation plan for the management level by studying the construction method of the integrated scheduling environment of the production workshop.

[0005] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a method for generating a workshop scheduling plan based on deep reinforcement learning, which method includes the following steps: first, obtaining a data set and dividing the data set into a training set and a verification set; constructing an overall framework of the intelligent manufacturing system; then, compiling the workshop scheduling environment based on the Markov chain, improving the deep reinforcement learning algorithm, and constructing an improved deep reinforcement learning model; next, training the improved deep reinforcement learning model based on the training set and the verification set, adjusting the model parameters, and obtaining a trained deep reinforcement learning model; finally, inputting the workshop system parameters and scheduling goals into the trained deep reinforcement learning model to obtain a visual recommendation plan.

[0006] In some exemplary embodiments, the overall framework of the intelligent manufacturing system is constructed, including: first, constructing an enterprise resource planning system, a manufacturing execution system and an underlying control system respectively; then, based on the agent models in the enterprise resource planning system, the manufacturing execution system and the underlying control system, a negotiation mechanism between agent models is constructed; finally, based on the negotiation mechanism between agent models and the constructed enterprise resource planning system, the manufacturing execution system and the underlying control system, the overall framework of the intelligent manufacturing system is obtained; wherein, the enterprise resource planning system includes enterprise agents; the manufacturing execution system includes product agents and central scheduling agents at each stage; the bottom control system includes warehouse agents, conveying agents, processing agents and maintenance agents, which are respectively responsible for the warehousing and out-of-warehouse of raw materials and finished products, the transportation of products, the processing and assembly of products, and the maintenance of equipment.

[0007] In some exemplary embodiments, based on the agent models in the enterprise resource planning system, the manufacturing execution system and the underlying control system, a negotiation mechanism between agent models is constructed, including: through the collection and analysis of information, the enterprise agent issues a production plan to the product agent, and the product agent, as an administrator, initiates task announcement information to the central scheduling agent of the warehouse agent, the transportation agent and the processing agent, and the product agent determines the type of agent that has successfully bid by evaluating its capabilities; each agent under the type of agent that has successfully bid negotiates with the transportation agent to determine the transportation route, the central scheduling agent ranks the characteristics of each bidding agent according to predefined rules, selects the agent with the highest ranking, and the central scheduling agent of the type to which the agent belongs sends a reward message for notification, and at the same time sends a directed reward message to the first transportation agent of the route for notification to start transporting the product.

[0008] In some exemplary embodiments, a workshop scheduling environment is compiled based on a Markov chain, a deep reinforcement learning algorithm is improved, and an improved deep reinforcement learning model is constructed, including: using the OpenAI Gym library to build an integrated scheduling environment that interacts with the deep reinforcement learning algorithm, and building a virtual workshop model based on the layout and equipment information of a real production workshop; the virtual workshop model includes the location, attributes and status information of various equipment, automatic guided vehicles (AGVs) and workpieces; encoding the workshop status into a vector or matrix form for input into the deep reinforcement learning algorithm for processing; conducting simulation experiments in the constructed integrated scheduling environment to verify the feasibility and effectiveness of the deep reinforcement learning algorithm in production scheduling; improving the deep reinforcement learning algorithm and constructing an improved deep reinforcement learning model.

[0009] In some exemplary embodiments, the deep reinforcement learning algorithm is improved to construct an improved deep reinforcement learning model, including: constructing the basic framework of the multi-agent proximal policy optimization (MAPPO) algorithm; adding the ε-greedy strategy to the action selection strategy part of the MAPPO algorithm to prevent the algorithm from falling into a local optimal solution, thereby obtaining an improved deep reinforcement learning model.

[0010] In some exemplary embodiments, after obtaining the trained deep reinforcement learning model and before inputting the workshop system parameters and scheduling objectives into the trained deep reinforcement learning model, it also includes: embedding the trained deep reinforcement learning model into a Web application.

[0011] In some exemplary embodiments, a trained deep reinforcement learning model is embedded in a Web application, including: implementing the core logic of the scheduling algorithm on the back end; creating a RESTful API that enables the front end to access the scheduling function through HTTP requests; creating a user interface that allows users to input parameters required for scheduling; ensuring that the front end and back end can interact with data through the network, the front end sends requests through the Fetch API, and the back end returns the scheduling results; deploying the back end and front end applications to a cloud platform or a local server to ensure that the front end and back end can communicate normally.

[0012] In some exemplary embodiments, the workshop system parameters and scheduling objectives are input into a trained deep reinforcement learning model, including: preprocessing the order and equipment data to be scheduled to obtain processed order and equipment data sets; inputting the processed order and equipment data sets into the trained deep reinforcement learning model to perform workshop scheduling.

[0013] On the second aspect, the embodiment of the present application also provides a workshop scheduling plan generation system based on deep reinforcement learning, including: a data set module, a system construction module, a deep reinforcement learning model module and a data processing module connected in sequence; the data set module is used to obtain the data set and divide the data set into a training set and a verification set; the system construction module is used to build the overall framework of the intelligent manufacturing system; the deep reinforcement learning model module is used to compile the workshop scheduling environment according to the Markov chain, improve the deep reinforcement learning algorithm, and construct an improved deep reinforcement learning model; the data processing module is used to train the improved deep reinforcement learning model according to the training set and the verification set, adjust the model parameters, and obtain a trained deep reinforcement learning model; the workshop system parameters and scheduling goals are input into the trained deep reinforcement learning model to obtain a visual recommendation plan.

[0014] In some exemplary embodiments, the overall framework of the system includes an order center module, an equipment center module and a scheduling center module; in the order center module, users can manage production orders, including importing complete production orders, adding, deleting and modifying order information, and scheduling orders; in the equipment center module, users can view various real-time status information of the equipment, including energy consumption, logistics information and equipment failure conditions, as well as add, delete or modify basic equipment information; in the scheduling center module, users can perform a series of settings and optimizations to generate the best production scheduling plan.

[0015] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0016] The embodiment of the present application provides a method and system for generating a workshop scheduling plan based on deep reinforcement learning, which includes the following steps: first, obtaining a data set and dividing the data set into a training set and a verification set; building an overall framework of the intelligent manufacturing system; then, compiling the workshop scheduling environment based on the Markov chain, improving the deep reinforcement learning algorithm, and building an improved deep reinforcement learning model; next, training the improved deep reinforcement learning model based on the training set and the verification set, adjusting the model parameters, and obtaining a trained deep reinforcement learning model; finally, inputting the workshop system parameters and scheduling goals into the trained deep reinforcement learning model to obtain a visual recommendation plan.

[0017] The method and system for generating workshop scheduling solutions based on deep reinforcement learning provided by this application utilize the interaction between the integrated scheduling environment and the deep reinforcement learning algorithm to realize the system-level debugging of the intelligent manufacturing factory. Through the real-time mapping and interaction between the integrated scheduling environment and the deep reinforcement learning algorithm, it provides remote monitoring, production process control and optimization, big data analysis and other functional services, which can timely grasp each other's dynamic changes and respond in real time, avoiding the problems of high cost, long cycle and high risk of traditional on-site debugging, and providing a strong guarantee for the system-level debugging and optimization of intelligent manufacturing factories. At the same time, this application introduces the MAPPO algorithm into the field of manufacturing collaborative optimization, and adds the ε-greedy strategy to the action selection strategy part of the MAPPO algorithm, further improving the algorithm's exploration trend in action selection, increasing the probability of obtaining better scheduling results, and preventing the algorithm from falling into a local optimal solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0019] Figure 1A flowchart of a method for generating a workshop scheduling solution based on deep reinforcement learning is provided in accordance with an embodiment of the present application.

[0020] Figure 2 A schematic diagram of a technical route for a workshop scheduling solution generation method based on deep reinforcement learning provided in one embodiment of the present application.

[0021] Figure 3 A schematic diagram of the overall system architecture of a method for generating a workshop scheduling solution based on deep reinforcement learning provided in one embodiment of the present application.

[0022] Figure 4 A schematic diagram of the structure of a workshop scheduling solution generation system based on deep reinforcement learning provided in one embodiment of the present application.

[0023] Figure 5 A schematic diagram of a process for generating a workshop scheduling solution based on deep reinforcement learning is provided in accordance with an embodiment of the present application.

[0024] Figures 6A to 6D A schematic diagram of a login interface, order center, equipment center, and scheduling center results of a workshop scheduling system based on deep reinforcement learning is provided in one embodiment of the present application.

[0025] Fig. 7A and 7B A workshop scheduling plan generation method based on deep reinforcement learning provided in an embodiment of the present application trains a certain round and a Gantt chart of the final scheduling results.

[0026] Fig. 8A and 8B A schematic diagram of reward curves before and after algorithm improvement of a workshop scheduling solution generation method based on deep reinforcement learning provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] As can be seen from the background technology, workshop scheduling faces many challenges, including multi-variety, small batch production, equipment failure, worker skill differences, etc., and traditional scheduling methods mainly rely on experience and manual judgment, which is difficult to adapt to rapidly changing production needs. Developing a workshop scheduling solution generation system based on deep reinforcement learning is of great significance to improving the response speed and flexibility of workshop scheduling.

[0028] In view of the shortcomings in the prior art, the embodiment of the present application provides a method and system for generating a workshop scheduling plan based on deep reinforcement learning, which includes the following steps: first, obtain a data set and divide the data set into a training set and a verification set; build an overall framework of the intelligent manufacturing system; then, compile the workshop scheduling environment based on the Markov chain, improve the deep reinforcement learning algorithm, and build an improved deep reinforcement learning model; next, based on the training set and the verification set, train the improved deep reinforcement learning model, adjust the model parameters, and obtain a trained deep reinforcement learning model; finally, input the workshop system parameters and scheduling goals into the trained deep reinforcement learning model to obtain a visual recommendation plan. The purpose of this application is to provide a method and system for generating a workshop scheduling plan based on deep reinforcement learning, which can redistribute the existing work progress and actual progress of the workshop, and provide a more intelligent recommendation plan for the management level by studying the construction method of the integrated scheduling environment of the production workshop.

[0029] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

[0030] See also Figure 1 , the embodiment of the present application provides a method for generating a workshop scheduling plan based on deep reinforcement learning, the method comprising the following steps:

[0031] Step S101: Acquire a data set, and divide the data set into a training set and a validation set.

[0032] Step S102: construct the overall framework of the intelligent manufacturing system.

[0033] Step S103: compile the workshop scheduling environment based on the Markov chain, improve the deep reinforcement learning algorithm, and build an improved deep reinforcement learning model.

[0034] Step S104: Based on the training set and the validation set, the improved deep reinforcement learning model is trained, and the model parameters are adjusted to obtain a trained deep reinforcement learning model.

[0035] Step S105: Input the workshop system parameters and scheduling objectives into the trained deep reinforcement learning model to obtain a recommended visualization solution.

[0036] It should be noted that in step S102, when constructing the framework, the overall framework of the intelligent manufacturing system is constructed based on deep reinforcement learning (DRL) and multi-agent system (MAS), and the characteristics and interrelationships of machine characteristics, automatic guided vehicle (AGV) characteristics, and workpiece characteristics are clarified to ensure stable operation of production under this framework and support the coordinated optimization of the system. The machine characteristics include processing equipment, working status, standby status, etc. The AGV characteristics include operating time, working status, power, etc. The equipment receives the workpiece and processes it according to the production scheduling task to ensure equipment utilization and production efficiency. The AGV is responsible for transporting the workpiece from one device to another to ensure coordinated operation between devices.

[0037] In some embodiments, step S102 constructs the overall framework of the intelligent manufacturing system, including:

[0038] Step S1021, constructing an enterprise resource planning (ERP) system, a manufacturing execution system (MES) and an underlying control system respectively.

[0039] Step S1022: Based on the agent models in the enterprise resource planning system, the manufacturing execution system and the underlying control system, a negotiation mechanism between the agent models is established.

[0040] Step S1023: Based on the negotiation mechanism between the agent models and the constructed enterprise resource planning system, manufacturing execution system and underlying control system, the overall framework of the intelligent manufacturing system is obtained.

[0041] Among them, the enterprise resource planning system includes the enterprise agent (EA); the manufacturing execution system includes the product agent (PA) and the central scheduling agent at each stage; the bottom control system includes the warehouse agent (WA), the conveying agent (CA), the processing agent (MA) and the maintenance agent (SA), which are responsible for the warehousing and outbound of raw materials and finished products, the transportation of products, the processing and assembly of products, and the maintenance of equipment respectively.

[0042] Specifically, in step S102, the intelligent manufacturing system is divided into the following three parts: enterprise resource planning system, manufacturing execution system and bottom control system. Each production and processing stage adopts a framework with one intelligent agent as the center to coordinate and dispatch other intelligent agents. The specific contents are as follows: the intelligent agent in the bottom control system is responsible for controlling the beat, output and manufacturing method of each production and processing stage, the intelligent agent in the manufacturing execution system is mainly responsible for the coordinated scheduling of multiple intelligent agents in different production and processing stages, and the intelligent agent in the enterprise resource planning system is responsible for issuing production plans and corresponding command and dispatch to the intelligent agents in the production management system. The three parts are connected through the information transmission system, and finally the purpose of basically realizing the intelligence and informatization of the manufacturing system is achieved. In order to realize the adaptive self-evolutionary collaborative optimization of the intelligent manufacturing system, the collaborative optimization is divided into manufacturing and inventory, distribution, assembly, and maintenance according to the collaborative stage. By modeling each part, designing algorithms, simulating and testing, etc., the purpose of improving the global collaboration and optimization capabilities of the intelligent manufacturing system is achieved.

[0043] In some embodiments, step S1022 builds a negotiation mechanism between agent models based on the agent models in the enterprise resource planning system, the manufacturing execution system, and the underlying control system, including:

[0044] Through the collection and analysis of information, the EA agent issues a production plan to the PA agent. The PA agent, as an administrator, initiates task announcement information to the central scheduling agent of the warehouse agent, the delivery agent, and the MA agent. The PA agent determines the type of agent that has won the bid by evaluating their capabilities. Each agent under the type of agent that has won the bid negotiates with the CA agent to determine the delivery route. The central scheduling agent ranks the characteristics of each bidding agent according to some predefined rules. The highest-ranked agent will be selected and the central scheduling agent of that type will send a reward message for notification. At the same time, a directed reward message will be sent to the first CA agent on the route for notification to start the transportation of the product.

[0045] Specifically, a contractor can receive multiple task notification messages from different managers. Tasks enter the message queue for buffering, and messages that enter the queue earlier will be processed earlier, that is, the contractor does not sort these messages. The contractor's decision-making process is as follows: the contractor first analyzes the earliest task process in the message queue. If the current capacity can meet the requirements of the task, then judge its own status. If it is idle, it sends a bidding message to the manager and enters the waiting period. Otherwise, it immediately sends a response message to the manager to facilitate the manager to make a better decision.

[0046] The negotiation process for determining the delivery route is as follows: The CA agent is responsible for transporting the product. Except for the CA agent responsible for the center's coordinated scheduling, each of the other CA agents has its controllable / reachable range, that is, different CA agents have different operating ranges and different accessible sets. Agents belonging to the same area can access their own data sets. At the same time, CA agents can access their own data sets to facilitate the handover of products. After the CA agent responsible for the center's coordinated scheduling starts the search process, the CA agent continues to search until a valid route is found.

[0047] The manager selects the following ranking rules for bidders: The manager will select a contractor from the bidders that have sent a bid message. The bidders plan to use multiple rules for ranking, and the rules to be adopted are determined according to the different types of bidders. The specific rules are as follows: bidders with shorter transportation routes are ranked higher; bidders with lower processing accuracy or testing accuracy are ranked higher (this rule saves more resources for other operations on the premise that all bidders can meet the accuracy requirements); bidders with shorter time are ranked higher. Once a bidder obtains a higher score than other bidders, it wins the task and the ranking process is terminated. Although the manager's ranking process is based on local decisions, our goal is to approach global optimization by introducing a reward mechanism, giving priority to the shortest transportation route and saving more precise and faster resources for other operations.

[0048] In some embodiments, in step S103, the shop scheduling environment is compiled based on the Markov chain, the deep reinforcement learning algorithm is improved, and the improved deep reinforcement learning model is constructed, including:

[0049] Step S1031: Use the OpenAI Gym library to build an integrated scheduling environment that interacts with the deep reinforcement learning algorithm, and build a virtual workshop model based on the layout and equipment information of the actual production workshop; the virtual workshop model includes the location, attributes and status information of various equipment, automatic guided vehicles (AGVs) and workpieces.

[0050] Step S1032: Encode the workshop state into a vector or matrix form so as to be input into the deep reinforcement learning algorithm for processing.

[0051] Step S1033: Conduct simulation experiments in the constructed integrated scheduling environment to verify the feasibility and effectiveness of the deep reinforcement learning algorithm in production scheduling; improve the deep reinforcement learning algorithm and construct an improved deep reinforcement learning model.

[0052] Through the above detailed construction method, an integrated scheduling environment that can interact with the reinforcement learning algorithm can be built to verify and optimize the scheduling plan of the production workshop and improve production efficiency and resource utilization.

[0053] In some embodiments, in step S103, the deep reinforcement learning algorithm is improved to construct an improved deep reinforcement learning model, including:

[0054] Step S1131, constructing the basic framework of the multi-agent proximal policy optimization (MAPPO) algorithm;

[0055] Step S1132: adding the ε-greedy strategy to the action selection strategy of the MAPPO algorithm to further improve the exploration trend of the algorithm in action selection, increase the probability of obtaining better scheduling results, and prevent the algorithm from falling into a local optimal solution.

[0056] When improving deep reinforcement learning algorithms, enable them to perform adaptive collaborative optimization in different production environments, and apply these algorithms to actual production scheduling problems. Model different production environments in detail, including the characteristics of equipment, processes, material flows, etc. In terms of state expression, use more complex data structures to describe various relationships and state changes in the production environment. Select a suitable reinforcement learning framework, make customized improvements based on the specific characteristics of the production environment, and increase module processing uncertainty. Integrate real-time production data into the scheduling environment so that the algorithm can be trained and debugged in a situation closer to actual production.

[0057] In some embodiments, after obtaining the trained deep reinforcement learning model in step S103 and before inputting the workshop system parameters and the scheduling target into the trained deep reinforcement learning model in step S104, the method further includes:

[0058] Step S1030: embed the trained deep reinforcement learning model into the Web application.

[0059] In some embodiments, step S1030 embedding the trained deep reinforcement learning model into the Web application includes:

[0060] Step S1030a: Implement the core logic of the scheduling algorithm at the back end.

[0061] Step S1030b: Create a RESTful API so that the front end can access the scheduling function through HTTP requests.

[0062] Step S1030c: create a user interface to allow the user to input the parameters required for scheduling.

[0063] Step S1030d: Ensure that the front-end and back-end can exchange data through the network. The front-end sends a request through the Fetch API, and the back-end returns the scheduling result.

[0064] Step S1030e: deploy the back-end and front-end applications to the cloud platform or local server to ensure that they can communicate normally.

[0065] In some embodiments, in step S105, the workshop system parameters and the scheduling objectives are input into the trained deep reinforcement learning model to obtain a recommended visualization solution, including:

[0066] Step S1051: pre-process the order and equipment data to be scheduled to obtain a pre-processed order and equipment data set.

[0067] Step S1052: input the preprocessed order and equipment data set into the trained deep reinforcement learning model to perform workshop scheduling.

[0068] Specifically, the preprocessing of the order, equipment and other data sets to be scheduled is the same as the preprocessing of the training set, validation set and test set. The order, equipment and other data sets to be scheduled are preprocessed so that the dimensions of the preprocessed order, equipment and other data sets are the same as the dimensions of the data sets in the training set, validation set and test set. The preprocessed order, equipment and other data sets to be scheduled are input into the trained deep reinforcement learning model to obtain the scheduling results, and then a visualization solution can be generated.

[0069] See also Figure 2 , the embodiment of the present application provides a technical route diagram of a method for generating a workshop scheduling plan based on deep reinforcement learning, comprising the following steps:

[0070] Step S201: Build an overall framework of an advanced and innovative intelligent manufacturing system, perform software simulation modeling based on this framework, and write an environment under the OpenAI Gym library.

[0071] Step S202: Use the deep reinforcement learning algorithm to run and debug in the environment to obtain the optimal scheduling plan under the model definition, and verify it in simulation modeling, and finally obtain a deep reinforcement learning algorithm and model suitable for the industrial multi-agent environment.

[0072] In addition, an embodiment of the present application also provides a workshop scheduling plan generation system based on deep reinforcement learning, including: a data set module, a system construction module, a deep reinforcement learning model module and a data processing module connected in sequence; the data set module is used to obtain the data set and divide the data set into a training set and a verification set; the system construction module is used to build the overall framework of the intelligent manufacturing system; the deep reinforcement learning model module is used to compile the workshop scheduling environment based on the Markov chain, and improve the deep reinforcement learning algorithm to construct an improved deep reinforcement learning model; the data processing module is used to train the improved deep reinforcement learning model according to the training set and the verification set, adjust the model parameters, and obtain a trained deep reinforcement learning model; the workshop system parameters and scheduling goals are input into the trained deep reinforcement learning model to obtain a visual recommendation plan.

[0073] See also Figure 3 , the embodiment of the present application provides a schematic diagram of the overall architecture of an intelligent manufacturing system for a workshop scheduling scheme generation method based on deep reinforcement learning, and the implementation process of the system includes the following steps:

[0074] Step S301: The enterprise resource planning system collects real-time data on the production line, analyzes and processes the data, and then formulates and issues a production plan to the production management system. The data results of the production process will also be directly fed back to the analysis system to facilitate timely adjustment of the production plan.

[0075] Step S302, the manufacturing execution system can perform digital workshop modeling, and through the collection of production information, form a complete product production process database, realize real-time traceability of products, and continuously optimize and shorten the inspection cycle through analysis; real-time monitoring of production progress, equipment information, etc. can also predict possible situations in production, make emergency plans and appropriate adjustments in advance, and improve production capacity and production efficiency.

[0076] Step S303: The manufacturing execution system plays a role in the entire intelligent manufacturing system, providing effective data support for the enterprise resource planning system, issuing production plans, and completing the task allocation of production equipment. The bottom control system is mainly composed of various digital equipment in the workshop, which is mainly responsible for the transportation, processing and assembly of raw materials.

[0077] See also Figure 4The overall framework of the system includes the order center module, the equipment center module and the scheduling center module; in the order center module, users can manage production orders, including importing complete production orders, adding, deleting and modifying order information, and scheduling orders; in the equipment center module, users can view various real-time status information of the equipment, including energy consumption, logistics information and equipment failure conditions, as well as add, delete or modify basic equipment information; in the scheduling center module, users can make a series of settings and optimizations to generate the best production scheduling plan.

[0078] Specifically, the implementation process of the system includes the following steps:

[0079] Step S401, in the order center module, users can manage production orders, including importing complete production orders, adding, deleting and modifying order information. In addition, users can also click the order scheduling button, and the system will select the current order to be processed based on a series of factors and issue workpiece production tasks to the workshop. Users can import complete production orders by uploading files or manually entering information. Each order includes key information such as order number, workpiece name, planned production quantity, order arrival time, processing plan, and delivery time. The system should be able to parse the uploaded file and store the order information in the database. At the same time, users can also fill in the order information directly on the interface and save it. Users can add new order information through the Add button on the interface. Users need to fill in the order number, workpiece name, planned production quantity, order arrival time, processing plan, delivery time and other information, and save it to the system. Ensure the uniqueness of the order number and perform necessary data verification, such as the planned production quantity should be a positive integer. You can also select the order to be deleted and delete the order information by clicking the Delete button. Users can select the order to be modified and enter the modification page through the Edit button. Users can modify the various fields of the order and save the modified information.

[0080] Step S402: In the device center module, users can view various real-time status information of the device, including energy consumption, logistics information, and device failure conditions, and can also add, delete, or modify basic device information. At the same time, when a device fails, the application will remind the user through a pop-up window. Ensure the timeliness and accuracy of the reminder so that users can respond quickly and handle the failure.

[0081] Step S403, in the scheduling center module, the user can make a series of settings and optimizations to generate the best production scheduling plan. The user can set the pre-investment quantity of each equipment, that is, the number of equipment that has been prepared before the production plan starts. This helps the system consider the availability and production capacity of the equipment when scheduling. At the same time, the user can choose to optimize the production scheduling plan to the lowest cost. The system will consider the utilization efficiency and cost control of various resources when generating the plan to achieve the goal of minimizing costs; the system optimization goal can be set to the highest efficiency, that is, by reasonably arranging production tasks and equipment utilization, the idle time in the production process is minimized to improve production efficiency and output; if the user is concerned about energy consumption and environmental protection issues, the production scheduling plan can be optimized to the lowest energy consumption. The system will consider the energy utilization efficiency of the equipment and optimize the production plan to reduce energy consumption. After completing the above settings, the user can click the scheduling button, and the system will call the algorithm to generate the best production scheduling plan based on the user's parameter settings and optimization goals. The application displays the production scheduling plan to the user in the form of a Gantt chart, including the start time, end time, required equipment, workers and other information of each task. Users can clearly understand the arrangement and timeline of the production plan through the Gantt chart.

[0082] See also Figure 5 , the embodiment of the present application provides a schematic diagram of a process for generating a workshop scheduling solution based on deep reinforcement learning, including the following steps:

[0083] Step S501: Import production orders.

[0084] Step S502: Order scheduling, including raw material inventory status, delivery date, urgency, etc.

[0085] Step S503: Issue a workpiece production task.

[0086] Step S504: Set the basic parameters of the algorithm according to the actual situation, including the number of equipment invested, workshop layout, processing time of each process, etc.

[0087] Step S505: The user defines the optimization target, including cost, time, energy consumption, etc.

[0088] Step S506: Generate a processing Gantt chart.

[0089] Step S507: If an emergency occurs, perform specific processing according to the type of emergency.

[0090] Specifically, in step S506, in the production resource information management system, data collection is performed through a deep reinforcement learning algorithm, and production data is analyzed for adjustment and improvement of the algorithm, thereby generating a Gantt chart.

[0091] Specifically, in step S507, the current order status and the operating status of each device are monitored in real time in the production resource information management system. If an emergency occurs and the emergency is an inserted order, an urgent order, or an order cancellation, the order is rescheduled; if an emergency occurs and the emergency is an equipment failure, the processing Gantt chart is regenerated.

[0092] See also Figures 6A to 6D , an embodiment of the present application provides a login interface of a workshop scheduling solution generation system based on deep reinforcement learning, an order center, an equipment center, and a scheduling center result schematic diagram, including the following steps:

[0093] Step S601, input user name and password, log in to the system;

[0094] Step S602, upload the order file in the order center or directly enter the order information, including key information such as order number, workpiece name, planned production quantity, order arrival time, processing plan, delivery time, etc.;

[0095] Step S603, uploading equipment information, such as processing machines or automatic guided vehicles (AGVs), etc., to the equipment center;

[0096] Step S603: Input relevant parameters and optimization goals in the scheduling center to obtain a Gantt chart of scheduling results.

[0097] The embodiment of this application aims to achieve collaborative optimization of production scheduling, improve the autonomy and intelligence of the production workshop, and enable it to have the capabilities of adaptive optimization and collaborative optimization in the production environment. By studying the characteristics and interrelationships of various elements in production scheduling, an integrated scheduling environment that can interact with deep reinforcement learning algorithms is constructed, and at the same time, the deep reinforcement learning algorithm is improved so that it can be used for adaptive collaborative optimization of different production scheduling problems, and verified in the constructed integrated scheduling environment to ensure that the algorithm used can improve the collaborative optimization capabilities of the constructed system model, provide support for the collaborative optimization of intelligent manufacturing systems, and promote the intelligent transformation and development of the manufacturing industry.

[0098] The following is a detailed description of the workshop scheduling solution generation system based on deep reinforcement learning provided by the present application with respect to a specific embodiment.

[0099] The method for generating a workshop scheduling plan based on deep reinforcement learning proposed in the embodiment of the present application is based on the Windows 11 system and is implemented using Python 3.9.

[0100] First, the order file is extracted through the order center module and parsed into a fixed format, including order number, workpiece name, planned production quantity, order arrival time, processing plan, delivery time and other information to form a data set, which is saved after preprocessing.

[0101] It is worth noting that the embodiment of the present application is randomly generated by an example generation program after defining the basic features, and for the convenience of defining the basic features, the number of workpieces contained in each order is set to 1, and the basic features are defined as follows: the initial number of workpieces (orders) is 5, the number of newly arrived workpieces (orders) is 10, the arrival time of new workpieces (orders) obeys an exponential distribution with a parameter of 50, and the number of processes for each workpiece does not exceed 5. After defining the above basic features, the generated example is as follows: the arrival timetable of new workpieces is shown in Table 1; the number of processes required for each workpiece is shown in Table 2; the processing time required for each process of each workpiece on each machine is shown in Tables 3A to 3E.

[0102] Table 1 Arrival schedule of new workpieces in one embodiment

[0103] Workpiece serial number 1 2 3 4 5 6 7 8 9 10 11 1 2 1 3 14 15 Arrival time 0 0 0 0 0 26 63 0 18 7 4 10 21 25 38

[0104] Table 2 Number of steps required for a workpiece in an embodiment

[0105] Workpiece serial number 1 2 3 4 5 6 7 8 9 10 11 1 2 13 14 1 5 Number of processes 2 5 1 3 1 4 4 4 4 2 1 4 1 4 4

[0106] Table 3A - Processing time required for workpieces in the embodiment 1 on the machine

[0107]

[0108] Table 3B Processing time required for the second process of the workpiece on the machine in the embodiment

[0109]

[0110] Table 3C Processing time required for the workpiece in step 3 of an embodiment on the machine

[0111]

[0112] Table 3D Processing time required for the fourth step of the workpiece on the machine in the embodiment

[0113]

[0114] Table 3E Processing time required for process 5 of the workpiece on the machine in the embodiment

[0115]

[0116] Table 4: Delivery schedule of workpieces in an embodiment

[0117] Workpiece serial number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Delivery time 64 151 25 47 36 116 154 93 106 43 37 102 40 122 127

[0118] Then, the current equipment information is extracted through the equipment center module and the equipment status is monitored in real time. In the embodiment of the present application, the number of processing machines is 10.

[0119] Finally, the scheduling target is input into the scheduling center module, and the workshop intelligent scheduling is performed through the scheduling center module to obtain the recommended scheduling plan. The scheduling target of the embodiment of the present application is to minimize the total delay. Fig. 7A and Figure 7B By comparison, we can see that with the increase in the number of training rounds, the final completion time is optimized from 236 to 201, which reduces the completion time and obtains better scheduling results. The machine production scheduling becomes more reasonable and resources are more fully utilized. Fig. 8A and Figure 8B Comparison of the reward curves of the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm before and after the algorithm improvement shows that the effect is better after the algorithm improvement. The algorithm improvement enhances the exploration performance of the algorithm, increases the probability of obtaining better scheduling results, and prevents the algorithm from falling into the local optimal solution.

[0120] Based on the above technical scheme, the embodiment of the present application provides a method and system for generating a workshop scheduling plan based on deep reinforcement learning, which method includes the following steps: first, obtaining a data set and dividing the data set into a training set and a verification set; building an overall framework of the intelligent manufacturing system; then, compiling the workshop scheduling environment based on the Markov chain, improving the deep reinforcement learning algorithm, and building an improved deep reinforcement learning model; next, based on the training set and the verification set, training the improved deep reinforcement learning model, adjusting the model parameters, and obtaining a trained deep reinforcement learning model; finally, inputting the workshop system parameters and scheduling goals into the trained deep reinforcement learning model to obtain a visual recommendation plan.

[0121] The method and system for generating workshop scheduling solutions based on deep reinforcement learning provided by this application utilize the interaction between the integrated scheduling environment and the deep reinforcement learning algorithm to realize the system-level debugging of the intelligent manufacturing factory. Through the real-time mapping and interaction between the integrated scheduling environment and the deep reinforcement learning algorithm, it provides remote monitoring, production process control and optimization, big data analysis and other functional services, which can timely grasp each other's dynamic changes and respond in real time, avoiding the problems of high cost, long cycle and high risk of traditional on-site debugging, and providing a strong guarantee for the system-level debugging and optimization of intelligent manufacturing factories. At the same time, this application introduces the multi-agent proximal policy optimization (MAPPO) algorithm into the field of manufacturing collaborative optimization, and adds the ε-greedy strategy to the action selection strategy part of the MAPPO algorithm, further improving the algorithm's exploration trend in action selection, increasing the probability of obtaining better scheduling results, and preventing the algorithm from falling into a local optimal solution.

[0122] Those skilled in the art can understand that the above-mentioned embodiments are specific examples for implementing the present application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application, so the scope of protection of the present application shall be based on the scope defined in the claims.

Claims

1. A method for generating a workshop scheduling plan based on deep reinforcement learning, characterized in that: The following steps are involved: Obtain a data set, and divide the data set into a training set and a validation set; Build the overall framework of intelligent manufacturing system; Compile the workshop scheduling environment based on the Markov chain, improve the deep reinforcement learning algorithm, and build an improved deep reinforcement learning model; Based on the training set and the validation set, the improved deep reinforcement learning model is trained and the model parameters are adjusted to obtain a trained deep reinforcement learning model; Input the workshop system parameters and scheduling objectives into the trained deep reinforcement learning model to obtain a visual recommendation plan.

2. The method for generating a workshop scheduling plan based on deep reinforcement learning according to claim 1, characterized in that: Build the overall framework of the intelligent manufacturing system, including: Build the enterprise resource planning system, manufacturing execution system and underlying control system respectively; Based on the agent models in the enterprise resource planning system, the manufacturing execution system and the underlying control system, a negotiation mechanism between the agent models is established; Based on the negotiation mechanism between agent models and the constructed enterprise resource planning system, manufacturing execution system and underlying control system, the overall framework of the intelligent manufacturing system is obtained; Wherein, the enterprise resource planning system includes an enterprise agent; The manufacturing execution system includes a product agent and a central scheduling agent at each stage; The bottom control system includes a warehouse agent, a conveying agent, a processing agent and a maintenance agent, which are respectively responsible for the storage and retrieval of raw materials and finished products, the transportation of products, the processing and assembly of products and the maintenance of equipment.

3. The method for generating a workshop scheduling plan based on deep reinforcement learning according to claim 2, characterized in that: Based on the agent models in the enterprise resource planning system, the manufacturing execution system and the underlying control system, a negotiation mechanism between the agent models is constructed, including: Through the collection and analysis of information, the enterprise agent issues a production plan to the product agent. As the administrator, the product agent initiates task announcement information to the central scheduling agent of the warehouse agent, the transportation agent and the processing agent. By evaluating their capabilities, the product agent determines the type of agent that has won the bid. Each agent under the type of agent that has won the bid negotiates with the transportation agent to determine the transportation route. The central scheduling agent ranks the characteristics of each bidding agent according to predefined rules, selects the agent with the highest ranking, and sends a reward message to the central scheduling agent of the type to which the agent belongs for notification. At the same time, a directed reward message is sent to the first transportation agent on the route for notification to start transporting the product.

4. The method for generating a workshop scheduling plan based on deep reinforcement learning according to claim 1, characterized in that: The workshop scheduling environment is compiled based on the Markov chain, the deep reinforcement learning algorithm is improved, and the improved deep reinforcement learning model is constructed, including: Using the OpenAI Gym library, we built an integrated scheduling environment that interacts with deep reinforcement learning algorithms, and built a virtual workshop model based on the layout and equipment information of the actual production workshop; the virtual workshop model includes the location, attributes, and status information of various equipment, automated guided vehicles, and workpieces; Encode the workshop state into vector or matrix form so that it can be input into the deep reinforcement learning algorithm for processing; Conduct simulation experiments in the constructed integrated scheduling environment to verify the feasibility and effectiveness of deep reinforcement learning algorithms in production scheduling; Improve the deep reinforcement learning algorithm and build an improved deep reinforcement learning model.

5. The method for generating a workshop scheduling plan based on deep reinforcement learning according to claim 1 or 4, characterized in that: Improve the deep reinforcement learning algorithm and build an improved deep reinforcement learning model, including: Construct the basic framework of multi-agent proximal strategy optimization algorithm; The ε-greedy strategy is added to the action selection strategy part of the multi-agent proximal strategy optimization algorithm to prevent the algorithm from falling into the local optimal solution, and an improved deep reinforcement learning model is obtained.

6. The method for generating a workshop scheduling plan based on deep reinforcement learning according to claim 1, characterized in that: After obtaining the trained deep reinforcement learning model and before inputting the workshop system parameters and scheduling targets into the trained deep reinforcement learning model, it also includes: embedding the trained deep reinforcement learning model into a Web application.

7. The method for generating a workshop scheduling plan based on deep reinforcement learning according to claim 6, characterized in that: Embed the trained deep reinforcement learning model into the web application, including: Implement the core logic of the scheduling algorithm in the backend; Create a RESTful API that enables the front end to access the scheduling function through HTTP requests; Create a user interface that allows users to enter the parameters required for scheduling; Ensure that the front-end and back-end can exchange data through the network. The front-end sends requests through the Fetch API, and the back-end returns the scheduling results. Deploy the back-end and front-end applications to the cloud platform or local server to ensure that the front-end and back-end can communicate normally.

8. The method for generating a workshop scheduling plan based on deep reinforcement learning according to claim 1, characterized in that: Input the workshop system parameters and scheduling objectives into the trained deep reinforcement learning model, including: Pre-process the order and equipment data to be scheduled to obtain the processed order and equipment data set; The processed order and equipment data sets are input into the trained deep reinforcement learning model for workshop scheduling.

9. A workshop scheduling scheme generation system based on deep reinforcement learning, characterized in that: include: The dataset module, system building module, deep reinforcement learning model module and data processing module are connected in sequence; The data set module is used to obtain a data set and divide the data set into a training set and a validation set; The system building module is used to build the overall framework of the intelligent manufacturing system; The deep reinforcement learning model module is used to compile the workshop scheduling environment based on the Markov chain, and improve the deep reinforcement learning algorithm to construct an improved deep reinforcement learning model; The data processing module is used to train the improved deep reinforcement learning model according to the training set and the verification set, adjust the model parameters, and obtain a trained deep reinforcement learning model; input the workshop system parameters and scheduling goals into the trained deep reinforcement learning model to obtain a visual recommendation plan.

10. The workshop scheduling scheme generation system based on deep reinforcement learning according to claim 9 is characterized in that: The overall framework of the system includes an order center module, an equipment center module and a dispatch center module; In the Order Center module, users can manage production orders, including importing complete production orders, adding, deleting and modifying order information, and scheduling orders; In the equipment center module, users can view various real-time status information of equipment, including energy consumption, logistics information and equipment failure conditions, and add, delete or modify basic equipment information; In the scheduling center module, users can make a series of settings and optimizations to generate the best production scheduling plan.

Images