A method for planning the shortest production time of a workshop assembly line
By building a deep reinforcement learning network algorithm library and iterative optimization training, combined with multiple deep reinforcement learning models, the problem of single deep reinforcement learning technology in workshop assembly line production planning is solved, the shortest planning of workshop assembly line production time is realized, and learning efficiency is improved and the advantages of decision-making strategies.
Patent Information
- Application Number
- CN202211200517.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-09-29
AI Technical Summary
In the existing workshop assembly line production planning methods, the deep reinforcement learning technology is single, and it is not effective to combine the advantages of multiple deep reinforcement models and the advantages of decision-making results, making it difficult to obtain the optimal solution with the shortest production time in workshop assembly line scheduling.
By collecting workshop assembly line task data indicators, a deep reinforcement learning network algorithm library is built, and each deep reinforcement learning network is trained independently until the loss function converges, the optimal decision strategy is obtained, the strategy with the shortest production time is selected, and the strategy is saved on other networks Iterative optimization training is carried out in the memory database to improve learning efficiency.
The advantages of multiple deep reinforcement learning models have been combined, the efficiency of workshop assembly line production time planning has been improved, the better learning experience data has been obtained, the learning efficiency of deep reinforcement learning network has been improved, and the shortest decision-making strategy for production time has been obtained. .
Smart Images

Figure CN115564223B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of workshop assembly line planning and relates to deep reinforcement learning technology, and specifically is a method for planning the shortest production time of a workshop assembly line. Background Art
[0002] The flow shop scheduling problem is one of the classic combinatorial optimization problems. It is difficult to find the optimal solution and the computational complexity is high. In order to improve the production efficiency of the system, effective and efficient manufacturing and scheduling technology solutions must be used. Many researchers have proposed a variety of algorithms for this problem. The exact algorithm can get the optimal solution, but the solution time is often too long and is not suitable for large-scale problems; the NEH heuristic algorithm is currently a more commonly used flow shop scheduling algorithm, which can be used to solve the optimization problem with the minimum processing time as the goal. It is difficult for heuristic algorithms to guarantee the quality of the solution. The algorithm needs to be redesigned for different types of problems. Therefore, heuristic algorithms have limitations in universality. In recent years, artificial intelligence has shown great potential in solving combinatorial optimization problems. Vinyals et al. proposed a PointerNetwork based on the Seq2Seq model, which can solve the problem that the number of output targets depends on the length of the input sequence and map one sequence to another. The traditional Seq2Seq model consists of three parts: encoder, decoder, and connected intermediate state vector. The encoder encodes the input into a fixed-length vector by learning, and the decoder outputs the content through the intermediate vector. However, as the sequence becomes longer, the RNN will have gradient diffusion, resulting in a large amount of information loss. PointerNetwork introduces an attention mechanism that allows the network to focus on the most valuable information and assign higher weights to it, effectively improving the model effect and solving the problem of information loss. The output of the network is a permutation of the input sequence, and it has achieved good results in the traveling salesman problem and the knapsack problem;
[0003] However, the deep reinforcement learning used in most workshop assembly line production planning problems is often relatively simple, failing to combine the advantages of multiple deep reinforcement models and the excellence of decision-making results;
[0004] Therefore, a method for planning the shortest production time of workshop assembly lines is proposed. Summary of the invention
[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a workshop assembly line shortest production time planning method, which determines the various data indicators of the workshop assembly line tasks; collects the deep reinforcement learning network algorithm to form an algorithm library; and allows each deep reinforcement learning network to be trained independently until the loss function converges; then inputs the actual workshop production data into each deep reinforcement learning network to obtain the corresponding optimal decision strategy; selects the decision strategy with the shortest production time from all decision strategies, saves the decision strategy in the memory library of all other deep reinforcement learning, and retrains; continuously brings better learning experience data to each deep reinforcement learning network, and improves the learning efficiency of the deep reinforcement learning network.
[0006] To achieve the above object, according to an embodiment of the first aspect of the present invention, a method for planning the shortest production time of a workshop assembly line is provided, comprising the following steps:
[0007] Step 1: Collect various data indicators of workshop assembly line tasks;
[0008] Step 2: Determine the deep reinforcement learning network algorithm library;
[0009] Step 3: According to the data indicators of the assembly line tasks and the shortest production time target, determine the corresponding target loss function, value function and update strategy for each deep learning network algorithm;
[0010] Step 4: Each deep reinforcement learning network algorithm iteratively trains the deep reinforcement learning network according to its own strategy update method;
[0011] Step 5: For the actual production demand of the workshop assembly line, convert the data indicators of the production demand into the input form received by each deep reinforcement learning algorithm;
[0012] Step 6: Use the production demand data indicators as input to each trained deep reinforcement learning network to obtain the optimal strategy of each deep reinforcement learning network;
[0013] Step 7: Calculate the production time required for the optimal strategy generated by each deep reinforcement learning network, and obtain the decision strategy with the shortest production time;
[0014] Step 8: Take the decision strategy with the shortest production time and the decision strategy set generated during the training process of each deep reinforcement learning network as historical experience data, and retrain each deep reinforcement learning network;
[0015] Step 9: Repeat steps 5 to 8 to continuously iterate and optimize the training of each deep reinforcement learning network; and each actual production demand is executed using the decision-making strategy with the shortest production time;
[0016] The data indicators of the workshop assembly line task are the basic data required to solve the shortest production time of the assembly line; in a preferred embodiment, the basic data may include a set of workpieces and a set of intelligent processing equipment; each workpiece contains a limited number of processes; each process has a limited number of corresponding intelligent processing equipment; each workpiece has a known processing time in each process; and each workpiece has a preset delivery deadline, that is, each workpiece needs to complete all processes before the delivery deadline; for workpieces that fail to complete all processes before the delivery deadline, a penalty value is preset;
[0017] The deep reinforcement learning network algorithm library collects offline deep reinforcement learning algorithms; the offline deep reinforcement learning algorithm is a deep reinforcement learning algorithm that learns by sampling and observing historical decision data;
[0018] Among them, the target loss function is the loss function generated by each deep reinforcement learning network in the deep reinforcement learning network algorithm library according to its own network characteristics, the data indicators of the workshop assembly line tasks and the final goal; the value function is the reward function that can be obtained by making a decision for each deep reinforcement learning network according to its own network characteristics and the actual reward and punishment data of the workshop assembly line; at the same time, the decision strategy of each deep reinforcement learning network makes each decision according to its own network characteristics for exploration; each offline deep reinforcement learning network updates the value function and loss function after making each decision, and saves each decision in the memory library of the corresponding network;
[0019] The training process of each deep reinforcement learning network includes making each decision according to its own network decision-making method, and synchronously updating the value function and the loss function; with the goal of minimizing the loss function value, iterative training is continuously performed until the loss function value converges and the training ends; it can be understood that when the loss function value converges, it can be considered that the deep reinforcement learning network has reached a local minimum;
[0020] When actual production is carried out in the production workshop, the production data indicators are used as input and input into each deep reinforcement learning network to obtain the corresponding decision strategy for each step; according to the execution steps of each decision strategy combination generated by deep reinforcement learning, the production time required for the decision strategy combination can be calculated; thus, the decision strategy with the shortest production time can be selected from all decision strategies; it can be understood that this decision strategy is better than the decision strategies generated by all other deep reinforcement learning networks; therefore, this decision strategy can be used as the object of learning observation for other deep reinforcement learning networks;
[0021] A decision quantity cycle N is preset based on actual experience; the actual production data of the workshop for each decision quantity cycle N times is used as a memory preservation cycle; the decision strategy with the shortest production time generated within a memory preservation cycle is merged and saved in the memory bank of each deep reinforcement learning network at the end of the memory preservation cycle for learning and observation of the deep reinforcement learning network.
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] The present invention designs a deep reinforcement learning network algorithm library, combines the advantages of different deep reinforcement learning models, and uses the best decision-making strategy in each decision as the learning target of other deep reinforcement learning networks; continuously brings better learning experience data to each deep reinforcement learning network, thereby improving the learning efficiency of the deep reinforcement learning network. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0025] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] like Figure 1 As shown, a method for planning the shortest production time of a workshop assembly line includes the following steps:
[0027] Step 1: Collect various data indicators of workshop assembly line tasks;
[0028] Step 2: Determine the deep reinforcement learning network algorithm library;
[0029] Step 3: According to the data indicators of the assembly line tasks and the shortest production time target, determine the corresponding target loss function, value function and update strategy for each deep learning network algorithm;
[0030] Step 4: Each deep reinforcement learning network algorithm iteratively trains the deep reinforcement learning network according to its own strategy update method;
[0031] Step 5: For the actual production demand of the workshop assembly line, convert the data indicators of the production demand into the input form received by each deep reinforcement learning algorithm;
[0032] Step 6: Use the production demand data indicators as input to each trained deep reinforcement learning network to obtain the optimal strategy of each deep reinforcement learning network;
[0033] Step 7: Calculate the production time required for the optimal strategy generated by each deep reinforcement learning network, and obtain the decision strategy with the shortest production time;
[0034] Step 8: Take the decision strategy with the shortest production time and the decision strategy set generated during the training process of each deep reinforcement learning network as historical experience data, and retrain each deep reinforcement learning network;
[0035] Step 9: Repeat steps 5 to 8 to continuously iterate and optimize the training of each deep reinforcement learning network; and each actual production demand is executed using the decision-making strategy with the shortest production time;
[0036] The data indicators of the workshop assembly line task are the basic data required to solve the shortest production time of the assembly line; in a preferred embodiment, the basic data may include a set of workpieces and a set of intelligent processing equipment; each workpiece contains a limited number of processes; each process has a limited number of corresponding intelligent processing equipment; each workpiece has a known processing time in each process; and each workpiece has a preset delivery deadline, that is, each workpiece needs to complete all processes before the delivery deadline; for workpieces that fail to complete all processes before the delivery deadline, a penalty value is preset;
[0037] It is understandable that since the shortest production time planning problem of the workshop assembly line is a problem that has been proven to be NP-hard; therefore, the traditional planning algorithm can often only obtain some approximate optimal solutions, but cannot obtain its true global optimal solution; at the same time, the workshop assembly line production planning problem is also a real-time decision-making problem with an ultimate production goal; therefore, using reinforcement learning to train and continuously approach the global optimal solution has become a hot topic of research; but reinforcement learning has decision-making capabilities, but is helpless with perception problems; and deep learning has strong perception capabilities, but lacks certain decision-making capabilities; therefore, there are already many deep reinforcement learning models that combine reinforcement learning and deep learning;
[0038] In a preferred embodiment, the deep reinforcement learning network algorithm library collects offline deep reinforcement learning algorithms; specifically, offline deep reinforcement learning algorithms may include DQN, DDQN, DPG, and SAC, etc.; the offline deep reinforcement learning algorithm is a deep reinforcement learning algorithm that learns by sampling and observing historical decision data;
[0039] Among them, the target loss function is the loss function generated by each deep reinforcement learning network in the deep reinforcement learning network algorithm library according to its own network characteristics, the data indicators of the workshop assembly line tasks and the final goal; the value function is the reward function that can be obtained by making a decision for each deep reinforcement learning network according to its own network characteristics and the actual reward and punishment data of the workshop assembly line; at the same time, the decision strategy of each deep reinforcement learning network makes each decision according to its own network characteristics for exploration; for example: in the DQN network, the decision strategy for each step is to randomly select according to a certain probability; each Offline deep reinforcement learning network updates the value function and loss function after making each decision, and saves each decision in the memory library of the corresponding network;
[0040] The training process of each deep reinforcement learning network includes making each decision according to its own network decision-making method, and synchronously updating the value function and the loss function; with the goal of minimizing the loss function value, iterative training is continuously performed until the loss function value converges and the training ends; it can be understood that when the loss function value converges, it can be considered that the deep reinforcement learning network has reached a local minimum;
[0041] When actual production is carried out in the production workshop, the production data indicators are used as input and input into each deep reinforcement learning network to obtain the corresponding decision strategy for each step; according to the execution steps of each decision strategy combination generated by deep reinforcement learning, the production time required for the decision strategy combination can be calculated; thus, the decision strategy with the shortest production time can be selected from all decision strategies; it can be understood that this decision strategy is better than the decision strategies generated by all other deep reinforcement learning networks; therefore, this decision strategy can be used as the object of learning observation for other deep reinforcement learning networks;
[0042] In a preferred embodiment, since the memory bank of the decision strategies generated by each deep reinforcement learning network itself is relatively large, a decision quantity cycle N is preset based on actual experience. The actual production data of the workshop for each decision quantity cycle N is used as a memory storage cycle. The decision strategy with the shortest production time generated within a memory storage cycle is merged and saved in the memory bank of each deep reinforcement learning network at the end of the memory storage cycle for learning and observation of the deep reinforcement learning network. It can be understood that by setting the decision quantity cycle N, the problem of frequent training of the deep reinforcement learning network is avoided.
[0043] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.
Claims
1. A method for planning the shortest production time of a workshop assembly line, characterized in that: The following steps are involved: Step 1: Collect various data indicators of workshop assembly line tasks; Step 2: Determine the deep reinforcement learning network algorithm library; Step 3: According to the data indicators of the assembly line tasks and the shortest production time target, determine the corresponding target loss function, value function and update strategy for each deep learning network algorithm; Step 4: Each deep reinforcement learning network algorithm iteratively trains the deep reinforcement learning network according to its own strategy update method; Step 5: For the actual production demand of the workshop assembly line, convert the data indicators of the production demand into the input form received by each deep reinforcement learning algorithm; Step 6: Use the production demand data indicators as input to each trained deep reinforcement learning network to obtain the optimal strategy of each deep reinforcement learning network; Step 7: Calculate the production time required for the optimal strategy generated by each deep reinforcement learning network, and obtain the decision strategy with the shortest production time; Step 8: Take the decision strategy with the shortest production time and the decision strategy set generated during the training process of each deep reinforcement learning network as historical experience data, and retrain each deep reinforcement learning network; Step 9: Repeat steps 5 to 8 to continuously iterate and optimize the training of each deep reinforcement learning network; and each actual production demand is executed using the decision-making strategy with the shortest production time.
2. A workshop assembly line shortest production time planning method according to claim 1, characterized in that: The data indicators of the workshop assembly line task are the basic data required to solve the shortest production time of the assembly line; the basic data include a set of workpieces and a set of intelligent processing equipment; each workpiece contains a limited number of processes; each process has a limited number of corresponding intelligent processing equipment; each workpiece has a known processing time in each process; and each workpiece has a preset delivery deadline, that is, each workpiece needs to complete all processes before the delivery deadline; for workpieces that fail to complete all processes before the delivery deadline, a penalty value is preset.
3. The method for planning the shortest production time of a workshop assembly line according to claim 1 is characterized in that: The deep reinforcement learning network algorithm library collects offline deep reinforcement learning algorithms; the offline deep reinforcement learning algorithm is a deep reinforcement learning algorithm that learns by sampling and observing historical decision data.
4. The method for planning the shortest production time of a workshop assembly line according to claim 1, characterized in that: The target loss function is the loss function generated by each deep reinforcement learning network in the deep reinforcement learning network algorithm library according to its own network characteristics, the data indicators of the workshop assembly line tasks and the final goal; the value function is the reward function that can be obtained by making a decision for each deep reinforcement learning network according to its own network characteristics and the actual reward and punishment data of the workshop assembly line; at the same time, the decision-making strategy of each deep reinforcement learning network makes each decision according to its own network characteristics for exploration; after making each decision, each Offline deep reinforcement learning network updates the value function and the loss function, and saves each decision in the memory library of the corresponding network.
5. The method for planning the shortest production time of a workshop assembly line according to claim 1, characterized in that: The training process of each deep reinforcement learning network includes making each decision according to its own network decision-making method, and synchronously updating the value function and loss function; with the goal of minimizing the loss function value, iterative training is continuously performed until the loss function value converges and the training ends.
6. A workshop assembly line shortest production time planning method according to claim 1, characterized in that: A decision quantity cycle N is preset based on actual experience; the actual production data of the workshop for each decision quantity cycle N times is used as a memory preservation cycle; the decision strategy with the shortest production time generated within a memory preservation cycle is merged and saved in the memory bank of each deep reinforcement learning network at the end of the memory preservation cycle for learning and observation of the deep reinforcement learning network.
