An intelligent interactive decision-making method for discrete manufacturing systems

Abstract

The application discloses an intelligent interactive decision-making method for a discrete manufacturing system, and comprises the following steps: step 1, establishing a discrete manufacturing production scheduling optimization model and strategy for an actual application scene; step 2, training a scheduling strategy by using existing production data based on a deep reinforcement learning algorithm, and storing states with high rewards in a memory during the training process; step 3, updating the states by using prior knowledge in the memory; step 4, inputting the updated states into a deep reinforcement learning network to obtain corresponding rewards, and updating the memory according to the rewards; and step 5, repeatedly executing step 4 until model parameters converge, saving the model, and putting the model into an actual production scene. The application reduces the calculation amount of model training by using the memory mechanism, improves the speed of model training, and can iteratively output an optimal solution more quickly.

Description

Technical Field

[0001] This invention relates to the field of computer data science technology, and in particular to an intelligent interactive decision-making method for discrete manufacturing systems. Background Technology

[0002] With the development of the national economy, my country's discrete manufacturing industry has entered a new stage of industrialization. Today, discrete manufacturing, characterized by multi-variety, small-batch, and short-delivery models, features long and complex production processes, flexible and ever-changing production methods, and is susceptible to dynamic events. This poses challenges to the intelligent decision-making and rapid response capabilities of manufacturing enterprises. Therefore, how to achieve rapid allocation of production resources through intelligent decision-making to improve production capacity is a research hotspot in the field of discrete manufacturing, and the methods for achieving this have significant practical implications and application value.

[0003] Existing intelligent decision-making methods are mostly based on the assumption of a static production environment, where the information in the manufacturing workshop is completely known and unchanging. However, existing discrete manufacturing processes involve dynamic factors such as operator turnover, order insertions and reorders, and equipment failures, which can disrupt existing production scheduling plans, leading to production chaos and reduced efficiency. Therefore, an intelligent decision-making method capable of dynamically scheduling the production process is of great practical significance.

[0004] Currently, production scheduling decision-making methods based on genetic algorithms are widely used. This algorithm is a metaheuristic algorithm that simulates the Darwinian evolutionary process and can be applied to solving various production problems, yielding high-quality solutions. However, this type of algorithm has a huge computational load, which grows exponentially with the problem size, and the model training speed is too slow. In addition, it is only suitable for small to medium-sized dynamic scheduling problems and cannot solve large-scale dynamic scheduling problems, thus having low practical application value. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an intelligent interactive decision-making method for discrete manufacturing systems, which uses memory mechanisms to reduce the computational load of model training, improve the speed of model training, and can iterate to find the optimal solution more quickly.

[0006] To address the aforementioned technical problems, this invention provides an intelligent interactive decision-making method for discrete manufacturing systems, comprising the following steps:

[0007] Step 1: Establish a discrete manufacturing production scheduling optimization model and strategy for actual application scenarios;

[0008] Step 2: Based on the deep reinforcement learning algorithm, train the scheduling strategy using existing production data, and store the states with high rewards during the training process into memory;

[0009] Step 3: Update the state using prior knowledge stored in memory;

[0010] Step 4: Input the updated state into the deep reinforcement learning network to obtain the corresponding reward, and update the memory according to the reward;

[0011] Step 5: Repeat step 4 until the model parameters converge, save the model, and put it into the actual production scenario.

[0012] Preferably, in step 1, the specific steps for establishing a discrete manufacturing production scheduling optimization model and strategy for the actual application scenario are as follows: Based on the actual scheduling objective, the discrete manufacturing production problem is transformed into a sequential decision problem. Based on the objective, states, actions, rewards, exploration, and utilization strategies are defined. The objective is to maximize the product value produced within a production cycle, and the reward is the total value of the products within a production cycle. The states include production information such as the total types of products to be produced in the workshop, the production batches and processing stages of each type of product, the processing time and processing sequence of each product, the machine equipment status information such as the processing equipment allocated to the products, equipment load rate, and equipment in normal operation or failure, and the environmental status such as workshop temperature and humidity. The actions are to adjust the production scheduling strategies such as the product processing sequence and the allocated processing equipment. Then, based on the actual application scenario, an appropriate deep reinforcement learning algorithm framework is selected to establish a discrete manufacturing production scheduling optimization model.

[0013] Preferably, in step 2, the scheduling strategy is trained based on the deep reinforcement learning algorithm using existing production data, and the high-reward states during the training process are stored in memory. Specifically, production data for a certain production cycle is collected from the discrete manufacturing workshop that has been put into production as pre-training samples. A deep reinforcement learning algorithm is selected, and a discrete manufacturing production scheduling optimization model R is trained using the collected production data. The model takes the production data sampled from the current workshop as input and outputs a decision scheme for scheduling optimization of the workshop production line.

[0014] Define a memory D with capacity M to store state-reward pairs (s, r). The formula for the action strategy to obtain the corresponding reward r in the workshop state s is:

[0015]

[0016] In the above formula, Q(s) t ,a t Let s be the action value function, where s t This indicates the current state of the manufacturing workshop, a t This indicates the action scheduling to be adopted by the production workshop at the current moment. This action strategy has a probability of 1-ε for the current workshop state s. tRandomly select an action or evaluate the values ​​of each action q in the current state with probability ε according to the action value function Q, and select the current optimal action a for the current workshop state s. t Execute the optimal scheduling policy a and obtain the reward r. t and the next state s t+1 And update the memory.

[0017] Preferably, the memory update process is as follows:

[0018] (a) If memory D is not full and there is no memory with s t Similar s i (i = 1, 2, 3, ..., M), then enter the state-reward pair (s t ,r t The similarity calculation formula is as follows:

[0019]

[0020] In the above formula, Sim it s t With s i The similarity if Sim it If δ >, then s is considered i With s t Similarity, where δ is the threshold for judging similarity;

[0021] (b) If memory D contains a match with s t Similar s i If the state with the higher reward is selected for replacement, the update formula is as follows:

[0022]

[0023] In the above formula, r i It is state s i The corresponding reward, r t It is state s t The corresponding reward, "~" indicates similarity;

[0024] Finally, the memory is initialized, and the memory stores M workshop states with the highest reward. This means that, based on the actual production data collected, the manufacturing workshop has a higher reward in these M states and is closer to the production scheduling goal.

[0025] Preferably, in step 3, updating the state using prior knowledge in memory specifically involves: collecting production data from discrete manufacturing workshops with different production cycles than in step 1, using this data as training and updating samples, and obtaining the data in memory related to s through similarity calculation. t Most similar high-reward state s m , and st Perform a weighted sum to obtain the new workshop status. And As the input to neural network R; the specific formula is:

[0026]

[0027] In the above formula, For the updated new state, s t This is the current workshop status input, s m It comes from memory and s t The most similar high-reward state, where α and β are weight parameters, and s m The selection formula is:

[0028]

[0029] In the above formula, s i This is the workshop state that has been recorded in memory D.

[0030] The above process utilizes existing high-reward states in memory as prior knowledge to generate new states. These new workshop states are more likely to have high reward values, i.e., better production results (shorter production time, lower production costs, etc.), thereby improving the speed of model iteration convergence and reducing the training time of discrete manufacturing production scheduling optimization models.

[0031] Preferably, in step 4, the updated state is input into the deep reinforcement learning network to obtain the corresponding reward, and the memory is updated according to the reward. Specifically, the updated workshop state is input into the discrete manufacturing production scheduling optimization model R to further optimize the optimal strategy output by network R, and the memory is updated according to the reward corresponding to the state. The formula for obtaining the reward is:

[0032]

[0033] In the above formula, Let be the action value function, where Indicates the updated manufacturing workshop status, a t This represents the scheduling strategy adopted by the workshop, which assumes the current state s with a probability of 1-ε. t Randomly select an action or evaluate the values ​​of each action q in the current state with probability ε based on network Q, and select the current optimal action a for the current state. Perform the optimal action 'a' and receive a reward. and the next state s t+1 And update the memory.

[0034] Preferably, the memory update process is as follows:

[0035] (a) If memory D contains a... Similar s i The updated formula is as follows:

[0036]

[0037] (b) If memory D does not exist... Similar s i Then iterate through memory D and select the one with the smallest reward (s). min ,r min Replace the state with a better reward, and update the formula as follows:

[0038]

[0039] In the above formula, s i This is the state that has been recorded in memory D.

[0040] Preferably, in step 5, step 4 is repeatedly executed until the model parameters converge, the model is saved, and it is put into the actual production scenario. Specifically, step 4 is returned to step 3, and steps 3 and 4 are repeatedly executed to continuously optimize memory D. The production scheduling optimization model R is quickly updated by using the interaction between memory D and model R until the parameters of model R converge, indicating that the production scheduling optimization model R has reached the optimal decision model. Model R is then put into the manufacturing workshop, and the production scheduling arrangement of the workshop is intelligently decided by the production scheduling optimization model.

[0041] The beneficial effects of this invention are as follows: This invention fully utilizes manufacturing big data and gradually learns the optimal strategy through interaction with the environment, which meets the current demand for adaptive adjustment of workshop production status in the discrete manufacturing field. It realizes intelligent decision-making in workshop production, interactively learns and updates optimization strategies, ensures real-time decision control, and guarantees the stable operation of the workshop in a multi-disturbance environment. It overcomes the problems of complex models, high solution difficulty, and applicability only to small and medium-sized production scales in traditional precise modeling methods, and is applicable to large-scale dynamic scheduling decision problems. It incorporates a memory mechanism, which uses prior knowledge stored in memory to quickly adjust the direction of decision learning, enabling faster iteration to reach the optimal decision, improving the model training speed, reducing training costs, and ensuring that the model can be put into production more quickly and adjust parameters more quickly to adapt to the dynamically changing production environment. It has practical and economic value. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the deep reinforcement learning algorithm of the present invention.

[0043] Figure 2 This is a schematic diagram of the intelligent watering decision-making method for discrete manufacturing systems according to the present invention. Detailed Implementation

[0044] like Figure 1 and 2 As shown, an intelligent interactive decision-making method for discrete manufacturing systems includes the following steps:

[0045] Step 1: Establish a discrete manufacturing production scheduling optimization model and strategy for actual application scenarios;

[0046] Step 2: Based on the deep reinforcement learning algorithm, train the scheduling strategy using existing production data, and store the states with high rewards during the training process into memory;

[0047] Step 3: Update the state using prior knowledge stored in memory;

[0048] Step 4: Input the updated state into the deep reinforcement learning network to obtain the corresponding reward, and update the memory according to the reward;

[0049] Step 5: Repeat step 4 until the model parameters converge, save the model, and put it into the actual production scenario.

[0050] In step 1, the discrete manufacturing production scheduling optimization model and strategy are established specifically for practical application scenarios as follows: Based on the actual scheduling objectives, the discrete manufacturing production problem is transformed into a sequential decision-making problem. Based on the objectives, states, actions, rewards, exploration, and utilization strategies are defined. The objective is to maximize the product value produced within a production cycle, and the reward is the total product value within a production cycle. States include production information such as the total types of products to be produced in the workshop, the production batches and processing stages of each type of product, the processing time and sequence of each product, the status information of the processing equipment allocated to the products, equipment load rates, and whether equipment is operating normally or malfunctioning, as well as environmental conditions such as workshop temperature and humidity. Actions involve adjusting the production scheduling strategies, such as the product processing sequence and the allocated processing equipment. Then, based on the practical application scenarios, an appropriate deep reinforcement learning algorithm framework is selected to establish the discrete manufacturing production scheduling optimization model.

[0051] In step 2, based on the deep reinforcement learning algorithm, the scheduling strategy is trained using existing production data. Specifically, the state with high reward during the training process is stored in memory: production data of a certain production cycle is collected from the discrete manufacturing workshop that has been put into production as pre-training samples. The deep reinforcement learning algorithm is selected, and a discrete manufacturing production scheduling optimization model R is trained using the collected production data. The model takes the production data sampled from the current workshop as input and outputs a decision scheme for scheduling optimization of the workshop production line.

[0052] Define a memory D with capacity M to store state-reward pairs (s, r). The formula for the action strategy to obtain the corresponding reward r in the workshop state s is:

[0053]

[0054] In the above formula, Q(s) t ,a t Let s be the action value function, where s t This indicates the current state of the manufacturing workshop, a t This indicates the action scheduling to be adopted by the production workshop at the current moment. This action strategy has a probability of 1-ε for the current workshop state s. t Randomly select an action or evaluate the values ​​of each action q in the current state with probability ε according to the action value function Q, and select the current optimal action a for the current workshop state s. t Execute the optimal scheduling policy a and obtain the reward r. t and the next state s t+1 And update the memory.

[0055] The memory update process is as follows:

[0056] (a) If memory D is not full and there is no memory with s t Similar s i (i = 1, 2, 3, ..., M), then enter the state-reward pair (s t ,r t The similarity calculation formula is as follows:

[0057]

[0058] In the above formula, Sim it s t With s i The similarity if Sim it If δ >, then s is considered i With s t Similarity, where δ is the threshold for judging similarity;

[0059] (b) If memory D contains a match with s t Similar s i If the state with the higher reward is selected for replacement, the update formula is as follows:

[0060]

[0061] In the above formula, r i It is state s i The corresponding reward, r t It is state s t The corresponding reward, "~" indicates similarity;

[0062] Finally, the memory is initialized, and the memory stores M workshop states with the highest reward. This means that, based on the actual production data collected, the manufacturing workshop has a higher reward in these M states and is closer to the production scheduling goal.

[0063] In step 3, updating the state using prior knowledge in memory specifically involves: collecting production data from discrete manufacturing workshops with different production cycles than in step 1, using this data as training and updating samples, and obtaining the data in memory related to s through similarity calculation. t Most similar high-reward state s m , and s t Perform a weighted sum to obtain the new workshop status. And As the input to neural network R; the specific formula is:

[0064]

[0065] In the above formula, For the updated new state, s t This is the current workshop status input, s m It comes from memory and s t The most similar high-reward state, where α and β are weight parameters, and s m The selection formula is:

[0066]

[0067] In the above formula, s i This is the workshop state that has been recorded in memory D.

[0068] The above process utilizes existing high-reward states in memory as prior knowledge to generate new states. These new workshop states are more likely to have high reward values, i.e., better production results (higher economic benefits, lower production costs, etc.), thereby improving the speed of model iteration convergence and reducing the training time of discrete manufacturing production scheduling optimization models.

[0069] In step 4, the updated state is input into the deep reinforcement learning network to obtain the corresponding reward, and the memory is updated according to the reward. Specifically, the updated shop floor state is input into the discrete manufacturing production scheduling optimization model R to further optimize the optimal policy output by network R, and the memory is updated according to the reward corresponding to the state. The formula for obtaining the reward is:

[0070]

[0071] In the above formula, Let be the action value function, where Indicates the updated manufacturing workshop status, a tThis represents the scheduling strategy adopted by the workshop, which assumes the current state s with a probability of 1-ε. t Randomly select an action or evaluate the values ​​of each action q in the current state with probability ε based on network Q, and select the current optimal action a for the current state. Perform the optimal action 'a' and receive a reward. and the next state s t+1 And update the memory.

[0072] The memory update process is as follows:

[0073] (a) If memory D contains a... Similar s i The updated formula is as follows:

[0074]

[0075] (b) If memory D does not exist... Similar s i Then iterate through the memory and select the one with the smallest reward (s). min ,r min Replace the state with a better reward, and update the formula as follows:

[0076]

[0077] In the above formula, s i This is the state that has been recorded in memory D.

[0078] In step 5, step 4 is executed repeatedly until the model parameters converge. The model is then saved and put into the actual production scenario. Specifically, step 3 is returned, and steps 3 and 4 are executed repeatedly to continuously optimize memory D. The interaction between memory D and model R is used to quickly update the production scheduling optimization model R until the parameters of model R converge. This indicates that the production scheduling optimization model R has reached the optimal decision model. Model R is then put into the manufacturing workshop, where the production scheduling optimization model intelligently makes the production scheduling arrangements for the workshop.

Claims

1. An intelligent interactive decision-making method for discrete manufacturing systems, characterized in that, Includes the following steps: Step 1: Establish a discrete manufacturing production scheduling optimization model and strategy for actual application scenarios; Step 2: Based on the deep reinforcement learning algorithm, train the scheduling strategy using existing production data, and store the states with high rewards during the training process into memory; Production data for a certain production cycle is collected from discrete manufacturing workshops that are put into production and used as pre-training samples. A deep reinforcement learning algorithm is selected, and a discrete manufacturing production scheduling optimization model R is trained using the collected production data. The model takes the production data sampled from the current workshop as input and outputs a decision scheme for scheduling optimization of the workshop production line. Define a memory D with capacity M to store state-reward pairs (s, r). The formula for the action strategy to obtain the corresponding reward r in the workshop state s is: In the above formula, Q(s) t ,a t Let s be the action value function, where s t This indicates the current state of the manufacturing workshop, a t This indicates the action scheduling to be adopted by the production workshop at the current moment. This action strategy has a probability of 1-ε for the current workshop state s. t Randomly select an action or evaluate the values ​​of each action q in the current state with probability ε according to the action value function Q, and select the current optimal action a for the current workshop state s. t Execute the optimal scheduling policy a and obtain the reward r. t and the next state s t+1 And update the memory; The memory update process is as follows: (a) If memory D is not full and there is no memory with s t Similar s i (i = 1, 2, 3, ..., M), then enter the state-reward pair (s t ,r t The similarity calculation formula is as follows: In the above formula, Sim it s t With s i The similarity if Sim it If δ >, then s is considered i With s t Similarity, where δ is the threshold for judging similarity; (b) If memory D contains a match with s t Similar s i If the state with the higher reward is selected for replacement, the update formula is as follows: In the above formula, r i It is state s i The corresponding reward, r t It is state s t The corresponding reward, "~" indicates similarity; Finally, the memory is initialized, and the memory stores M workshop states with the highest reward. This means that, based on the actual production data collected, the manufacturing workshop has a higher reward in these M states and is closer to the production scheduling goal. Step 3: Update the state using prior knowledge stored in memory; Step 4: Input the updated state into the deep reinforcement learning network to obtain the corresponding reward, and update the memory according to the reward; Step 5: Repeat step 4 until the model parameters converge, save the model, and put it into the actual production scenario.

2. The intelligent interactive decision-making method for discrete manufacturing systems as described in claim 1, characterized in that, In step 1, the discrete manufacturing production scheduling optimization model and strategy are established specifically for the actual application scenario as follows: Based on the actual scheduling objective, the discrete manufacturing production problem is transformed into a sequential decision problem. According to the objective, states, actions, rewards, exploration, and utilization strategies are defined. The objective is to maximize the product value produced within a production cycle, and the reward is the total value of the products within a production cycle. The states include the total types of products that need to be produced in the workshop, the production batches and processing stages of each type of product, the processing time and processing sequence of each product, the processing equipment allocated to the products, the equipment load rate, the status information of equipment in normal operation or failure, and the environmental status such as workshop temperature and humidity. The actions are to adjust the production scheduling strategies such as the product processing sequence and the allocated processing equipment. Then, based on the actual application scenario, a deep reinforcement learning algorithm framework is selected to establish the discrete manufacturing production scheduling optimization model.

3. The intelligent interactive decision-making method for discrete manufacturing systems as described in claim 1, characterized in that, In step 3, updating the state using prior knowledge in memory specifically involves: collecting production data from discrete manufacturing workshops with different production cycles than in step 1, using this data as training and updating samples, and obtaining the data in memory related to s through similarity calculation. t Most similar high-reward state s m , and s t Perform a weighted sum to obtain the new workshop status. And As the input to neural network R; the specific formula is: In the above formula, For the updated new state, s t This is the current workshop status input, s m It comes from memory and s t The most similar high-reward state, where α and β are weight parameters, and s m The selection formula is: In the above formula, s i This is the workshop state that has been recorded in memory D.

4. The intelligent interactive decision-making method for discrete manufacturing systems as described in claim 1, characterized in that, In step 4, the updated state is input into the deep reinforcement learning network to obtain the corresponding reward, and the memory is updated according to the reward. Specifically, the updated shop floor state is input into the discrete manufacturing production scheduling optimization model R to further optimize the optimal policy output by network R, and the memory is updated according to the reward corresponding to the state. The formula for obtaining the reward is: In the above formula, Let be the action value function, where Indicates the updated manufacturing workshop status, a t This represents the scheduling strategy adopted by the workshop, which assumes the current state s with a probability of 1-ε. t Randomly select an action or evaluate the values ​​of each action q in the current state with probability ε based on network Q, and select the current optimal action a for the current state. Perform the optimal action 'a' and receive a reward. and the next state s t+1 And update the memory.

5. The intelligent interactive decision-making method for discrete manufacturing systems as described in claim 4, characterized in that, The memory update process is as follows: (a) If memory D contains a... Similar s i The updated formula is as follows: (b) If memory D does not exist... Similar s i Then iterate through the memory and select the one with the smallest reward (s). min ,r min Replace the state with a better reward, and update the formula as follows: In the above formula, s i This is the state that has been recorded in memory D.

6. The intelligent interactive decision-making method for discrete manufacturing systems as described in claim 1, characterized in that, In step 5, step 4 is executed repeatedly until the model parameters converge, the model is saved, and it is then deployed to the actual production scenario. Return to step 3, repeat steps 3 and 4, continuously optimize memory D, and use the interaction between memory D and model R to quickly update the production scheduling optimization model R until the parameters of model R converge, indicating that the production scheduling optimization model R has reached the optimal decision model. Put model R into the manufacturing workshop, and let the production scheduling optimization model intelligently decide the production scheduling arrangement of the workshop.

Images