A manufacturing system reconfiguration planning method based on deep reinforcement learning

Abstract

The application discloses a manufacturing system reconfiguration planning method based on deep reinforcement learning, and the implementation process of the method is as follows: analyzing an existing reconfigurable machine tool and a product to be processed to obtain a machine tool configuration set and a product feature set to be processed; a Markov decision process is used to model a manufacturing system reconfiguration planning problem, a Markov model of the reconfiguration planning is formed, a state space and an action space are determined according to the machine tool configuration set and the product feature set to be processed, and a reward function and a transition probability matrix are determined according to a determined optimization target; based on the Markov model of the reconfiguration planning, a neural network of a DQN algorithm is trained by using sample data obtained through continuous interaction between a reconfiguration planning intelligent agent and an environment; and the trained DQN network is used to input workpiece features, and an optimization scheme of the manufacturing system reconfiguration planning is quickly solved. The application can solve the manufacturing system reconfiguration planning problem.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent manufacturing equipment industry, and particularly relates to a manufacturing system reconstruction planning method based on deep reinforcement learning. BACKGROUND

[0002] Traditional manufacturing systems in the intelligent manufacturing equipment industry often adopt fixed system configurations that can meet the processing needs of multiple workpieces at the same time during the design process, and the production capacity cannot be fully utilized. Unlike traditional rigid manufacturing systems, reconfigurable manufacturing systems (RMS) have adjustable structures that can be reconfigured at the system and device levels to achieve manufacturing system flexibility, with production capacity closely combined with processing needs. The first step in implementing RMS is to determine the manufacturing system reconstruction scheme. A reasonable reconstruction scheme can greatly improve the efficiency of manufacturing system reconstruction and reduce reconstruction costs, but the introduction of reconstruction factors increases the complexity of manufacturing system planning, and traditional planning methods cannot meet this need. Therefore, it is necessary to study manufacturing reconstruction planning and optimization technology.

[0003] In the process of implementing RMS, appropriate machine configurations need to be selected to form a manufacturing system according to the process plan requirements of multiple products. In RMS configuration design, a certain number of devices can be designed into multiple different layout schemes through position transformation, sequence transformation, or device self-reconstruction. For this reason, many scholars have conducted a large number of studies on the optimization design of manufacturing system reconstruction schemes. Youssef et al. proposed a novel RMS configuration selection method, which first determines near-optimal configuration candidate schemes using meta-heuristic algorithms, real-coded genetic algorithms, and tabu search algorithms, and then uses integer-coded genetic algorithms to determine the optimal scheme from the candidate schemes. Kumar et al. proposed a heuristic algorithm to determine a set of linear machine sequences that can simultaneously meet the processing needs of multiple products, and compared this scheme with schemes given by two classical layout design methods, proving the effectiveness of the heuristic algorithm in manufacturing system layout design. Lin and Yang proposed a machine selection method based on the analytic hierarchy process, which can select the most suitable machine from a set of potential available machines. Karim and Karmaker combined the analytic hierarchy process and the technique for order preference by similarity to ideal solution (TOPSIS) to propose a machine selection decision support system. Juan et al. proposed a method combining tabu search algorithm and meme algorithm to solve the machine planning and layout integration problem, and simultaneously determined the machine layout scheme and workpiece processing path. Zhao Xiaobo et al. established an optimization problem based on a stochastic model to match product families with a set of feasible configurations, and used two different algorithms to solve the problem to achieve configuration selection of RMS in the design stage. Liu Yankai et al. proposed a greedy algorithm based on matroids to achieve fast RMS layout optimization design, and verified it through the reorganization practice of a factory.

[0004] Based on the existing reinforcement learning scheduling method and combined with the characteristics of the reconfigurable manufacturing system, the existing technical defects are summarized as follows:

[0005] (1) The intelligent level is not high. In the existing research, the analytic hierarchy process, genetic algorithm, heuristic algorithm or comprehensive consideration are often used to solve the design problem of manufacturing system reconfiguration scheme, but human intervention and decision-making are often required at the end, and autonomous decision-making cannot be realized. Therefore, a more intelligent optimization algorithm needs to be explored.

[0006] (2) The accuracy and optimization efficiency of the design scheme are too low. At present, due to the personalized and diversified surge of product demand and the intensification of production environment uncertainty, the states that may appear in the production process increase exponentially, and the traditional method lacks the ability to quickly handle such problems and cannot accurately obtain the optimal design scheme that meets the demand. SUMMARY

[0007] Therefore, the present application provides a manufacturing system reconfiguration planning method based on deep reinforcement learning, which can solve the manufacturing system reconfiguration planning problem.

[0008] The present application is realized by the following technical scheme:

[0009] A manufacturing system reconfiguration planning method based on deep reinforcement learning, the implementation process of which is as follows:

[0010] Step one, analyze the environment: analyze the existing reconfigurable machine tool and the product to be processed, obtain the machine tool configuration set and the product feature set to be processed;

[0011] Step two, Markov process modeling is carried out, and the four-tuple <S, A, R, P> of reconfiguration planning process is determined: the manufacturing system reconfiguration planning problem is modeled by using Markov decision process, the Markov model of reconfiguration planning is formed, the state space S and the action space A are determined according to the machine tool configuration set and the product feature set to be processed, and the reward function R and the transition probability matrix P are determined according to the determined optimization target, i.e. the minimum production cost as the optimization target of manufacturing system reconfiguration planning;

[0012] Step three, deep reinforcement learning, i.e. deep neural network training: based on the Markov model of reconfiguration planning, the neural network of DQN algorithm is trained by using the sample data obtained by the interaction between the reconfiguration planning agent and the environment;

[0013] Step four, obtaining the optimization scheme of reconfiguration planning based on DQN: using the trained DQN network, inputting the workpiece features, quickly solving and obtaining the optimization scheme of manufacturing system reconfiguration planning.

[0014] Further, in step two, the state space S of the Markov model is the environment state, S = (current available machine set M, workpiece type E, workpiece feature completion status F, current machine configuration C);

[0015] The current available machine set M represents a set of machine tools that can be used directly without reconfiguration adjustment, and the current available machine set M at time t is t The mathematical expression is:

[0016]

[0017] In the formula, indicates that the i-th feature of the workpiece is processed by the n-th machine tool of the j-th machine tool; m is the number of machine tools; ij

[0018] The workpiece type E = {e | 1, 2, 3, …, n}, n is the number of workpieces;

[0019] The workpiece feature completion status F indicates that x i is defined as the feature completion status of a workpiece i, and "0" indicates that the feature has not been processed; "1" indicates that the feature has been processed, i.e. x i = {0, 1}; The feature completion status F is in the form of a "set", which represents the processing status of all features of a workpiece, and F t at time t is:

[0020] F t = {x1,x2,x3,…,x n} (2)

[0021] x n is the feature completion status of the n-th workpiece, and when F t = {1, 1, 1, …, 1}, it indicates that all features of the current workpiece have been processed;

[0022] The mathematical expression of the current machine configuration C t at time t is:

[0023]

[0024] Based on the above description, the state space s t at time t can be represented as:

[0025]

[0026] Further, in step two, the action space A of the Markov model includes the current available machine set M and all machine configurations that can be obtained by reconfiguring M; The action a t at time t can be represented as: ​

[0027]

[0028] This indicates that at time t, the agent selected machine tool n of type j according to the policy. ij The i-feature of the shaped workpiece;

[0029] Based on equation (5), the action space can be represented as:

[0030]

[0031] In the formula, and This indicates that machine tool 1 has two configurations, n1 and n2, corresponding to the two features k1 and k2 of the workpiece that can be processed, and so on. This indicates that machine tool m has 1…n m These configurations correspond to workpieces that can be processed, k1…k. m One characteristic.

[0032] Furthermore, in step two, the reward function R of the Markov model is set with the objective of minimizing production costs;

[0033] Assume the current environmental state is as follows: ① The available machine tool group M is known; ② The feature completion status F of the workpiece is known, and the next machine tool configuration will be selected to process the next feature to be processed; ③ The workpiece is in... The configuration position, i.e., the location where feature i has been processed, is at machine tool j. The reward function r at time t is defined. t for:

[0034]

[0035] In equation (7), i′ represents the next feature to be processed. The target machine tool configuration required for machining feature i′;

[0036] For use The machining cost required for the configuration machining feature i′; d(j,j′) is the transportation cost required to move the workpiece from machine tool j to machine tool j′.

[0037] The reconstruction cost can be divided into four cases, as shown in equation (8):

[0038]

[0039] When j = j′, When reconfiguration occurs at the current machine tool, reconfiguration costs are incurred; when j≠j′, When j ≠ j′, the agent directly selects the machine tool in M ​​without reconfiguration, meaning the reconfiguration cost is zero; When j = j′, another machine tool will be selected, and the machine tool will need to be reconfigured, incurring reconfiguration costs; When machining two features i and i′ on the current machine tool, this situation is not allowed to occur;

[0040] The intelligent agent forms a trajectory [s] during the interaction with the environment. t ,a t ,r t ,s t+1 ,a t+1 ,r t+1 ,…,s T Minimizing production costs is equivalent to maximizing the cumulative reward obtained from multiple interaction trajectory sequences, i.e., the reward function R, as shown in equation (9):

[0041]

[0042] In the formula, T represents the total time of interaction between the agent and the environment.

[0043] Furthermore, in step two, each element in the transition probability matrix P of the Markov model is 1.

[0044] Furthermore, in step three, deep reinforcement learning is performed, i.e., deep neural network training:

[0045] The agent employs a greedy-exploration algorithm, specifically the ε-greedy algorithm, to select actions. This algorithm can be categorized into random selection and selection based on the Q-value. Before selecting an action, a random number k is generated. If k is less than the greedy value ε, a random action is selected. If k is greater than or equal to ε, the action is selected based on a = argmaxQ. * (s,a) performs a greedy action selection, where a is the action and Q is the value of Q. * Let s be the maximum Q value and s be the state.

[0046] Furthermore, in step three, deep reinforcement learning is performed, i.e., deep neural network training:

[0047] The experience playback mechanism is used to record every "experience" that the agent goes through. t ={s t ,a t ,r t ,s t+1 All of them are placed in the "experience pool", denoted by "D", forming D = {e1, e2, ..., e}. t}, e t Let t be the experience at time t, and D be the experience set; during training, a "group" is randomly sampled from D each time to update the Q network.

[0048] Further, in step three, deep reinforcement learning is performed, i.e., deep neural network training is performed:

[0049] In the Q network updating process, in order to make the loss function L=(y-Q(s,a,θ)) 2 Minimum, gradient descent method is used to iterate the loss function L about the neural network parameters θ, wherein Q(s,a,θ) represents the output of the current neural network, which is used to evaluate the value function of the current state action pair; y is the output of the target network;

[0050] Every π steps, the neural network parameters θ' of the real-time updated Q network are replaced with the neural network parameters θ of the fixed target Q network, i.e., θ' = θ.

[0051] Beneficial effects:

[0052] (1) The design of the multi-workpiece manufacturing system reconfiguration planning scheme needs to consider the optimization of the flow of multiple workpieces between limited reconfigurable machine tools, which includes workpiece transportation, machine tool processing and machine tool reconfiguration, and the sequence relationship among the three is the key to determine whether the reconfiguration scheme is efficient and reasonable; the present application aims at the manufacturing system reconfiguration planning problem, firstly constructs a Markov decision process model for the reconfiguration planning problem, and clearly defines the reconfiguration planning state space, action space and reward function, then introduces deep reinforcement learning to iteratively optimize the reconfiguration planning process to obtain the maximum cumulative return, i.e., applies the DQN algorithm to optimize and solve the problem model, and further obtains the optimal scheme of the manufacturing system reconfiguration planning; finally, the characteristics of deep reinforcement learning in learning process are used to continuously interact with the environment and obtain feedback to optimize its own behavior, thereby improving the effectiveness and optimization efficiency of the reconfiguration planning scheme.

[0053] (2) The state space S=(current available machine tool group M, workpiece type E, workpiece feature completion condition F, current machine tool configuration C) of the present application completely considers the environmental factors that need to be considered in the reconfigurable manufacturing system, and provides a basis for the subsequent Markov process modeling of the action space A, reward function R and transition probability matrix P.

[0054] (3) The reward function R of the Markov model of the present application is set to minimize the production cost, the setting of the reward function determines the direction of the algorithm iteration, and is directly related to the realization of the optimization goal, therefore, the reward function is an important bridge between the engineering personnel and the optimization problem to be solved.

[0055] (4) When the deep neural network training is used, i.e., the deep Q network (DQN) algorithm is used for training, the experience replay mechanism is used to store the experience e t ={s t ,a t ,r ts t+1} are put into the "experience pool", denoted as D, to form D={e1, e2, …, e t}t, e t is the experience at time t, and D is the experience set; during training, a "group" is randomly sampled from D each time to update the Q network, which can reduce the number of interactions between the agent and the environment, increase the relevance of the data, and make the training effect better.

[0056] (5) The deep neural network training is adopted, that is, when the deep Q network (DQN) algorithm is used for training, the gradient descent method is used to iterate the loss function L with respect to the neural network parameter θ, so that the loss function L=(y-Q(s,a,θ))2 can be minimized.

[0057] In summary, the reconfiguration planning scheme of the manufacturing system is considered at the beginning of the design, the bottom mathematical logic of the reconfiguration planning problem is abstracted and modeled, and the intelligent algorithm is used for solving, which can greatly improve the design and optimization efficiency of the reconfiguration planning scheme;

[0058] By analyzing the manufacturing system reconfiguration planning problem, it is judged that it has Markov property, so the Markov decision process (MDP) is used for modeling, and the state space, action space and behavior rules of the model can be flexibly increased, modified and closer to the real scene planning process;

[0059] In combination with the Markov decision process, the deep reinforcement learning (DQN) is further introduced to find the optimal reconfiguration planning scheme, and the intelligent level of the scheme design is improved. In addition, the DQN algorithm realizes the strategy optimization by training the neural network, only a limited number of parameters need to be stored, the memory occupation is reduced, and the optimization efficiency of the decision process is further improved;

[0060] A manufacturing system reconfiguration planning method based on deep reinforcement learning (DQN) is proposed, which interacts with the environment in a "trial and error" manner, and realizes strategy optimization by using the feedback reward signal, thereby improving the accuracy of the scheme. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The Markov decision process of the application;

[0062] Figure 2 The DQN algorithm implementation process of the application;

[0063] Figure 3 The machine tool layout and current configuration in the manufacturing unit in embodiment 2;

[0064] Figure 4 The machine tool configuration and feature mapping in embodiment 2;

[0065] Figure 5 Feature processing sequence chart for Example 2;

[0066] Figure 6 DQN algorithm convergence for Example 2;

[0067] Figure 7 Manufacturing system reconfiguration planning process for Example 2. DETAILED DESCRIPTION

[0068] The application will be described in detail below with reference to the accompanying drawings and examples.

[0069] Example 1:

[0070] This example provides a manufacturing system reconfiguration planning method based on deep reinforcement learning, and the basic implementation process is as follows:

[0071] Step one, analyze the environment: analyze the existing reconfigurable machine tool and the product to be processed, obtain the machine tool configuration set and the product feature set to be processed;

[0072] Step two, model the Markov process and determine the four-tuple <S, A, R, P> of the reconfiguration planning process: model the manufacturing system reconfiguration planning problem using Markov decision process, form the Markov model of the reconfiguration planning, determine the state space S and the action space A according to the machine tool configuration set and the product feature set to be processed, and determine the reward function R and the transition probability matrix P according to the determined optimization target, i.e., minimizing the production cost as the optimization target of the manufacturing system reconfiguration planning;

[0073] Step three, deep reinforcement learning, i.e., deep neural network (DQN) training: based on the Markov model of the reconfiguration planning, train the neural network of the DQN algorithm using the sample data obtained by the reconfiguration planning agent (hereinafter referred to as agent or Agent) and the environment interacting with each other;

[0074] Step four, obtain the optimization scheme of the reconfiguration planning based on DQN: use the trained DQN network to input the workpiece features, quickly solve and obtain the optimization scheme of the manufacturing system reconfiguration planning;

[0075] Thus, the implementation process of the manufacturing system reconfiguration planning method based on deep reinforcement learning is completed.

[0076] In step two, the specific process of modeling the Markov process is as follows:

[0077] Analysis of the manufacturing system reconfiguration planning problem reveals that the future state of the manufacturing system depends only on the current state of the workpieces and resources, and is independent of the past state, exhibiting Markov properties. Therefore, Markov decision processes are suitable for modeling manufacturing system reconfiguration planning.

[0078] The Markov decision process is typically defined as a quadruple (S, A, R, P), reconstructing the interaction between the planning agent and the environment, as follows: Figure 1 As shown; where S refers to the state space, A to the action space, R to the reward function, and P to the transition probability matrix, the state space S, action space A, reward function R, and transition probability matrix P are described in detail below:

[0079] (1) State space S:

[0080] The state space is the environmental state. The state space of a reconfigurable manufacturing system needs to consider four main environmental factors: the currently available machine tool group, the workpiece type, the workpiece feature completion status, and the workpiece location. The workpiece location can be represented by the machine tool position. The machine tool position is fixed, and reconfiguration does not change the machine tool position. Therefore, the workpiece location can be directly represented by the machine tool configuration. Thus, the environmental state of a reconfigurable manufacturing system is represented by a quadruple, i.e., state space = (currently available machine tool group, workpiece type, workpiece feature completion status, current machine tool configuration), abbreviated as S = (M, E, F, C), where M is the currently available machine tool group, E is the workpiece type, F is the workpiece feature completion status, and C is the current machine tool configuration.

[0081] The currently available machine tool group M represents a set of machine tools that can be used directly without reconfiguration or adjustment. The currently available machine tool group M at time t... t The mathematical expression is:

[0082]

[0083] In the formula, The i-th feature of the workpiece is represented by the n-th feature of the j-th machine tool. ij Configuration processing; m is the number of machine tools; if selected, it does not belong to M t The machine tool is used for processing, which will require reconfiguration to adapt to feature processing and incur reconfiguration costs. At this time, M t It will also dynamically adjust as reconstruction events occur;

[0084] Workpiece type E = {e|1, 2, 3, ..., n}, where n is the number of workpieces. ij different;

[0085] The completion status F of the workpiece features is defined as follows: x i"0" means the feature has not been processed; "1" means the feature has been processed, i.e. x i = {0, 1}; the feature completion status F is in the form of "set", which is used to represent the processing status of all features of the workpiece, and the feature completion status F of the workpiece at time t is denoted as t

[0086] F t = {x1, x2, x3,..., x n} (2)

[0087] x n is the feature completion status of the workpiece n, when F t = {1, 1, 1,..., 1}, it means that all features of the current workpiece are processed;

[0088] The mathematical expression of the current machine tool configuration C t at time t is:

[0089]

[0090] Based on the above description, the state space at time t, i.e. s t , can be represented as:

[0091]

[0092] (2) Action space A:

[0093] The action space A is defined as: selecting the machine tool configuration required for processing the next feature of the workpiece; therefore, the action space A includes the current available machine tool set M and all machine tool configurations that can be reconstructed by M. The action a t at time t can be represented as:

[0094]

[0095] It shows that at time t, the Agent selects the n ij configuration of machine j to process the i feature of the workpiece according to the strategy.

[0096] Based on equation (5), the action space can be represented as:

[0097]

[0098] In the formula, and indicate that machine tool 1 has two configurations n1 and n2, which correspond to two features k1 and k2 that can be processed, and so on, indicate that machine tool m has 1...n m configurations, which correspond to k1...k m ​One characteristic.

[0099] (3) Reward function R:

[0100] The goal of reinforcement learning can be reduced to maximizing the expected sum of scalar rewards received by the agent (called the return). Therefore, the design of the reward function is a crucial bridge between engineers and the optimization problem to be solved; it must ensure that the designed reward function R satisfies both the agent's training requirements and the problem's objective. In this embodiment, the reward function R is designed with the goal of minimizing production costs. Assume the current environmental state is: ① The currently available machine tool group M is known; ② The feature completion status F of the workpiece is known, and the next machine tool configuration will be selected to process the next feature to be processed; ③ The workpiece is in... The configuration position, i.e., the position where feature i has been processed, is located at machine tool j. Define the reward function r at time t. t for:

[0101]

[0102] In equation (7), i′ represents the next feature to be processed. The target machine tool configuration required for machining feature i′;

[0103] For use The machining cost required for the configuration machining feature i′; since the workpiece will incur transportation costs when moving between different machine tools, d(j,j′) is the transportation cost required to move the workpiece from machine tool j to machine tool j′;

[0104] The reconstruction cost can be divided into four cases, as shown in equation (8):

[0105]

[0106] When j = j′, When reconfiguration occurs at the current machine tool, reconfiguration costs are incurred; when j≠j′, When j ≠ j′, the agent directly selects the machine tool in M ​​without reconfiguration, meaning the reconfiguration cost is zero; When j = j′, another machine tool will be selected, and the machine tool will need to be reconfigured, incurring reconfiguration costs; At this time, if two features i and i′ are being processed on the current machine tool, this situation is not allowed to occur.

[0107] like Figure 1 As shown, the agent and the environment will form a trajectory [s] during the interaction process. t ,a t ,r t ,st+1 a t+1 r t+1 ,…,s T ], minimizing the production cost is equivalent to maximizing the cumulative return, i.e. the reward function R, obtained from multiple interactive trajectory sequences, as shown in equation (9) :

[0108]

[0109] In the formula, T is the total time of the interaction process between the agent and the environment.

[0110] (4) Transition probability matrix P:

[0111] The transition probability in the reconfigurable manufacturing system is always determined and always equal to 1, i.e. in state s t Take action a t , the next state s t+1 of the environment and the received immediate reward r t are determined, so each element in the transition probability matrix P is 1.

[0112] In step three, deep reinforcement learning is performed, i.e. the specific process of deep neural network (DQN) training is as follows:

[0113] The Markov decision process essentially builds a mathematical model of the interaction between the agent and the environment, and does not have the function of iterative optimization itself, so it is usually necessary to further introduce a deep reinforcement learning algorithm for training.

[0114] At present, the method of Q learning can be used for training. In Q learning, the values of each state-action pair are stored in the form of a table, and the optimal action is obtained by updating the table, but the disadvantage is that the memory occupancy is large and the processing efficiency is low when dealing with models with too many states.

[0115] At present, the deep neural network algorithm, i.e. the deep Q network algorithm (DQN), can also be used for training. The deep Q network algorithm (DQN) is widely used in the field of engineering. For example, Seunghoon et al. proposed a mold production planning algorithm based on DQN for the planning problem of injection mold production workshops in complex systems and dynamic environments and achieved effective practical results. Unlike Q learning, which stores Q values in a table, DQN estimates Q values using a neural network. The state and action are input into the neural network, and the Q value of the action is output after the neural network analysis, reducing the memory occupancy while also improving the output speed of the Q value. In this process, to ensure the accuracy of the neural network output results, the feedback data generated during the interaction between the agent and the environment will constantly train the neural network.

[0116] Therefore, this embodiment uses the Deep Q-Network (DQN) algorithm for training. The key points of the DQN algorithm implementation are as follows:

[0117] (1) Action Selection: The agent uses a greedy-exploration (ε-greedy) algorithm to select actions, which can be divided into two forms: random selection and selection based on the Q value. Before selecting an action, a random number k is generated. When k is less than the greedy value ε, random action selection is performed. When k is greater than or equal to ε, the action is selected according to a = argmaxQ. * (s,a)(where a is the action, Q) * Greedy action selection is performed (where Q is the maximum value and s is the state).

[0118] (2) Experience Replay: The experience replay mechanism records every "experience" experienced by the agent. t ={s t ,a t ,r t ,s t+1 All of them are placed in an "experience pool", denoted by "D", forming D = {e1, e2, ..., e...} t}, e t Let t be the experience at time t, and D be the set of experiences. During training, a "batch" (e.g., 10 experiences) is randomly sampled from D each time to update the Q network. Experience replay can reduce the number of interactions between the agent and the environment, increase the correlation of data, and make the training effect better.

[0119] (3) Gradient descent method: In the Q-network update process, in order to make the loss function L=(yQ(s,a,θ)) 2 To minimize this, gradient descent is used to iterate the loss function L with respect to the neural network parameters θ, where Q(s,a,θ) represents the current output of the neural network and is used to evaluate the value function of the current state-action pair; y is the output of the target network.

[0120] (4) Target network update: Every π steps, replace the fixed target Q network neural network parameter θ with the real-time updated Q network neural network parameter θ′, that is, let θ′=θ.

[0121] Figure 2 Table 1 illustrates the reconfiguration planning process of the manufacturing system trained using DQN. At time t, if the workpiece to be processed or its features exist, the system is trained and optimized using the DQN algorithm until no workpiece to be processed or its features exist, at which point the reconfiguration planning optimization process ends.

[0122] Table 1. Implementation process of DNQ algorithm

[0123]

[0124]

[0125] Example 2

[0126] This example is based on example 1, and shows the specific implementation of example 1 by showing the solution of a small-scale manufacturing system reconfiguration planning scheme.

[0127] The task of this small-scale problem can be described as: given a set of conditions, including a family of workpieces, existing resources and workpiece process constraints, and an optimization objective of minimizing production cost, solve the optimal manufacturing system reconfiguration planning scheme that meets the above conditions.

[0128] Step S1, analyze the environment and set the conditions:

[0129] (1) Analyze the basic information of the workpieces to be processed: the basic situation of a workpiece family is shown in Table 2, and the workpiece characteristics are numbered in digital form. Different workpieces are represented by the same number, and the workpieces in the workpiece family are processed in order from small to large.

[0130] Table 2: Workpieces to be processed and their characteristics

[0131]

[0132] (2) Process the workpieces: due to the requirements of the processing technology, the processing sequence of the characteristics of each workpiece needs to meet certain order, which can be represented by a "precedence diagram", as shown in Figure 5 S1 represents the 1st characteristic, and so on. S5 represents the 5th characteristic.

[0133] (3) Analyze the basic information of the existing machine tools: the existing machine tool situation is shown in Figure 3 There are 5 reconfigurable machine tools in a manufacturing cell, and the positions and current configurations of the machine tools are determined.

[0134] In the case of fixed machine tool positions, the manufacturing cell can meet the processing of different workpieces by machine tool configuration reconfiguration. The machine tool configuration is represented by where "C" is the machine tool code; represents the number of machine tools "ω" represents the ωth configuration of the machine tool. The total configuration of each reconfigurable machine tool is shown in Table 3:

[0135] Table 3: Machine tool configurations of each RMT machine tool

[0136]

[0137] Each machine tool configuration is designed to independently complete the machining of a specific feature; for example, configuration C21 alone can complete the machining of feature number 1. Therefore, there is a mapping relationship between machine tool configurations and workpiece features, such as... Figure 4 As shown. In the machine tool configuration Based on this, feature i was added to form a new machine tool configuration representation method. This indicates that feature i can be processed. Machine tool No. ω configuration. It is worth noting that the configuration... When processing different features a and b sequentially, it is necessary to perform a process from... arrive Reconstruction;

[0138] Based on the combined results of manufacturing cell layout and configuration-feature mapping, the current resource situation is shown in Table 4:

[0139] Table 4 Existing Resource Set

[0140]

[0141] Step S2: Perform Markov process modeling to determine the refactoring planning process quadruple.<S,A,R,P> :

[0142] (1) Determine the state space S and motion space A based on the basic information of the workpiece and machine tool in step S1;

[0143] (2) Determine the optimization objective: Minimize the production cost as the optimization objective of the refactoring planning problem, which is equivalent to the agent maximizing the cumulative reward R (i.e. the reward function), as shown in equation (9).

[0144] (3) Calculate the reward function r based on the determined optimization objective. t The setting of the reward function determines the direction of algorithm iteration and is directly related to the achievement of the optimization objective. The reward function r t The calculation is shown in formula (7), where the processing cost is... As shown in Table 5, the reconfiguration costs of various machine tool configurations are... As shown in Table 6, the workpiece handling cost d(j,j′) between machine tools is shown in Table 7.

[0145] Table 5. Workpiece processing costs

[0146]

[0147] Table 5 (continued) Workpiece processing costs

[0148]

[0149] Table 6(a) Reconfiguration costs among different configurations of RMT1

[0150] Reconstruction cost g C111 C112 C211 C511 C111 0 -41 -42 -50 C112 -36 0 -43 -42 C211 -33 -32 0 -49 C511 -41 -43 -39 0

[0151] Table 6(b) Reconstruction cost between each configuration of RMT2

[0152] Reconstruction cost g C221 C121 C322 C422 C423 C221 0 -37 -50 -50 -37 C121 -30 0 -50 -40 -49 C322 -31 -43 0 -35 -33 C422 -48 -40 -39 0 -47 C423 -45 -35 -37 -44 0

[0153] Table 6(c) Reconstruction cost between each configuration of RMT3

[0154]

[0155]

[0156] Table 6(d) Reconstruction cost between each configuration of RMT4

[0157] Reconstruction cost g C241 C341 C342 C541 C542 C241 0 -39 -47 -34 -41 C341 -33 0 -49 -35 -38 C342 -40 -35 0 -38 -40 C541 -38 -40 -34 0 -32 C542 -31 -42 -43 -46 0

[0158] Table 6(e) Reconstruction cost between each configuration of RMT5

[0159] Reconstruction cost g C451 C352 C552 C451 0 -47 -37 C352 -43 0 -46 C552 -40 -34 0

[0160] Table 7 Transport cost between machines

[0161] Carrying cost D RMT1 RMT2 RMT3 RMT4 RMT5 RMT1 0 -5 -7 -12 -10 RMT2 -5 0 -7 -11 -6 RMT3 -7 -7 0 -5 -6 RMT4 -12 -11 -5 0 -3 RMT5 -10 -6 -6 -4 0

[0162] Step S3, deep neural network (DQN) training is performed, and an optimized scheme of reconstruction planning is obtained:

[0163] The selection of algorithm parameters has a close influence on the quality of the solution. Therefore, after determining the conditions of the task in step S1 and the optimization target in step S2, the hyperparameters of the DQN algorithm need to be set, as shown in Table 8.

[0164] Table 8 DQN hyperparameter setting

[0165]

[0166]

[0167] The method is coded in Python language on PyCharm, and is run on a 3.20 GHz AMD Ryzen 7 5800H processor windows operating system PC platform. The goal of reinforcement learning is to maximize the cumulative return of the agent's selected actions. The cumulative return obtained by the agent in each round of the training process is counted, and the time spent is 2 min 49 s, and the algorithm training condition is as shown in Figure 6 .

[0168] Observation Figure 6It can be seen that, in the case of given model and algorithm parameters, the cumulative return of the agent is constantly rising in the training process, and reaches convergence after about 16000 training times, the maximum cumulative return value is -713, the reward corresponding to each action is [-70, -39, -99, -62, -59, -77, -76, -42, -105, -84], and the optimal strategy is:

[0169] π*=[manufacturing unit A: C131, C241, C341, C451; manufacturing unit B: C111, C552, C341; manufacturing unit C: C221, C541, C342;]

[0170] According to the strategy π*, the manufacturing system will reconfigure the machining requirements of workpiece 1, workpiece 2 and workpiece 3, and sequentially form virtual manufacturing units A, B and C, as shown in the following figure. Figure 7 Take the machining of workpiece 2 in the virtual manufacturing unit B as an example for illustration. First, workpiece 2 is machined by C111 of No. 1 machine tool to complete the machining of No. 1 feature; then the workpiece is transported to No. 5 machine tool, and No. 5 machine tool is reconfigured (from C451 to C552), and workpiece 2 is machined by C552 to complete the machining of No. 5 feature; finally, the workpiece is transported to No. 4 machine tool, and is machined by C341 to complete the machining of No. 3 feature; thus, the virtual manufacturing unit B completes the machining of workpiece 2.

[0171] In summary, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A manufacturing system reconfiguration planning method based on deep reinforcement learning, characterized in that, The implementation process of this method is as follows: Step 1: Analyze the environment: Analyze the existing reconfigurable machine tools and products to be processed to obtain the set of machine tool configurations and the set of product features to be processed; Step 2: Perform Markov process modeling to determine the four-tuple of the refactoring planning process.<S,A,R,P> The Markov decision process is used to model the manufacturing system reconfiguration planning problem, forming a Markov model for reconfiguration planning. The state space S and action space A are defined based on the machine tool configuration set and the product feature set to be processed. The reward function R and the transition probability matrix P are determined based on the determined optimization objective, namely minimizing production cost as the optimization objective of manufacturing system reconfiguration planning. The state space S of the Markov model is the environment state. The state space S = (currently available machine tool group M, workpiece type E, workpiece feature completion status F, current machine tool configuration C). The currently available machine tool group M represents a set of machine tools that can be used directly without reconfiguration or adjustment. The currently available machine tool group M at time t... t The mathematical expression is: In the formula, The i-th feature of the workpiece is represented by the n-th feature of the j-th machine tool. ij Configuration processing; m is the number of machine tools; Workpiece type E = {e|1, 2, 3, ..., n}, where n is the number of workpieces; The completion status F of the workpiece features is defined as follows: x i To represent the completion status of a feature of a workpiece i, "0" indicates that the feature has not yet been processed; "1" indicates that the feature has been processed, i.e., x i = {0, 1}; The feature completion status F is in "set" form, used to characterize the processing status of all features of the workpiece, and denoted as the workpiece feature completion status F at time t. t for: F t ={x1,x2,x3,…,x n } (2) x n For the feature completion status of workpiece n, when F t When the value is {1,1,1,…,1}, it means that all features of the current workpiece have been processed. Current machine tool configuration C at time t t The mathematical expression is: Based on the above description, the state space at time t, i.e., s t It can be represented as: The action space A of the Markov model includes the currently available machine tool group M and all machine tool configurations that can be reconstructed from M; the action a at time t t It can be represented as: This indicates that at time t, the agent selected machine tool n of type j according to the policy. ij The i-feature of the shaped workpiece; Based on equation (5), the action space can be represented as: In the formula, and This indicates that machine tool 1 has two configurations, n1 and n2, corresponding to the two features k1 and k2 of the workpiece that can be processed, and so on. This indicates that machine tool m has 1…n m These configurations correspond to workpieces that can be processed, specifically k1…k. m One feature; The reward function R of the Markov model is set with the objective of minimizing production costs; Assume the current environmental state is as follows: ① The available machine tool group M is known; ② The feature completion status F of the workpiece is known, and the next machine tool configuration will be selected to process the next feature to be processed; ③ The workpiece is in... The configuration position, i.e., the location where feature i has been processed, is at machine tool j. The reward function r at time t is defined. t for: In equation (7), i′ represents the next feature to be processed. The target machine tool configuration required for machining feature i′; For use The machining cost required for the configuration machining feature i′; d(j,j′) is the transportation cost required to move the workpiece from machine tool j to machine tool j′. The reconstruction cost can be divided into four cases, as shown in equation (8): When j = j′, When reconfiguration occurs at the current machine tool, reconfiguration costs are incurred; when j≠j′, When j ≠ j′, the agent directly selects the machine tool in M ​​without reconfiguration, meaning the reconfiguration cost is zero; When j = j′, another machine tool will be selected, and the machine tool will need to be reconfigured, incurring reconfiguration costs; When machining two features i and i′ on the current machine tool, this situation is not allowed to occur; The intelligent agent forms a trajectory [s] during the interaction with the environment. t ,a t ,r t ,s t+1 ,a t+1 ,r t+1 ,…,s T Minimizing production costs is equivalent to maximizing the cumulative reward obtained from multiple interaction trajectory sequences, i.e., the reward function R, as shown in equation (9): In the formula, T is the total time of interaction between the agent and the environment; Step 3: Perform deep reinforcement learning, i.e., deep neural network training: Based on the Markov model of reconstruction planning, use the sample data obtained by the reconstruction planning agent in continuous interaction with the environment to train the neural network of the DQN algorithm. Step 4: Obtain the optimization scheme of the reconfiguration plan based on DQN: Using the trained DQN network and the input workpiece features, the optimization scheme of the manufacturing system reconfiguration plan is quickly obtained.

2. The manufacturing system reconfiguration planning method based on deep reinforcement learning as described in claim 1, characterized in that, In step two, each element in the transition probability matrix P of the Markov model is 1.

3. The manufacturing system reconfiguration planning method based on deep reinforcement learning as described in claim 1, characterized in that, In step three, deep reinforcement learning is performed, i.e., deep neural network training: The agent employs a greedy-exploration algorithm, specifically the ε-greedy algorithm, to select actions. This algorithm can be categorized into random selection and selection based on the Q-value. Before selecting an action, a random number k is generated. If k is less than the greedy value ε, a random action is selected. If k is greater than or equal to ε, the action is selected based on a = argmaxQ. * (s,a) performs a greedy action selection, where a is the action and Q is the value of Q. * Let s be the maximum Q value and s be the state.

4. The manufacturing system reconfiguration planning method based on deep reinforcement learning as described in claim 3, characterized in that, In step three, deep reinforcement learning is performed, i.e., deep neural network training: The experience playback mechanism is used to record every "experience" that the agent goes through. t ={s t ,a t ,r t ,s t+1 All of them are placed in the "experience pool", denoted by "D", forming D = {e1, e2, ..., e}. t }, e t Let be the experience at time t, and D be the experience set; during training, a "group" is randomly sampled from D each time to update the Q network.

5. The manufacturing system reconfiguration planning method based on deep reinforcement learning as described in claim 4, characterized in that, In step three, deep reinforcement learning is performed, i.e., deep neural network training: During the Q-network update process, in order to ensure that the loss function l=(yQ(s,a,θ)) 2 To minimize this, gradient descent is used to iterate the loss function L with respect to the neural network parameters θ, where Q(s,a,θ) represents the current output of the neural network, used to evaluate the value function of the current state-action pair; y is the output of the target network. Every π steps, the fixed target Q network's neural network parameters γ are replaced with the real-time updated Q network's neural network parameters θ′, i.e., θ′ = θ.

Images