Feature processing step sorting method based on dynamic mask and deep reinforcement learning
Through dynamic masking and deep reinforcement learning methods, Actor and Critic networks are built, which solves the problems of long calculation time of step sorting and poor optimization results in CNC machining, and realizes automatic learning and online optimization, meets processing constraints, and is suitable for a large number of Sorting of steps.
Patent Information
- Application Number
- CN202411011011.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
AI Technical Summary
The existing characteristic work step sorting methods in CNC machining have a long calculation time and limited optimization effect when processing a large number of work steps, and cannot learn sorting experience, which is easy to violate processing constraints.
Using a feature processing step sorting method based on dynamic mask and deep reinforcement learning, we use the Actor Action Network and Critic Reviewer Network to build a network of Actor Actioners and a Critic Reviewer Network, combining dynamic masks and local consumption, we automatically learn step sorting experience and optimize the step sorting order.
Obtain better sorting results within an acceptable calculation time, meet processing constraints, have online optimization capabilities, reduce iterative calculations, improve the automated learning ability of step sorting, and is suitable for a large number of step sorting problems.
Smart Images

Figure CN118966654A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent numerical control machining, and in particular to a feature machining step sequencing method based on dynamic masking and deep reinforcement learning. Background Art
[0002] The sorting result of CNC machining feature steps greatly affects the efficiency and quality of CNC machining. Reasonable feature steps should minimize fixture replacement, tool replacement and tool path idle travel while meeting machining constraints.
[0003] The methods for solving the feature step sorting problem can be divided into exact solution methods and approximate solution methods. The exact solution method finds the optimal solution by traversing the solution space. When the number of feature steps in the sorting problem to be solved is small, the exact solution method can obtain the ideal sorting result in a short time by searching the solution space. However, when the number of feature steps in the sorting problem is large, the size of the solution space grows exponentially, and the calculation time of the exact solution becomes unacceptable. Therefore, when solving the sorting problem with a large number of feature steps, approximate solution methods are usually used for sorting. Existing approximate solution methods are mainly heuristic algorithms, such as genetic algorithms and their variants. Although the approximate solution method calculates a better solution within an acceptable computing time, the degree of optimization of the step sorting is limited. And when facing the same sorting problem, it is still necessary to iterate and solve again, and the experience of previous sorting has not been learned. The approximate solution obtained may also violate the processing constraints.
[0004] Therefore, the three-dimensional process model is an important component of realizing digital integrated manufacturing in CNC machining. The automatic construction method of the process model is of great significance for reducing the cost of manual interaction and improving production efficiency. Summary of the invention
[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a feature processing step sorting method based on dynamic masking and deep reinforcement learning. The feature processing step sorting method based on dynamic masking and deep reinforcement learning can achieve better sorting effect within an acceptable computing time, and can automatically learn the sorting experience of feature steps.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0007] A feature processing step sequencing method based on dynamic mask and deep reinforcement learning includes the following steps.
[0008] Step 1. Construct a deep reinforcement learning model for feature step sorting: the input vector of the deep reinforcement learning model for feature step sorting is the feature processing step in an unordered state, and the output vector is the feature processing step with a processing order; the deep reinforcement learning model for feature step sorting includes an Actor network and a Critic network.
[0009] Step 2, constructing an input mode of the input vector: each feature processing step of the input vector has a set input mode; wherein the set input mode must be based on the geometric information and process chain information of the corresponding processing feature;
[0010] Step 3: Build the Actor network and the Critic network: The Actor network includes the Actor mapping layer, the loop encoder A1, and the loop decoder A2; where:
[0011] The Actor mapping layer is used to map the input vector constructed in step 2 to the input format of the recurrent encoder A1.
[0012] The recurrent encoder A1 is a first long short-term memory network LSTM, and the recurrent encoder A1 has two output vectors, namely a cell state vector c and a historical state vector h; wherein the historical state vector h is saved as a reference vector e.
[0013] The recurrent decoder A2 is a combination of a second long short-term memory network LSTM, a self-attention residual network AR, and a feature step selection network Pt.
[0014] The input vector of the second long short-term memory network LSTM includes two output vectors of the recurrent encoder A1, and the output vectors are the updated cell state vector c and the updated historical state vector h; wherein the updated historical state vector h is saved as a decoding vector d.
[0015] The input vectors of the self-attention residual network AR are the reference vector e and the decoding vector d, and the output vector is the updated decoding vector D.
[0016] The input vector of the feature step selection network Pt is the reference vector e, the updated decoding vector D, the dynamic mask m and the local cost f, and the output vector is the selection probability of each feature processing step in the input vector of the feature step sorting deep reinforcement learning model.
[0017] The critic network consists of a critic mapping layer, a recurrent encoder C1, and a prediction network.
[0018] The Critic mapping layer is used to map the input vector constructed in step 2 to the input format of the cyclic encoder C1.
[0019] The output of the cyclic encoder C1 is used as the input of the prediction network, and the output of the prediction network is the predicted total cost P of the feature processing steps after sorting.
[0020] Step 4, determine the dynamic mask m: determine the dynamic mask m of each feature processing step in the input vector according to the processing constraints that need to be met during the execution of the feature step; wherein, the dynamic mask m of the feature step that does not meet the processing constraints is assigned a maximum value; the dynamic mask m of the feature step that meets the processing constraints is assigned 0 or a minimum value.
[0021] Step 5, determine the local cost f: local cost refers to the cost when switching between two different local feature steps; for each feature processing step in the input vector of the deep reinforcement learning model for feature step sorting, the corresponding local cost f is calculated; among them, the local cost f includes fixture replacement cost, tool replacement cost and idle travel cost.
[0022] Step 6. Determine the deep reinforcement learning model for feature step sorting: Substitute the dynamic mask m determined in step 4 and the local cost f determined in step 5 into step 3 to obtain a complete Actor network; substitute the complete Actor network into step 1 to obtain a feature step sorting deep reinforcement learning model with a complete network; at this time, the actor network in the feature step sorting deep reinforcement learning model can calculate the selection probability of each feature processing step currently selected, and obtain the sorted feature processing steps through the cyclic encoder calculation n times, which forms the output vector of the feature step sorting deep reinforcement learning model.
[0023] Step 7, model training: Use the sample data set to train the feature step sorting deep reinforcement learning model determined in step 6 to obtain the trained feature step sorting deep reinforcement learning model; during the training process, calculate the actual total cost C of the sorted feature processing steps according to the output vector of the Actor network; obtain the predicted total cost P according to the Critic network; based on the predicted total cost P and the actual total cost C, on the one hand, according to the mean square error of PC, use the Adam optimizer to reduce the prediction error of the predicted total cost P in the Critic network; on the other hand, according to the CP error, use the Adam optimizer to optimize the feature step sorting results of the Actor network.
[0024] Step 8: Feature processing step sorting: Input a set of n unordered feature processing steps to be processed into the feature processing step sorting deep reinforcement learning model trained in step 7, and use the Actor network to obtain the feature processing steps with a processing order after sorting.
[0025] In step 2, the input mode is set to include feature position, feature step index, feature index, parent feature index, geometric pre-feature index, process pre-feature index, process chain index, feed direction index and machining tool index information.
[0026] In step 4, the processing constraints include processing resource constraints, geometric pre-feature constraints, process pre-feature constraints, and process chain step constraints; the method for determining the dynamic mask m includes the following steps:
[0027] Step 4-1, construct a mask set M: create a mask set M, the number of mask elements in the mask set M is equal to the total number n of all feature steps, and all mask elements are initialized to 0 or set to a minimum value.
[0028] Step 4-2: Determine the processing execution status: traverse the mask element value m of the jth feature step in the mask set M j ; Where 1≤j≤n; Determine whether the jth feature step is unprocessed; if so, give the mask element value m j Assign a maximum value; otherwise, go to step 4-3.
[0029] Step 4-3: Determine the execution status of geometric pre-feature processing, which specifically includes the following steps:
[0030] Step 4-3A: According to the input mode of step 2, determine the processing feature index a to which the j-th feature step belongs.
[0031] Step 4-3B, determine whether the processing feature index a has a geometric pre-feature or a process pre-feature that has not been processed; if so, it indicates that the geometric pre-feature constraint and the process pre-feature constraint are not satisfied, so the mask element value m is given j Assign a maximum value; otherwise, go to step 4-4.
[0032] Step 4-4: Determine the execution status of the process chain process steps, including the following steps:
[0033] Step 4-4A: According to the input mode of step 2, determine the process chain index b of the feature process chain to which the j-th feature step belongs.
[0034] Step 4-4B: Determine whether there is an unprocessed feature step before b in the feature process chain; if so, it indicates that the process chain step constraint is not met, so the mask element value m is given. j Assign a maximum value; otherwise, repeat steps 4-2 to 4-4.
[0035] In step 5, let the local cost of two different feature steps i and j be f i,j , then f i,j The calculation formula is:
[0036]
[0037] Where λ1, λ2 and λ3 are the fixture replacement cost coefficient, tool replacement cost coefficient and idle stroke cost coefficient, respectively, and all are set values.
[0038] is the variable of feature step i with respect to feed direction k; when feed direction k is the feed direction of feature step i, The value of is 1; otherwise, The value of is 0; where 1≤k≤p; p is the total number of possible feed directions.
[0039] is the variable of feature step j with respect to feed direction k; when feed direction k is the feed direction of feature step j, The value of is 1; otherwise, The value of is 0.
[0040] is the variable of feature step i about tool h; when tool h is the tool used in feature step i, The value of is 1; otherwise, The value of is 0; where 1≤h≤q; q is the total number of available tools.
[0041] is the variable of feature step j with respect to tool h; when tool h is the tool used in feature step j, The value of is 1; otherwise, The value of is 0.
[0042] dis(i,j) is the idle travel distance of the tool from the processing feature corresponding to feature step i to the processing feature corresponding to feature step j. In step 7, the calculation formula for the actual total cost C of the feature processing steps after sorting is:
[0043] C=c1+c2+c3
[0044] in:
[0045]
[0046] Where c1 is the fixture replacement cost, c2 is the tool replacement cost, and c3 is the tool path idle travel cost.
[0047] e ij is the order variable of the work step in the sorting result; when j=i+1, e ij =1; otherwise, e ij =0.
[0048] λ1=100, λ2=20, λ3=0.05.
[0049] In step 6, let the feature processing step selected for the x-1th time be feature processing step i, and the probability value of the feature processing step selected for the xth time being feature processing step j is p x,j , then p x,j The calculation formula is:
[0050]
[0051] in:
[0052] u' x,j =u x,j -m j -f i,j
[0053] In the formula, u' x,j It is the score value when feature processing step j is selected during the xth selection.
[0054] u x,j The score is calculated for the correlation between the reference vector e and the updated decoded vector D output by the self-attention residual network AR for the xth selection.
[0055] m j It is the corresponding mask value of feature processing step j.
[0056] In step 7, when training the model, the loss function formula of the Actor network is:
[0057] min(CP)logp(τ|fs)
[0058] in:
[0059]
[0060] Where τ is the feature step ranking result output by the current actor network.
[0061] fs is the disordered state feature processing step that needs to be sorted at present.
[0062] p(τ|fs) is the probability that the current actor network produces a ranking result of τ given fs.
[0063] Max j (p x,j ) is the maximum value of all feature processing step probabilities generated when the feature processing step is selected for the xth time. In step 7, when the model is trained, the loss function formula of the Critic critic network is:
[0064] ||PC|| 2 .
[0065] The present invention has the following beneficial effects:
[0066] (1) The method of the present invention can meet the processing resource constraints, geometric pre-feature constraints, process pre-feature constraints, and process chain feature step constraints of the part processing process.
[0067] (2) The method of the present invention has the ability to learn feature step sequencing experience, and can realize automatic learning of feature step sequencing by inputting various feature step sequencing problems.
[0068] (3) Compared with the existing heuristic sorting algorithm, the method of the present invention does not require repeated iterative calculations. It only needs to perform forward propagation calculations through the actor neural network module to obtain the feature step sorting optimization results. The calculation and solution time is less affected by the problem scale, and the optimization effect is better than the existing method. It can be used to solve sorting problems with a large number of feature steps.
[0069] (4) The method of the present invention has the ability of online optimization. If time permits, users can use the actor network to perform online deep reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 This is a framework schematic diagram of a feature processing step sequencing method based on dynamic masking and deep reinforcement learning in the present invention.
[0071] Figure 2 Schematic diagram of the self-attention residual network.
[0072] Figure 3 Select the network diagram for the feature step.
[0073] Figure 4 Flowchart for dynamic mask calculation.
[0074] Figure 5 Schematic diagram of the average cost of sorting results during actor network training.
[0075] Figure 6 Schematic diagram of the loss value during the critic network training process. DETAILED DESCRIPTION
[0076] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific preferred embodiments.
[0077] In the description of the present invention, it should be understood that the terms "left side", "right side", "upper part", "lower part" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. "First", "second" and the like do not indicate the importance of the components, and therefore cannot be understood as limiting the present invention. The specific dimensions used in this embodiment are only for illustrating the technical solution, and do not limit the scope of protection of the present invention.
[0078] like Figure 1 As shown, the feature processing step sorting method based on dynamic mask and deep reinforcement learning includes the following steps.
[0079] Step 1. Construct a deep reinforcement learning model for feature step sorting: the input vector of the deep reinforcement learning model for feature step sorting is the feature processing step in an unordered state, and the output vector is the feature processing step with a processing order; the deep reinforcement learning model for feature step sorting includes an Actor network and a Critic network.
[0080] Step 2, constructing the input mode of the input vector: each feature processing step of the input vector has a set input mode; wherein, the set input mode must be based on the geometric information and process chain information of the corresponding processing feature.
[0081] The above-mentioned setting input mode preferably includes feature position, feature step index, feature index, parent feature index, geometric precursor feature index, process precursor feature index, process chain index, feed direction index and processing tool index information, etc.
[0082] The input mode of each input vector includes corresponding processing constraints, wherein the processing constraints preferably include processing resource constraints, geometric pre-feature constraints, process pre-feature constraints, and process chain step constraints.
[0083] (1) Processing resource constraints
[0084] There may be multiple feature orientations for machining features, and the tool can be fed from multiple directions (such as X and Y) to complete the machining. However, in actual machining, any feature step i can only use one tool and select one direction for feeding;
[0085] (2) Geometric pre-feature constraints
[0086] If feature u is the geometric predecessor of feature v, feature step i must be placed before feature step j. Geometric predecessor means that feature u and feature v have a geometric dependency relationship, and feature u must be processed before feature v can be processed. For example, the intersecting features in a part are decomposed layer by layer to obtain several sub-features, and the upper-level sub-features are the geometric predecessors of the lower-level sub-features.
[0087] (3) Process Pre-feature Constraints
[0088] If feature u is the process prerequisite feature of feature v, feature step i must be placed before feature step j. Process prerequisite means that feature u and feature v have a process dependency relationship, and feature u must be processed before feature v. For example, when feature u is the processing datum of feature v, feature u must be processed before feature v. The process dependency relationship between features is specified by the process personnel according to the process requirements.
[0089] (4) Process chain step constraints
[0090] For any feature process chain of feature i, the feature steps in the process chain are arranged in the order of coarse first and fine later. The feature steps in the sorting result need to satisfy the process chain step constraints, that is, the feature steps in the feature process chain still maintain the relative order in the process chain in the sorting result.
[0091] In this embodiment, the input mode of the feature step input vector is shown in the following table.
[0092] Feature step input vector format
[0093]
[0094] In the above table, the processing constraints are preferably reflected in the input information of each feature step as follows:
[0095] (1) Geometric Predecessor Constraint: The seventh dimension gives the index of the geometric prerequisite feature of the current process step.
[0096] (2) Process prerequisite feature constraint: The eighth dimension gives the index of the process prerequisite feature of the current step.
[0097] (3) Process chain step constraints: directly given in the 9th dimension. For the steps in the process chain of the same feature, the step with a smaller index value must be executed before the step with a larger index value.
[0098] Step 3: Build the Actor network and the Critic network
[0099] The Actor network consists of an Actor mapping layer, a recurrent encoder A1, and a recurrent decoder A2.
[0100] The Actor Mapping layer is used to map the input vector constructed in step 2 to the input format of the recurrent encoder A1. The Actor Mapping layer uses a linear layer (the input of this layer is 10 neurons and the output is 128 neurons) to transform the feature vector of the input actor network into the input format of the recurrent encoder A1. i The feature vector F mapped from the low-dimensional space (10 dimensions) to the high-dimensional space i (128 dimensions), making the network input more expressive. Then the feature vector F of n feature steps i Input to circular encoder A1.
[0101] The recurrent encoder A1 is the first long short-term memory network LSTM. The recurrent encoder A1 has two output vectors, namely the cell state vector c (128 dimensions) and the historical state vector h (128 dimensions); wherein the historical state vector h is saved as a reference vector e for subsequent calculations.
[0102] The recurrent decoder A2 is a combination of a second long short-term memory network LSTM, a self-attention residual network AR, and a feature step selection network Pt.
[0103] The input vector of the second long short-term memory network LSTM includes a 128-dimensional 0 vector (corresponding to Figure 1 The output vectors of the cyclic encoder A1 are the updated cell state vector c (128 dimensions) and the updated historical state vector h (128 dimensions). The updated historical state vector h is saved as a decoding vector d for subsequent AR unit calculations. In the subsequent cyclic calculation process, the 128-dimensional 0 vector is updated to the high-dimensional vector F corresponding to the feature step selected by the Pt network. i .
[0104] like Figure 2 As shown, the input vectors of the self-attention residual network AR (also called AR module) are the reference vector e and the decoding vector d, and the output vector is the updated decoding vector D.
[0105] exist Figure 2 In the self-attention residual network AR, there are several layers of self-attention layers
[0106] Inside the attention layer, such as the correlation calculation process of the l-th self-attention layer, it can be specifically expressed as:
[0107]
[0108] In the formula, e j It is the reference vector saved in the cyclic encoding stage of cyclic encoder A1; here, the subscript j indicates that the vector is the reference vector corresponding to the jth processing feature.
[0109] is the decoding vector output by the recurrent decoder A2 in the l-th self-attention layer at the x-th feature selection step.
[0110] W1 l 、W2 l 、V l are the learning parameters of the l-th self-attention layer.
[0111] is the input vector e in the xth feature selection step in the lth self-attention layer j The transformed query vector. is the input vector D in the xth feature selection step in the lth self-attention layer x The corresponding conversion key value.
[0112] is the input vector d in the xth feature selection step in the lth self-attention layer x and e j Calculate the correlation score.
[0113] is the number of features selected in the xth step in the lth self-attention layer. x and e j The corresponding correlation calculation weights. The vector of the lth layer is input to the xth feature selection step, and is updated iteratively between self-attention layers. The calculation formula is shown below;
[0114]
[0115] d x It is the decoding vector output by the LSTM module of the cyclic decoder A2 when selecting the feature step for the xth time. LSTM is a module connected in front of the AR module.
[0116] D x It is the final output of the AR module when the feature step is selected for the xth time.
[0117] Input the vector of the l=4th layer for the xth feature selection step (this embodiment takes the attention layer with 4 layers as an example)
[0118] It is the decoding vector output by the recurrent decoder A2 in the l=4th self-attention layer at the xth feature selection step. Enter the vector of the l-1th layer for the xth feature selection step.
[0119] is the decoding vector output by the recurrent decoder A2 in the l-1th self-attention layer at the xth feature selection step.
[0120] The input vector of the feature step selection network Pt is the reference vector e, the updated decoding vector D, the dynamic mask m and the local cost f, and the output vector is the selection probability of each feature processing step in the input vector of the feature step sorting deep reinforcement learning model.
[0121] The network structure of the feature step selection network Pt is as follows Figure 3 As shown, in Figure 3 In, e j is the reference vector saved in the A1 loop encoding stage. The subscript j indicates that the vector is the reference vector corresponding to the jth feature. x is the output result of the feature processing step selected by the self-attention residual network AR at the xth time. x and all e j Perform correlation calculation and get the corresponding output u x,j .
[0122] Assume that the feature processing step selected for the x-1th time is feature processing step i, and the probability value of the feature processing step selected for the xth time is feature processing step j is p x,j , then p is calculated by Log-Softmax function x,j The calculation formula is:
[0123]
[0124] in:
[0125] u' x,j =u x,j -m j -f i,j
[0126] In the formula, u' x,j It is the score value when feature processing step j is selected during the xth selection.
[0127] m j It is the corresponding mask value of feature processing step j.
[0128] u x,j is the correlation calculation score between the reference vector e and the updated decoded vector D output by the self-attention residual network AR at the xth selection. x,j The calculation method of The relevant formula used when l=1 is:
[0129] u x,j =Vtanh(q j +kx )
[0130] in:
[0131] q j =W1e j
[0132] k x =W2D x
[0133] Where W1, W2, and V are the learning parameters of the feature step selection network Pt.
[0134] q j In the feature step selection network Pt, the vector e is input when the feature step is selected for the xth time. j The transformed query vector.
[0135] k x In the feature step selection network Pt, input vector D when selecting the feature step for the xth time x The corresponding conversion key value.
[0136] The present invention introduces a dynamic mask m in the calculation process. j , assign a maximum value to the mask of the feature step that cannot be the next step, and assign 0 to the rest of the steps. Since dynamic masks are used in the calculation process, they are used to mark the feature steps that cannot be selected at present. The feature steps that cannot be selected include the feature steps that have been selected and executed and the feature steps that do not meet the processing conditions.
[0137] The critic network consists of a critic mapping layer, a recurrent encoder C1, and a prediction network.
[0138] The Critic mapping layer is used to map the input vector constructed in step 2 to the input format of the cyclic encoder C1.
[0139] The output of the cyclic encoder C1 is used as the input of the prediction network, and the output of the prediction network is the predicted total cost P of the feature processing steps after sorting.
[0140] Step 4, determine the dynamic mask m: determine the dynamic mask m of each feature processing step in the input vector according to the processing constraints that need to be met during the execution of the feature step; wherein, the dynamic mask m of the feature step that does not meet the processing constraints is assigned a maximum value; the dynamic mask m of the feature step that meets the processing constraints is assigned 0 or a minimum value.
[0141] like Figure 4 As shown, the method for determining the dynamic mask m includes the following steps.
[0142] Step 4-1: Construct mask set M
[0143] exist Figure 4 In the , first input FMS and MSI; FMS is the set of all feature processing steps; MSI is the set of all executed feature processing step indexes.
[0144] Then, a mask set M is created, the number of mask elements in the mask set M is equal to the total number n of all feature steps, and all mask elements are initialized to 0 or set to a minimum value.
[0145] Step 4-2: Determine the processing execution status: traverse the mask element value m of the jth feature step in the mask set M j ; Where 1≤j≤n; Determine whether the jth feature step is unprocessed; if so, give the mask element value m j Assign a maximum value; otherwise, go to step 4-3.
[0146] Step 4-3: Determine the execution status of geometric pre-feature processing, which specifically includes the following steps:
[0147] Step 4-3A: According to the input mode of step 2, determine the processing feature index a to which the j-th feature step belongs.
[0148] Step 4-3B, determine whether the processing feature index a has a geometric pre-feature or a process pre-feature that has not been processed; if so, it indicates that the geometric pre-feature constraint and the process pre-feature constraint are not satisfied, so the mask element value m is given j Assign a maximum value; otherwise, go to step 4-4.
[0149] Step 4-4: Determine the execution status of the process chain process steps, including the following steps:
[0150] Step 4-4A: According to the input mode of step 2, determine the process chain index b of the feature process chain to which the j-th feature step belongs.
[0151] Step 4-4B: Determine whether there is an unprocessed feature step before b in the feature process chain; if so, it indicates that the process chain step constraint is not met, so the mask element value m is given. j Assign a maximum value; otherwise, repeat steps 4-2 to 4-4.
[0152] Step 5, determine the local cost f: local cost refers to the cost when switching between two different local feature steps; for each feature processing step in the input vector of the deep reinforcement learning model for feature step sorting, the corresponding local cost f is calculated; among them, the local cost f includes fixture replacement cost, tool replacement cost and idle travel cost.
[0153] Assume that the local cost of two different characteristic steps i and j is fi,j , then f i,j The calculation formula is:
[0154]
[0155] Wherein, λ1, λ2 and λ3 are fixture replacement consumption coefficient, tool replacement consumption coefficient and idle stroke consumption coefficient, respectively, and are all set values; in this embodiment, preferably λ1=100, λ2=20, λ3=0.05.
[0156] is the variable of feature step i with respect to feed direction k; when feed direction k is the feed direction of feature step i, The value of is 1; otherwise, The value of is 0; where 1≤k≤p; p is the total number of possible feed directions.
[0157] is the variable of feature step j with respect to feed direction k; when feed direction k is the feed direction of feature step j, The value of is 1; otherwise, The value of is 0.
[0158] is the variable of feature step i about tool h; when tool h is the tool used in feature step i, The value of is 1; otherwise, The value of is 0; where 1≤h≤q; q is the total number of available tools.
[0159] is the variable of feature step j with respect to tool h; when tool h is the tool used in feature step j, The value of is 1; otherwise, The value of is 0.
[0160] dis(i,j) is the idle travel distance of the tool moving from the processing feature corresponding to feature step i to the processing feature corresponding to feature step j. The unit of the idle travel distance is set to millimeter, but in the present invention, only the corresponding value at the millimeter level is extracted for calculation, and the unit does not participate in the formula calculation.
[0161] Step 6. Determine the deep reinforcement learning model for feature step sorting: Substitute the dynamic mask m determined in step 4 and the local cost f determined in step 5 into step 3 to obtain a complete Actor network; substitute the complete Actor network into step 1 to obtain a feature step sorting deep reinforcement learning model with a complete network; at this time, the actor network in the feature step sorting deep reinforcement learning model can calculate the selection probability of each feature processing step currently selected, and obtain the sorted feature processing steps through the cyclic encoder calculation n times, which forms the output vector of the feature step sorting deep reinforcement learning model.
[0162] Step 7, model training: Use the sample data set to train the feature step sorting deep reinforcement learning model determined in step 6 to obtain the trained feature step sorting deep reinforcement learning model; during the training process, calculate the actual total cost C of the sorted feature processing steps according to the output vector of the Actor network; obtain the predicted total cost P according to the Critic network; based on the predicted total cost P and the actual total cost C, on the one hand, according to the mean square error of PC, use the Adam optimizer to reduce the prediction error of the predicted total cost P in the Critic network; on the other hand, according to the CP error, use the Adam optimizer to optimize the sorting results of the feature steps.
[0163] The calculation formula for the actual total cost C of the feature processing steps after the above sorting is:
[0164] C=c1+c2+c3
[0165] in:
[0166]
[0167] Where c1 is the fixture replacement cost, c2 is the tool replacement cost, and c3 is the tool path idle travel cost.
[0168] e ij is the order variable of the work step in the sorting result; when j=i+1, e ij =1; otherwise, e ij =0.
[0169] During model training, the loss function formula of the above Actor network is preferably:
[0170] min(CP)logp(τ|fs)
[0171] in:
[0172]
[0173] Where τ is the feature step ranking result output by the current actor network.
[0174] fs is the disordered state feature processing step that needs to be sorted at present.
[0175] p(τ|fs) is the probability that the current actor network produces a ranking result of τ given fs.
[0176] Max j (p x,j ) is the maximum value of all the feature processing step probabilities generated when the feature processing step is selected for the xth time. During model training, the loss function formula of the above Critic critic network is preferably:
[0177] ||PC|| 2 .
[0178] like Figure 5 and Figure 6 As shown in Figure 1, as the number of reinforcement learning iterations increases, both the actor network and the critic network converge to a relatively optimal cost level. The model parameters of the converged actor network are saved for subsequent decision-making.
[0179] Step 8: Feature processing step sorting: Input a set of n unordered feature processing steps to be processed into the feature processing step sorting deep reinforcement learning model trained in step 7, and use the Actor network to obtain the feature processing steps with a processing order after sorting.
[0180] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.
Claims
1. A feature processing step sorting method based on dynamic mask and deep reinforcement learning, characterized in that: The steps include: Step 1: Construct a deep reinforcement learning model for feature step sorting: the input vector of the deep reinforcement learning model for feature step sorting is the feature processing step in disordered state, and the output vector is the feature processing step with processing order; the deep reinforcement learning model for feature step sorting includes an Actor network and a Critic network; Step 2, constructing an input mode of the input vector: each feature processing step of the input vector has a set input mode; wherein the set input mode must be based on the geometric information and process chain information of the corresponding processing feature; Step 3: Build the Actor network and the Critic network: The Actor network includes the Actor mapping layer, the loop encoder A1, and the loop decoder A2; where: The Actor mapping layer is used to map the input vector constructed in step 2 to the input format of the loop encoder A1; The recurrent encoder A1 is a first long short-term memory network LSTM, and the recurrent encoder A1 has two output vectors, namely, a cell state vector c and a historical state vector h; wherein the historical state vector h is saved as a reference vector e; The recurrent decoder A2 is a combination of the second long short-term memory network LSTM, the self-attention residual network AR and the feature step selection network Pt; The input vector of the second long short-term memory network LSTM includes two output vectors of the recurrent encoder A1, the output vectors are the updated cell state vector c and the updated historical state vector h; wherein the updated historical state vector h is saved as a decoding vector d; The input vector of the self-attention residual network AR is the reference vector e and the decoding vector d, and the output vector is the updated decoding vector D; the input vector of the feature step selection network Pt is the reference vector e, the updated decoding vector D, the dynamic mask m and the local cost f, and the output vector is the selection probability of each feature processing step in the input vector of the feature step sorting deep reinforcement learning model; the Critic critic network includes the Critic mapping layer, the recurrent encoder C1 and the prediction network; The Critic mapping layer is used to map the input vector constructed in step 2 to the input format of the cyclic encoder C1; The output of the cyclic encoder C1 is used as the input of the prediction network, and the output of the prediction network is the predicted total cost P of the feature processing step after sorting; Step 4, determine the dynamic mask m: determine the dynamic mask m of each feature processing step in the input vector according to the processing constraints that need to be met during the execution of the feature step; wherein, the dynamic mask m of the feature step that does not meet the processing constraints is assigned a maximum value; the dynamic mask m of the feature step that meets the processing constraints is assigned 0 or a minimum value; Step 5, determine the local cost f: local cost refers to the cost of switching between two different local feature steps; for each feature processing step in the input vector of the feature step sorting deep reinforcement learning model, the corresponding local cost f is calculated; the local cost f includes fixture replacement cost, tool replacement cost and idle travel cost; Step 6, determine the deep reinforcement learning model for feature step sorting: substitute the dynamic mask m determined in step 4 and the local cost f determined in step 5 into step 3 to obtain a complete Actor network; substitute the complete Actor network into step 1 to obtain a feature step sorting deep reinforcement learning model with a complete network; at this time, the actor network in the feature step sorting deep reinforcement learning model can calculate the selection probability of each feature processing step currently selected, and obtain the sorted feature processing steps through the cyclic encoder calculation n times, that is, form the output vector of the feature step sorting deep reinforcement learning model; Step 7, model training: Use the sample data set to train the feature step sorting deep reinforcement learning model determined in step 6 to obtain the trained feature step sorting deep reinforcement learning model; during the training process, calculate the actual total cost C of the sorted feature processing steps according to the output vector of the Actor network; obtain the predicted total cost P according to the Critic network; based on the predicted total cost P and the actual total cost C, on the one hand, according to the mean square error of PC, use the Adam optimizer to reduce the prediction error of the predicted total cost P in the Critic network; on the other hand, according to the CP error, use the Adam optimizer to optimize the feature step sorting result of the Actor network; Step 8: Feature processing step sorting: Input a set of n unordered feature processing steps to be processed into the feature processing step sorting deep reinforcement learning model trained in step 7, and use the Actor network to obtain the feature processing steps with a processing order after sorting.
2. The feature processing step sorting method based on dynamic mask and deep reinforcement learning according to claim 1 is characterized in that: In step 2, the input mode is set to include feature position, feature step index, feature index, parent feature index, geometric pre-feature index, process pre-feature index, process chain index, feed direction index and machining tool index information.
3. The feature processing step sorting method based on dynamic mask and deep reinforcement learning according to claim 1 is characterized in that: In step 4, the processing constraints include processing resource constraints, geometric pre-feature constraints, process pre-feature constraints, and process chain step constraints; the method for determining the dynamic mask m includes the following steps: Step 4-1, constructing a mask set M: Create a mask set M, the number of mask elements in the mask set M is equal to the total number n of all feature steps, and initialize all mask elements to 0 or set a minimum value; Step 4-2: Determine the processing execution status: traverse the mask element value m of the jth feature step in the mask set M j ; Where 1≤j≤n; Determine whether the jth feature step is unprocessed; if so, give the mask element value m j Assign a maximum value; otherwise, go to step 4-3; Step 4-3: Determine the execution status of geometric pre-feature processing, which specifically includes the following steps: Step 4-3A, according to the input mode of step 2, determine the processing feature index a to which the j-th feature step belongs; Step 4-3B, determine whether the processing feature index a has a geometric pre-feature or a process pre-feature that has not been processed; if so, it indicates that the geometric pre-feature constraint and the process pre-feature constraint are not satisfied, so the mask element value m is given j Assign a maximum value; otherwise, go to step 4-4; Step 4-4: Determine the execution status of the process chain process steps, including the following steps: Step 4-4A, according to the input mode of step 2, determine the process chain index b of the feature process chain to which the jth feature step belongs; Step 4-4B, determine whether there is an unprocessed feature step before b in the feature process chain; if so, it indicates that the process chain step constraint is not met, and thus the mask element value m is given. j Assign a maximum value; otherwise, repeat steps 4-2 to 4-4.
4. The feature processing step sorting method based on dynamic mask and deep reinforcement learning according to claim 1 is characterized in that: In step 5, let the local cost of two different feature steps i and j be f i,j , then f i,j The calculation formula is: In the formula, λ1, λ2 and λ3 are the fixture replacement consumption coefficient, tool replacement consumption coefficient and idle stroke consumption coefficient, respectively, and are all set values; is the variable of feature step i with respect to feed direction k; when feed direction k is the feed direction of feature step i, The value of is 1; otherwise, The value of is 0; where 1≤k≤p; p is the total number of possible feed directions; is the variable of feature step j with respect to feed direction k; when feed direction k is the feed direction of feature step j, The value of is 1; otherwise, The value of is 0; is the variable of feature step i about tool h; when tool h is the tool used in feature step i, The value of is 1; otherwise, The value of is 0; where 1≤h≤q; q is the total number of available tools; is the variable of feature step j with respect to tool h; when tool h is the tool used in feature step j, The value of is 1; otherwise, The value of is 0; dis(i,j) is the idle travel distance of the tool moving from the processing feature corresponding to feature step i to the processing feature corresponding to feature step j.
5. The feature processing step sorting method based on dynamic mask and deep reinforcement learning according to claim 4 is characterized in that: In step 7, the calculation formula for the actual total cost C of the feature processing step after sorting is: C=c1+c2+c3 in: Where c1 is the fixture replacement cost, c2 is the tool replacement cost, and c3 is the tool path idle travel cost; e ij is the order variable of the work step in the sorting result; when j=i+1, e ij =1; otherwise, e ij =0.
6. The feature processing step sorting method based on dynamic mask and deep reinforcement learning according to claim 4 or 5, characterized in that: λ1=100, λ2=20, λ3=0.
05.
7. The feature processing step sequencing method based on dynamic mask and deep reinforcement learning according to claim 4 is characterized in that: In step 6, let the feature processing step selected for the x-1th time be feature processing step i, and the probability value of the feature processing step selected for the xth time being feature processing step j is p x,j , then p x,j The calculation formula is: in: u' x,j =u x,j -m j -f i,j In the formula, u' x,j is the score value of feature processing step j being selected when the feature processing step j is selected for the xth time; u x,j The correlation score between the reference vector e and the updated decoded vector D output by the self-attention residual network AR is calculated for the xth selection; m j It is the corresponding mask value of feature processing step j.
8. The feature processing step sorting method based on dynamic mask and deep reinforcement learning according to claim 7 is characterized in that: In step 7, when training the model, the loss function formula of the Actor network is: min(CP)logp(τ|fs) in: Where, τ is the feature step ranking result output by the current actor network; fs is the disordered state feature processing step that needs to be sorted at present; p(τ|fs) is the probability that the current actor network produces a ranking result of τ given fs; Max j (p x,j ) is the maximum value of all feature processing step probabilities generated when the feature processing step is selected for the xth time.
9. The feature processing step sequencing method based on dynamic mask and deep reinforcement learning according to claim 7 is characterized in that: In step 7, when the model is trained, the loss function formula of the Critic network is: ||P-C|| 2 。
