Multi-working-condition process parameter optimization method and system based on migration reinforcement learning

Abstract

The invention discloses a multi-working-condition process parameter optimization method and system based on migration reinforcement learning, and belongs to the technical field of intelligent manufacturing, and the method comprises the steps: constructing a generalized relation model between process parameters and process indexes under different working conditions for a subtractive manufacturing processing system; selecting different working conditions as a source working condition domain and a target working condition domain; the process parameter optimization problem of the source working condition and the target working condition is formalized into a Markov decision process, a migration reinforcement learning algorithm is adopted, strategy migration between the source working condition and the target working condition is constructed, and optimal process parameter combinations under different working conditions are adaptively learned; the strategy migration construction comprises the following steps: aligning processing state dynamic characteristics under different working conditions, and learning a sharing strategy in optimal process parameter strategies under different working conditions; and according to the learned optimal process parameter combination under the target working condition, carrying out actual processing operation under the target working condition. According to the method, efficient adaptive optimization of technological parameter strategies under different working conditions can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent manufacturing, and particularly relates to a multi-condition process parameter optimization method and system based on transfer reinforcement learning. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] In the process of intelligent manufacturing, especially for subtractive manufacturing processes such as milling and turning, the optimization of process parameters is of great significance for improving product quality, reducing energy consumption, and enhancing processing efficiency. In recent years, many studies have adopted intelligent optimization methods (such as meta-heuristic algorithms, neural network algorithms, etc.) to obtain static optimal process parameter combinations to improve processing performance. However, the existing methods only optimize static process parameters, and the same optimization parameters are used for processing throughout the process. In actual production, due to the significant dynamic coupling effect in the continuous cutting process, the static parameters actually cannot adapt to the dynamic changes of the processing state during the entire continuous cutting process, which may lead to defects on the processed surface and cannot guarantee the optimal energy consumption and processing time efficiency.

[0004] Moreover, with the development of advanced manufacturing modes such as flexible manufacturing and customized manufacturing, the working conditions of the processing process are becoming increasingly complex and variable, and the system needs to adjust the corresponding process parameter strategies according to the dynamic changes of the working conditions. However, the existing process parameter optimization methods usually rely on specific processing conditions to solve the optimal process parameters. Once the processing configuration is adjusted or the processing task is changed, the original optimal process parameters often cannot be directly applied and must be re-optimized, resulting in a high optimization cost.

[0005] That is, the optimization of the existing process parameters is mostly for the optimization under static specific working conditions. Once the processing state changes or the processing conditions are adjusted during the processing, re-optimization and solution are required, and it is difficult to achieve the processing goals of high quality, high efficiency, and low energy consumption.

[0006] For the optimization of process parameters under the above different working conditions, a method using transfer technology has also been proposed currently. By parameter transfer (i.e., fine-tuning the parameters of the neural network), the optimized process parameters of another working condition are obtained using the fine-tuned neural network model. However, due to the characteristic that the transfer effect is not ideal when there is a lack of correlation between tasks in parameter transfer, the existing method has a high requirement for the similarity of working conditions, and it is only applicable to the transfer between similar processing tasks and difficult to cope with different processing tasks with significantly different characteristics, making it challenging to achieve efficient multi-task process parameter optimization. Summary of the Invention

[0007] To address the deficiencies of the above-mentioned existing technologies, the present invention provides a multi-condition process parameter optimization method and system based on transfer reinforcement learning. By designing a novel Transfer Reinforcement Learning (TRL) algorithm and constructing a cross-condition knowledge transfer mechanism, the adaptive transfer optimization of process parameter strategies under different conditions / different tasks is realized, thereby improving the efficiency of multi-task process parameter optimization, effectively reducing the multi-condition optimization cost, and at the same time ensuring the stability and generalization ability of the optimization process, providing an efficient solution for intelligent and flexible process decision-making in manufacturing systems.

[0008] In the first aspect, the present invention provides a multi-condition process parameter optimization method based on transfer reinforcement learning.

[0009] A multi-condition process parameter optimization method based on transfer reinforcement learning includes: For a subtractive manufacturing processing system, construct a generalized relationship model between process parameters and process indicators under different conditions; Select different conditions from multiple conditions as the source condition domain and the target condition domain respectively; Formalize the process parameter optimization problems of the source condition and the target condition as Markov decision processes, and use a transfer reinforcement learning algorithm to construct a strategy transfer between the source condition and the target condition, and adaptively learn the optimal process parameter combinations under different conditions; among them, the construction of the strategy transfer includes: aligning the dynamic characteristics of the processing states under different conditions, and learning a shared strategy in the optimal process parameter strategies under different conditions; According to the learned optimal process parameter combination under the target condition, perform the actual processing operation of the target condition.

[0010] In a further technical solution, in the transfer reinforcement learning algorithm, aligning the dynamic characteristics of the processing states under different conditions, that is, state transition alignment, is: Regard the source condition domain and the target condition domain as specific domains, specify any condition as the shared domain, and design a mapping mechanism; among them, the mapping mechanism is: construct a mapping function and an inverse mapping function between each specific domain and the shared domain, and let each specific domain and the shared domain execute the mapped actions respectively, so that the specific domain and the shared domain generate the same next-moment processing state after completing the actions, that is, ensure that the states after completing the actions are consistent; among them, the action refers to the process parameters of each process. Both the mapping function and the inverse mapping function are modeled by a multi-layer perceptron; the input vector of the mapping function consists of state-action pairs in the specific domain, and the output vector is the action mapped to the shared domain; the input vector of the inverse mapping function consists of state-action pairs in the shared domain, and the output vector is the action mapped to the specific domain. Build a loss function to train the mapping function and the inverse mapping function, so that the mapping function can ensure the consistency of the state after the action is completed by mapping the actions between the specific domain and the shared domain; among them, the loss function is the sum of the mapping consistency loss term, the domain cycle loss term and the adversarial loss term.

[0011] In a further technical solution, in the transfer reinforcement learning algorithm, learn the shared policy from the optimal process parameter strategies under different working conditions, that is, policy distillation, which is: Extract the empirical knowledge of the optimal process parameter strategy in the source working condition domain, and distill the optimized empirical knowledge with generalization characteristics into the shared policy; Under the guidance of the shared policy, learn the optimal process parameter strategy of the target working condition domain.

[0012] In a further technical solution, extracting the empirical knowledge of the optimal process parameter strategy in the source working condition domain and distilling the optimized empirical knowledge with generalization characteristics into the shared policy includes: Build an Actor module based on a neural network to learn the shared policy. The input vector of this Actor module is the state of the shared domain, and the output vector is the action applied to the shared domain; Construct the shared policy loss of the Actor module to extract the optimal process parameter strategies of each source working condition in the source working condition domain as empirical knowledge, so that the shared policy approximates the optimal process parameter strategies of each source working condition and completes the distillation of the shared policy.

[0013] In a further technical solution, under the guidance of the shared policy, learning the optimal process parameter strategy of the target working condition domain includes: Based on a neural network, construct a Critic module and an Actor module respectively; among them, the input vector of the Critic module is the state-action pair, and the output value is the estimated value of the state-action value. This state-action value combines the guidance of the shared policy to realize that the policy learning in the target working condition domain approximates the shared policy; the input vector of the Actor module is the current processing state, and the output vector is the action, that is, the process parameter combination; Based on the shared policy, train the Critic module and the Actor module; among them, during the training process, optimize the parameters in the Critic network by minimizing the mean square Bellman error; optimize the parameters in the Actor network by maximizing the future cumulative reward of the target policy and the approximation degree with the shared policy.

[0014] In a further technical solution, adopt a transfer reinforcement learning algorithm to construct the policy transfer between the source working condition and the target working condition, and adaptively learn the optimal process parameter combinations under different working conditions, including: Determine a set of source operating condition domains and a set of target operating condition domains, introduce a shared domain, and respectively collect the state transition trajectories in the source operating condition domain and the target operating condition domain as well as the state transition trajectories in the shared domain; Based on the collected trajectories, respectively learn the mapping function and the inverse mapping function between the source operating condition domain and the shared domain, as well as the mapping function and the inverse mapping function between the target operating condition domain and the shared domain; Based on the optimal process parameter strategy in the source operating condition domain, construct a shared strategy loss to learn the shared strategy in the optimal process parameter strategy of the source operating condition; Based on the shared strategy, optimize the network parameters of the Critic module and the Actor module in each target operating condition domain to learn the optimal process parameter strategy, that is, the optimal process parameter combination, under each target operating condition domain.

[0015] In a second aspect, the present invention provides a multi-condition process parameter optimization system based on transfer reinforcement learning.

[0016] A multi-condition process parameter optimization system based on transfer reinforcement learning includes: A generalized relationship module construction module for constructing a generalized relationship model between process parameters and process indicators under different operating conditions for a subtractive manufacturing processing system; An operating condition domain determination module for selecting different operating conditions from multiple operating conditions as the source operating condition domain and the target operating condition domain respectively; A process parameter optimization module for formalizing the process parameter optimization problems of the source operating condition and the target operating condition as Markov decision processes, and adopting a transfer reinforcement learning algorithm to construct a strategy transfer between the source operating condition and the target operating condition, and adaptively learn the optimal process parameter combinations under different operating conditions; wherein, the construction of the strategy transfer includes: aligning the dynamic characteristics of the processing states under different operating conditions and learning the shared strategy in the optimal process parameter strategies of different operating conditions; A manufacturing processing execution module for performing the actual processing operation of the target operating condition according to the learned optimal process parameter combination under the target operating condition.

[0017] In a third aspect, the present invention further provides an electronic device, including: a memory for storing executable instructions; a processor for implementing the above multi-condition process parameter optimization method based on transfer reinforcement learning when executing the executable instructions stored in the memory.

[0018] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to implement the above multi-condition process parameter optimization method based on transfer reinforcement learning when executing the executable instructions.

[0019] Fifth aspect, the present invention further provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned multi-condition process parameter optimization method based on transfer reinforcement learning is implemented.

[0020] The above one or more technical solutions have the following beneficial effects: 1. The present invention provides a multi-condition process parameter optimization method and system based on transfer reinforcement learning. By designing a novel transfer reinforcement learning algorithm and constructing a cross-condition knowledge transfer mechanism, through policy sharing and transfer, the adaptive transfer optimization of process parameter strategies under different conditions / different tasks is realized, thereby improving the efficiency of multi-task process parameter optimization. Compared with traditional process parameter optimization algorithms, the present invention has better generalization ability when dealing with variable and complex processing environments, can meet various processing requirements, effectively reduces the repeated training cost under similar conditions, effectively reduces the multi-condition optimization cost, and at the same time ensures the stability and generalization ability of the optimization process. This method can improve the decision-making ability and adaptability of the intelligent manufacturing system to simultaneously optimize the multi-condition process parameter combinations, thus contributing to the flexible production and efficient operation of intelligent manufacturing.

[0021] 2. The present invention considers that in the Markov decision process, the differences between different conditions are mainly reflected in the state transition changes and reward function changes. Therefore, in response to these differences, the transfer reinforcement learning algorithm proposed by the present invention constructs a policy transfer between the source condition and the target condition to align the dynamic characteristics of the processing states under different conditions. On this basis, a shared policy with generalization characteristics is learned from the optimal process parameter strategies of different conditions to improve the adaptability of process parameter optimization across conditions, and an adaptive decision-making model for process parameter optimization is constructed, realizing the process parameter optimization under multiple conditions and effectively improving the generalization ability of process parameter optimization under multiple conditions.

[0022] The advantages of the additional aspects of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0024] Figure 1 It is the overall flowchart of the multi-condition process parameter optimization method based on transfer reinforcement learning described in the embodiment of the present invention; Figure 2Schematic diagram of the Markov decision process in the embodiment of the present invention; Figure 3 Schematic diagram of the overall structure of the novel transfer reinforcement learning algorithm in the embodiment of the present invention; Figure 4 Schematic diagram of the convergence curve of the proposed transfer reinforcement learning algorithm in the process of optimizing process parameters under multiple working conditions in the embodiment of the present invention. Detailed implementation manners

[0025] It should be noted that the following detailed descriptions are all exemplary and are only for describing the specific implementation manners, aiming to provide further explanations for the present invention, and are not intended to limit the exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. In addition, it should also be understood that when the terms "include" and / or "comprise" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Embodiment 1 This embodiment provides a method for optimizing process parameters under multiple working conditions based on transfer reinforcement learning. This method is as Figure 1 shown and includes the following steps: Step S1: For a subtractive manufacturing processing system, construct a generalized relationship model between process parameters and process indicators under different working conditions.

[0027] Specifically, for subtractive manufacturing processing systems such as milling and turning, by analyzing the energy efficiency characteristics of the processing system and combining data-driven methods, a generalized relationship model between process parameters (such as cutting speed, feed rate, and cutting depth during the cutting process) and process indicators (such as machining quality, energy consumption, machining time, etc.) under different working conditions is constructed.

[0028] In this embodiment, taking the numerical control multi-step face milling process / system as an example, the energy efficiency characteristics of this machining system are analyzed, and a generalized relationship model between process parameters and key performance indicators such as machining time and electrical energy consumption is constructed by combining data-driven methods. Among them, the face milling process follows a roughing-first and finishing-last relationship, including m multiple roughing operations and the last finishing operation , so a complete face milling process can be expressed as , and the modeling of machining time and total electrical energy consumption during the face milling process is as follows. The relevant parameters of this model are shown in Table 1 below.

[0029] Table 1: Summary of relevant parameters of the machining process

[0030] First, the total machining time during face milling (s) can be modeled as: ; where , are the startup time and standby time respectively, usually constants; The cutting time can be expressed as , in the formula represents the cutting time of the i th rough machining step, represents the cutting time of finish machining, which can be expressed as , where L is the workpiece length (mm), D is the tool diameter (mm), is the tool approach distance (mm), is the i th rough machining or finish machining spindle speed (r / min), is the i th rough machining or finish machining feed per tooth (mm / tooth), z is the number of tool teeth, represents the cutting width (mm); The non-cutting time can be expressed as , in the formula represents the non-cutting time of the i th rough machining step, represents the non-cutting time of finish machining, which can be expressed as , is the safety distance to avoid damaging the tool and workpiece during machining, usually selected as 2 - 5 mm, is the cutting width (mm), usually equal to the workpiece width; The tool change time can be expressed as , is the time (min) required to replace a worn tool, usually a constant; According to the Taylor formula for tool life, the tool life (min) for each cutting step can be expressed as , and the coefficients in the formula are related to machining conditions such as specific tool materials, workpiece materials to be machined, lubrication methods, and cutting parameters; represents the cutting depth of the i th rough machining or finish machining step, represents the cutting speed of the i th rough machining or finish machining step, represents the time of the i-th rough machining step, represents the time of finish machining .

[0031] Secondly, the electrical energy consumption during face milling (J) can be modeled as: ; Wherein, , , , and respectively represent start-up energy consumption, standby energy consumption, no-cut energy consumption, machining energy consumption and tool-changing energy consumption; the start-up energy consumption is generally a constant; the standby energy consumption and the tool-changing energy consumption can be expressed as , , is the standby power during CNC machining; the machining electrical energy consumption can be expressed as: ; in the formula is the standby power during CNC machining, represents the no-load power (W) of each cutting step, which is composed of the no-load power of driving the machine tool spindle to rotate and the no-load power of the feed system moving. Nonlinear fitting using experimental data gives: , is the spindle speed (r / min) of each cutting step, is the feed angular velocity (rad / s) of each cutting step, which can be expressed as , S is the pitch (mm), u is the number of helical lines; is the cutting power of each cutting step, which can be expressed as , is the cutting speed (m / min). According to the empirical formula, the cutting force (N) can be calculated as: , and the coefficients in the formula are related to the tool material, the material of the workpiece being machined, the lubrication method, and the cutting layer thickness; represents the power of the auxiliary system that maintains the operation of the coolant pump, oil mist separator, etc. during the cutting process, and is usually a constant; represents the additional load loss power of the machine tool system caused by the cutting load during each cutting step. Based on experimental data, it can be expressed by nonlinear fitting as ; the no-cut energy consumption can be expressed as: , in the formula is the standby power during CNC machining; Denote the no-load power (W) for each cutting step; Denote the power of the auxiliary system that maintains the operation of the coolant pump, oil mist separator, etc. during the cutting process, which is generally a constant. Therefore, the specific energy consumption (SEC) during face milling can be expressed as , where MRV is the volume of material removed ( ).

[0032] According to the above definitions, the multi-objective optimization mathematical model for the multi-step face milling process can be expressed as: ; Among them, the decision variable is composed of the process parameter combinations and the number of cutting steps for each process step, and can be expressed as .

[0033] In addition, several machining constraints are set in the model, as follows: 1) Process parameter range requirements: , , ; 2) Machining allowance requirements: , is the total cutting depth; 3) Machining power requirements: Ensure that the power required for the cutting operation does not exceed the limited power, , is the rated power of the spindle, is the overall efficiency of the machine tool spindle; 4) Tool life requirements: The tool life during the cutting process cannot be less than the user-defined tool life ; 5) Roughness requirements: , ensure that the machined surface formed by each cutting reaches the minimum surface roughness requirement, where and represent the rake angle and clearance angle of the tool respectively; 6) Finish machining requirements: Finish machining must be performed in the last step, .

[0034] Step S2: Select different working conditions as the source working condition domain and the target working condition domain respectively from multiple working conditions.

[0035] Specifically, among multiple working conditions, a set of working conditions that have been verified and adopt the optimal process parameter strategy (or optimal process parameter combination) is selected as the source working condition domain, and a set of working conditions that have not yet implemented the optimal process parameter optimization is used as the target working condition domain. Considering the possible differences in factors such as different machine tool configurations, workpiece materials, tool sizes, and machining allowances among different working conditions, based on the generalized relationship model of the machining system, the relationship between process parameters and process indicators under each specific working condition in the domain is specifically modeled to adjust and refine the relationship model.

[0036] In this embodiment, working conditions with different machining allowances are selected as cases for multi-working condition process parameter optimization to verify the adaptability and optimization performance of the proposed algorithm under multi-working conditions. Among them, the machining allowance in the generalized relationship model is adjusted, and three working conditions with different machining allowances (6mm, 10mm, and 14mm) are selected as the source working condition domain, and three working conditions with machining allowances of 8mm, 9mm, and 12mm are additionally selected as the target working condition domain.

[0037] Step S3: Formalize the process parameter optimization problems of the source working condition and the target working condition as a Markov decision process, and use the transfer reinforcement learning algorithm to construct the strategy transfer between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions; among them, the construction of the strategy transfer includes: aligning the dynamic characteristics of the machining state under different working conditions and learning the shared strategy in the optimal process parameter strategies of different working conditions.

[0038] Step S3.1: Formalize the process parameter optimization problems of the source working condition and the target working condition as a Markov Decision Process (MDP), denoted as . Among them, the source working condition domain is defined as , and the target working condition domain is defined as . Taking the cutting process as an example, each cutting task needs to be completed through multi-step cutting. The Markov decision processes of each working condition are as Figure 2 shown, which can be understood as: the process intelligent agent (Agent) selects the process parameter combination and executes the cutting operation according to the machining state of the current step according to a certain strategy to obtain the machining state after cutting and key performance indicators such as electric energy consumption and machining time (i.e., the reward ), and then proceeds to the next step of cutting until the cutting task ends. The process parameter optimization goal during the entire machining process is defined as: continuously optimizing the process parameter combination and the number of steps of each step to maximize the cumulative reward of the machining process, thereby improving the machining efficiency, reducing the electric energy consumption, and ensuring the machining quality. This optimization goal can be specifically expressed as: ; among them, the entropy regularization term , to improve the algorithm exploration ability and robustness during the optimization process, while uses the temperature coefficient to control the influence degree of entropy.

[0039] The Markov decision process is specifically described by the quadruple as follows: 1) State space S represents the machining state, which is a vector composed of process parameters, machining allowance, and process performance indicators: .

[0040] 2) Action space A represents the combination of process parameters selected for each cutting step. According to the current state st select an action , which can be expressed as: .

[0041] 3) Transition function maps the current state and the corresponding action to the next state .

[0042] 4) Reward function is designed based on the performance index and machining constraints.

[0043] In this embodiment, the action is designed as , including the cutting speed (m / min), the feed rate (mm / tooth), and the cutting depth (mm); the state is designed as , represents the process parameter combination at the previous moment represents the current machining allowance, represents the current machining type (rough machining or finish machining); the reward function is designed with two optimization objectives of specific energy consumption and machining time, and at the same time, machining constraints are introduced as penalty terms, which can be expressed as: ; in the formula, is the normalization function, and are the trade-off coefficients, is the penalty coefficient; is the penalty term, which can be expressed as: ; In the above formula, the in the first term and the fifth term is the indicator function, which is 1 when the whole machining is completed.

[0044] Step S3.2: Use the transfer reinforcement learning algorithm to construct the policy transfer between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions.

[0045] Specifically, to adapt to the changes in different working conditions, a new transfer reinforcement learning algorithm is designed to construct an adaptive decision-making model for process parameter optimization, thereby improving the generalization ability of process parameter optimization under multiple working conditions. Considering that in the Markov decision process, the differences between different working conditions are mainly reflected in the state transition changes and the reward function changes. To address these differences, the proposed transfer reinforcement learning algorithm constructs the policy transfer between the source working condition and the target working condition, aligns the dynamic characteristics of the processing states under different working conditions, and on this basis, the algorithm learns a shared policy with generalization characteristics from the optimal process parameter strategies of different working conditions to improve the adaptability of process parameter optimization across different working conditions.

[0046] Among them, the overall structure of the new transfer reinforcement learning algorithm is as Figure 3 shown. In the above transfer reinforcement learning algorithm, aligning the dynamic characteristics of the processing states under different working conditions, that is, state transition alignment, is as follows: To reduce the differences in state changes between the source working condition domain and the target working condition domain , the algorithm designates any working condition as the shared domain , and designs a mapping mechanism to construct a mapping function to align the state transitions between the source working condition domain, the target working condition domain (both regarded as specific domains) and the shared domain. In the mapping mechanism, it is necessary to construct the mapping function H and the inverse mapping function between the source working condition domain, the target working condition domain and the shared domain. By mapping the actions between the specific domain and the shared domain, the specific domain and the shared domain are made to execute the mapped actions respectively, so as to ensure that the same next moment processing state is generated after the specific domain and the shared domain complete the actions (that is, the process parameters of each process), that is, to ensure that the states after completing the actions are consistent. Further, the state transition trajectory in the specific domain is defined as , while the state transition trajectory in the shared domain is defined as .

[0047] The above mapping function H and the inverse mapping function are both modeled by a Multilayer Perceptron (MLP). Among them, the input vector of the mapping function H consists of state-action pairs in the specific domain, and the output vector is the action mapped to the shared domain; the inverse mapping function Conversely, the input vector consists of state-action pairs in the shared domain, while the output vector is the corresponding action mapped to a specific domain. To ensure the consistency of the next-state between the specific domain and the shared domain after action mapping, the proposed algorithm designs the following loss function to train the mapping function and the inverse mapping function, which can be specifically expressed as: ; where, and are balance factors.

[0048] The above loss function includes three loss terms, namely: the mapping consistency loss term , the domain cycle loss term and the adversarial loss term , which are respectively expressed as: (1) Mapping consistency loss term: The actions that are mutually mapped in the two domains under the same state can generate the same next-state, which can be expressed as: ; where, represents the state transition function in the shared domain.

[0049] (2) Domain cycle loss term: Under the same state condition, the actions in a specific domain remain unchanged after passing through the mapping function H and the inverse mapping function P , which can be expressed as: .

[0050] (3) Adversarial loss term: To ensure that the actions in a specific domain can be effectively embedded into the action distribution in the shared domain after passing through the mapping function H , the algorithm introduces an adversarial mechanism, which requires training a discriminator aiming to distinguish the true action distribution in the shared domain from the mapped actions. Among them, the discriminator is modeled by a multi-layer perceptron, the input vector is a certain action, and the output value is a probability value representing the confidence that the input action comes from the shared domain. One of the optimization objectives of the mapping function is to make the mapped actions as close as possible to the true action distribution in the shared domain, thereby maximizing the uncertainty of the discriminator.

[0051] Specifically, the discriminator and the mapping function are alternately trained based on the following adversarial loss term, and this adversarial loss is: ; where, is the discriminator of the action distribution in the shared domain; the first term represents the difference between the discriminator predicting the mapped action and the true action respectively; to ensure the convergence stability of training based on this objective, the gradient norm R 1, R 2,R 1 is the gradient norm under the real action distribution, which can be expressed as ; ; R 2 is the gradient norm under the mapped action distribution, which can be expressed as ; .

[0052] The adversarial training process of the inverse mapping function P and is similar to that of the mapping function H , and its adversarial loss term is: ; where is the discriminator of the action distribution in a specific domain; R 1 is the gradient norm under the real action distribution, which can be expressed as ; ; R 2 is the gradient norm under the mapped action distribution, which can be expressed as ; In addition, in the above adversarial loss can be selected as .

[0053] In the above transfer reinforcement learning algorithm, learning a shared policy in the optimal process parameter strategies under different working conditions, that is, policy distillation, is: Considering that there are different degrees of differences in the influence of process parameters on process performance between different working conditions, that is, differences in the reward function, resulting in different optimal process parameter strategies under different working conditions. To solve this problem, a policy distillation mechanism is introduced to achieve adaptive optimization of multi-task process parameters. This mechanism can effectively extract the empirical knowledge of the optimal process parameter strategy in the source working condition domain and distill the optimized empirical knowledge with generalization characteristics into the shared policy; then, under the guidance of the shared policy, the process parameter optimization knowledge can be efficiently transferred to other target working conditions to learn the optimal process parameter strategy in the target working condition domain. The above process is mainly constrained by the generalized policy distillation loss, so as to efficiently optimize the process parameter combination under multiple working conditions. The generalized policy distillation loss is: ; where is the balance factor, is the future cumulative reward, The term uses the KL divergence to minimize the distribution difference between the shared policy (including the source working condition policy and the target working condition policy), The First, learn the shared policy from the source operating condition domains. Specifically, given a set of known source operating condition domains, the optimal process parameter policy obtained through conventional deep reinforcement learning algorithms is ; while the shared policy is usually modeled using the Actor module in the deep reinforcement learning algorithm. The input vector of this Actor module is the state of the shared domain, and the output vector is the action applied to the shared domain. To obtain a shared policy with generalization characteristics, the algorithm fixes the of the above-mentioned generalized policy distillation loss, thus obtaining the shared policy loss function: , that is, ensuring that the shared policy approximates the optimal policies of each source operating condition as closely as possible . Based on this shared policy loss, training is carried out to distill the optimized empirical knowledge with generalization characteristics into the shared policy. Among them, represents the mapping function that transforms the actions of the source operating condition domain to the shared domain, ensuring the consistency of the state changes in the two domains.

[0054] Secondly, under the guidance of the shared policy, learn the optimal process parameter policy (i.e., the optimal policy or the optimal combination of process parameters) of the target operating condition domain. The optimal policy of the target operating condition domain is usually also learned using conventional deep reinforcement learning algorithms, and it is necessary to construct a Critic module and an Actor module.

[0055] Specifically, the Critic module is usually modeled by a neural network. The input vector is the state-action pair, and the output value is the estimated value of the state-action value (Q-value), which is mainly used to evaluate the future cumulative reward after adopting a specific combination of process parameters under different processing states. On this basis, the state-action value introduces the above-mentioned generalized policy distillation loss to guide the policy learning of the target operating condition domain to approximate the shared policy, which can be expressed as: ; Among them, represents the future cumulative reward, is the cross-entropy term, which is used to ensure that the policy learning of the target operating condition domain approximates the shared policy as closely as possible; represents the entropy regularization term, which is used for the exploration of the target policy; represents the mapping function that transforms the actions of the target operating condition domain to the shared domain, ensuring the consistency of the state changes in the two domains.

[0056] Further, to accurately estimate the values under different states and actions, the parameters in the Critic network are optimized by minimizing the mean squared Bellman error, which can be expressed as: ; Among them, Sample a batch of state-action pairs from the experience replay pool D (Replay Buffer); The target estimate value is defined as: ; In the above formula, is the target Critic network, and its parameters are obtained by the exponential moving average of the Critic network parameters , that is: , is the smoothing update coefficient.

[0057] Specifically, the Actor module is usually modeled by a neural network. The input vector is the current processing state, and the output vector is the action, that is, the process parameter combination. To efficiently output the optimal process parameter combination, the Actor network parameters are optimized by maximizing the future cumulative reward of the target policy and the approximation degree with the shared policy, which can be expressed as: ; where represents the mapping function that transforms the actions in the target working condition domain to the shared domain, ensuring the consistency of the state changes in the two domains.

[0058] As an implementation method, the specific process of multi-condition process parameter optimization is as follows: Step S3.2.1: Determine a set of source working condition domains and a set of target working condition domains , introduce the shared domain, and collect the state transition trajectories in the source working condition domain and the target working condition domain respectively and the state transition trajectories in the shared domain ; Step S3.2.2: Based on the collected trajectories, learn the mapping function between the action distributions of the source working condition domain and the shared domain , and the inverse mapping function , as well as the mapping function between the action distributions of the target working condition domain and the shared domain .

[0059] Step S3.2.3: Use the optimal process parameter strategy in the source working condition domain to construct a shared policy loss function, and learn the shared policy based on the shared policy loss function ; Step S3.2.4. Based on the optimization experience of the sharing strategy, based on and to optimize the Critic module and Actor module in each target working condition domain, so as to efficiently learn the optimal process parameter strategy under each target working condition domain.

[0060] Through the above-mentioned solution proposed in this embodiment, different from the traditional solution that uses a neural network as an optimizer and only optimizes static process parameters, this embodiment is based on reinforcement learning as an optimizer, which can realize the dynamic optimization of process parameters, that is, the parameters can be adjusted according to different passes; moreover, different from the traditional way of migrating and fine-tuning network parameters, the migration method proposed in this embodiment can effectively handle tasks with different processing dynamic characteristics and different processing requirements.

[0061] In this embodiment, during the state transition alignment process, the mapping function H and the inverse mapping function P as well as the corresponding discriminators and are all modeled by a multi-layer perceptron. Among them, the input vector of the mapping function is the state-action pair in a specific domain, and the output vector is the action mapped to the shared domain, which consists of two hidden layers. The neurons in each network layer are [8, 256, 256, 3]. The activation function after the output layer is the Tanh function, and the activation function after each hidden layer is the ReLU function; the network structure of the inverse mapping function P is the same as that of the mapping function H . The input vector is the state-action pair in the shared domain, and the output vector is the action mapped to the specific domain; the discriminators and take the specific action as the input vector, and the output value is the probability that the action belongs to the current working condition domain. It consists of two hidden layers. The neurons in each network layer are [3, 256, 64, 1]. The activation function after each hidden layer is the ReLU function. All networks are optimized based on the loss function and the Adam optimizer (Adaptive Moment Estimation) is used for parameter update.

[0062] During the policy distillation process, both the shared policy and the target working condition policy are constructed by a new type of deep reinforcement learning algorithm. The network structure and hyperparameter settings in this algorithm can be set according to specific situations. Preferably, in this embodiment, the network structure of the Critic module is mainly based on the KAN network layer (Kolmogorov-Arnold Network). The KAN network layer can replace the linear weight transformation and fixed activation function in the traditional neural network structure by constructing a learnable adaptive activation function. Each KAN network layer can be specifically expressed as: ; In the above formula, is the adaptive activation function between layer and layer , and there are in total. , and the basis function can be expressed as , while the spline function can be parameterized by the linear combination of B-spline functions . is a learnable parameter. In the high-dimensional continuous process parameter space, this network structure has excellent non-linear expression ability, and the B-spline function in its structure can provide better smoothness and approximation ability. Therefore, the KAN network layer can accurately estimate the complex non-linear influence relationship of each process parameter on the optimization target under different processing states. Especially for the locally unexplored area, it can effectively alleviate the estimation error of the Critic network and improve the evaluation accuracy and generalization ability of the Critic network.

[0063] To further enhance the evaluation ability of process parameters, a new type of energy update layer is introduced in the middle layer of the Critic network, which can be specifically expressed as: ; where , , , X is the input feature, Z is the output feature, and , and are all trainable weight matrices. Compared with the traditional self-attention mechanism, this energy update layer adopts a multi-iteration update method, and its update rule follows , and outputs after the final update ends.

[0064] The above update mechanism can enhance the feature extraction ability of the processing state information, capture the local effective features in the processing state and process parameters more efficiently, so as to further improve the value estimation ability of the Critic network for the state-action space.

[0065] Correspondingly, the Actor network adopts a network structure similar to that of the Critic network, with the KAN network layer as the backbone and an energy update layer added to the middle layer. Through a learnable adaptive activation function, the KAN network layer can effectively identify the complex non-linear relationship between different processing states and process parameter combinations, enhancing the generalization performance of the Actor network when outputting the optimal process parameters; while the energy update layer can improve the recognition ability of the Actor network for key process parameters by optimizing the local feature representation of the current processing state, thereby further enhancing the optimality of process parameter decision-making.

[0066] In this embodiment, in the first stage, the shared policy is modeled by the Actor module in the new deep reinforcement learning algorithm. Based on the loss function the Actor module is optimized for gradients, and the Adam optimizer is used for parameter update during the optimization process; in the second stage, the learning of the target working condition policy requires the use of the Actor module and the Critic module in the new deep reinforcement learning algorithm. Based on the loss function and these two networks are optimized for gradients, and the Adam optimizer is used for parameter update during the optimization process. The hyperparameters involved in the entire transfer reinforcement learning algorithm are shown in Table 2 below.

[0067] Table 2: Algorithm hyperparameter settings

[0068] As Figure 4 shown, it records the convergence graph of the new transfer reinforcement learning algorithm for optimizing process parameters in each target working condition domain. It can be seen that under the guidance of the shared policy, the cumulative reward obtained by the target working condition policy continuously increases and quickly converges. This indicates that the algorithm proposed in this embodiment can efficiently learn the optimal process parameter strategy for multiple working conditions. By comparing with the optimization results of the traditional transfer reinforcement learning algorithm, namely the Generalized Policy Improvement (GPI) algorithm, as shown in Table 3 below, it can be seen that the proposed algorithm can reduce the processing energy consumption by at least 49.6% and the processing time by 49.7%.

[0069] Table 3: Comparison of optimization results with traditional transfer reinforcement learning algorithms

[0070] Step S4: According to the learned optimal process parameter combination under the target working condition, perform the actual processing operation of the target working condition.

[0071] Specifically, apply the optimal process parameter combination obtained by the new transfer reinforcement learning algorithm to the corresponding target working condition for actual processing.

[0072] Example 2 This embodiment provides a multi - condition process parameter optimization system based on transfer reinforcement learning, including: A generalized relationship module construction module, which is used to construct a generalized relationship model between process parameters and process indicators under different conditions for a subtractive manufacturing processing system; A working condition domain determination module, which is used to select different working conditions from multiple working conditions as the source working condition domain and the target working condition domain respectively; A process parameter optimization module, which is used to formalize the process parameter optimization problems of the source working condition and the target working condition into a Markov decision process, and adopt a transfer reinforcement learning algorithm to construct a policy transfer between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different conditions; among them, the construction of the policy transfer includes: aligning the dynamic characteristics of the processing state under different conditions, and learning a shared policy in the optimal process parameter strategies under different conditions; A manufacturing processing execution module, which is used to perform actual processing operations on the target working condition according to the learned optimal process parameter combination under the target working condition.

[0073] Example 3 This embodiment provides an electronic device, including: a memory for storing executable instructions; a processor, when executing the executable instructions stored in the memory, implements the above - mentioned method provided by this embodiment.

[0074] Example 4 This embodiment also provides a computer - readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above - mentioned method provided by this embodiment.

[0075] Example 5 This embodiment provides a computer program product. The computer program product includes executable instructions, and the executable instructions are a kind of computer instructions; the executable instructions are stored in a computer - readable storage medium. When the processor of an electronic device reads the executable instructions from the computer - readable storage medium and the processor executes the executable instructions, the electronic device is caused to execute the above - mentioned method provided by this embodiment.

[0076] The steps involved in the above Examples 2 to 5 correspond to those in Method Example 1. For specific implementation manners, reference may be made to the relevant description part of Example 1. The term "computer - readable storage medium" should be understood to include a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to execute any method in the present invention.

[0077] Those skilled in the art should understand that the various modules or steps of the present invention described above can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0078] The above are only the preferred embodiments of the present invention. Although the specific implementation manners of the present invention are described in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A multi-condition process parameter optimization method based on transfer reinforcement learning, characterized in that Including: For a subtractive manufacturing processing system, a generalized relationship model between process parameters and process indicators under different working conditions is constructed; From multiple working conditions, different working conditions are selected as the source working condition domain and the target working condition domain respectively; The process parameter optimization problems of the source working condition and the target working condition are formalized as a Markov decision process, and a transfer reinforcement learning algorithm is used to construct a policy transfer between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions; among them, the construction of the policy transfer includes: aligning the dynamic characteristics of the processing state under different working conditions, and learning a shared policy in the optimal process parameter policies of different working conditions; According to the learned optimal process parameter combination under the target working condition, the actual processing operation of the target working condition is carried out.

2. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 1, wherein In the transfer reinforcement learning algorithm, aligning the dynamic characteristics of the processing state under different working conditions, that is, state transition alignment, is: Regarding the source working condition domain and the target working condition domain as specific domains, designating any working condition as the shared domain, and designing a mapping mechanism; among them, the mapping mechanism is: constructing a mapping function and an inverse mapping function between each specific domain and the shared domain, and making each specific domain and the shared domain execute the mapped actions respectively, so that the same next-moment processing state is generated after the actions are completed by the specific domain and the shared domain, that is, ensuring that the states after the actions are completed are consistent; where the action refers to the process parameters of each process; Both the mapping function and the inverse mapping function are modeled by a multi-layer perceptron; the input vector of the mapping function consists of state-action pairs in the specific domain, and the output vector is the action mapped to the shared domain; the input vector of the inverse mapping function consists of state-action pairs in the shared domain, and the output vector is the action mapped to the specific domain; A loss function is constructed to train the mapping function and the inverse mapping function, so that the mapping function ensures that the states after the actions are completed are consistent by mapping the actions between the specific domain and the shared domain; among them, the loss function is the sum of a mapping consistency loss term, a domain cycle loss term, and an adversarial loss term.

3. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 1, characterized in that, In the transfer reinforcement learning algorithm, learning a shared policy in the optimal process parameter policies of different working conditions, that is, policy distillation, is: Extracting the empirical knowledge of the optimal process parameter policy in the source working condition domain, and distilling the optimized empirical knowledge with generalization characteristics into the shared policy; Under the guidance of the shared policy, learning the optimal process parameter policy of the target working condition domain.

4. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 3, characterized in that Extracting the empirical knowledge of the optimal process parameter policy in the source working condition domain, and distilling the optimized empirical knowledge with generalization characteristics into the shared policy, including: Constructing an Actor module based on a neural network to learn the shared policy, the input vector of this Actor module is the state of the shared domain, and the output vector is the action applied to the shared domain; Constructing a shared policy loss of the Actor module to extract the optimal process parameter policies of each source working condition in the source working condition domain as empirical knowledge, and making the shared policy approximate the optimal process parameter policies of each source working condition to complete the distillation of the shared policy.

5. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 3, characterized in that Under the guidance of the shared policy, learning the optimal process parameter policy of the target working condition domain, including: Based on the neural network, a Critic module and an Actor module are respectively constructed. Among them, the input vector of the Critic module is the state-action pair, and the output value is the estimated value of the state-action value. This state-action value is combined with the guidance of the shared policy to realize the approximation of the policy learning in the target working condition domain to the shared policy. The input vector of the Actor module is the current processing state, and the output vector is the action, that is, the process parameter combination. Based on the shared policy, the Critic module and the Actor module are trained. Among them, during the training process, the parameters in the Critic network are optimized by minimizing the mean square Bellman error. The parameters in the Actor network are optimized by maximizing the future cumulative reward of the target policy and the approximation degree to the shared policy.

6. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 1, characterized in that The transfer reinforcement learning algorithm is adopted to construct the policy transfer between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions, including: Determine a set of source working condition domains and a set of target working condition domains, introduce a shared domain, and respectively collect the state transition trajectories in the source working condition domain and the target working condition domain as well as the state transition trajectories in the shared domain. Based on the collected trajectories, respectively learn the mapping function and the inverse mapping function between the source working condition domain and the shared domain, and the mapping function and the inverse mapping function between the target working condition domain and the shared domain. Based on the optimal process parameter strategy in the source working condition domain, construct a shared policy loss to learn the shared policy in the optimal process parameter strategy of the source working condition. Based on the shared policy, optimize the network parameters of the Critic module and the Actor module in each target working condition domain to learn the optimal process parameter strategy under each target working condition domain, that is, the optimal process parameter combination.

7. A multi-condition process parameter optimization system based on transfer reinforcement learning, characterized in that, Including: A generalized relationship module construction module, which is used to construct a generalized relationship model between process parameters and process indicators under different working conditions for the subtractive manufacturing processing system. A working condition domain determination module, which is used to select different working conditions from multiple working conditions as the source working condition domain and the target working condition domain respectively. A process parameter optimization module, which is used to formalize the process parameter optimization problems of the source working condition and the target working condition into a Markov decision process, and adopt the transfer reinforcement learning algorithm to construct the policy transfer between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions. Among them, the construction of the policy transfer includes: aligning the dynamic characteristics of the processing state under different working conditions and learning the shared policy in the optimal process parameter strategies of different working conditions. A manufacturing processing execution module, which is used to perform the actual processing operation of the target working condition according to the learned optimal process parameter combination under the target working condition.

8. An electronic device, characterized in that, Including: A memory, which is used to store executable instructions. A processor, when executing the executable instructions stored in the memory, realizes the multi-condition process parameter optimization method based on transfer reinforcement learning according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Stored with executable instructions, which are used to cause the processor to execute the executable instructions to realize the multi-condition process parameter optimization method based on transfer reinforcement learning according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes executable instructions stored in a computer-readable storage medium; When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the method for optimizing process parameters under multiple working conditions based on transfer reinforcement learning according to any one of claims 1-6 is implemented.

Images