Multi-condition process parameter optimization method and system based on transfer reinforcement learning

Abstract

The present invention discloses a multi-condition process parameter optimization method and system based on transfer reinforcement learning, belonging to the field of intelligent manufacturing technology, including: constructing a generalized relationship model between process parameters and process indicators under different working conditions for subtractive manufacturing processing systems; selecting different working conditions as source working condition domains and target working condition domains respectively; formalizing the process parameter optimization problem of the source working condition and the target working condition into a Markov decision process, and using a transfer reinforcement learning algorithm to construct a strategy migration between the source working condition and the target working condition, and adaptively learning the optimal process parameter combination under different working conditions; wherein the construction of the strategy migration includes: aligning the dynamic characteristics of the processing state under different working conditions, learning a shared strategy among the optimal process parameter strategies under different working conditions; and performing actual processing operations for the target working condition based on the learned optimal process parameter combination under the target working condition. The present invention can achieve efficient adaptive optimization of process parameter strategies under different working conditions.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent manufacturing technology, and in particular relates to a multi-condition process parameter optimization method and system based on transfer reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In intelligent manufacturing processes, especially for subtractive manufacturing processes such as milling and turning, the optimization of process parameters is of great significance for improving product quality, reducing energy consumption, and increasing processing efficiency. In recent years, many studies have used intelligent optimization methods (such as metaheuristic algorithms and neural network algorithms) to obtain the static optimal process parameter combination to improve processing performance. However, existing methods only optimize static process parameters, and the same optimization parameters are used throughout the entire process. In actual production, due to the significant dynamic coupling effect in the continuous cutting process, static parameters are actually unable to adapt to the dynamic changes in the processing state throughout the continuous cutting process. This may lead to defects in the machined surface and cannot guarantee the optimal energy consumption and processing time efficiency.

[0004] Furthermore, with the development of advanced manufacturing models such as flexible and customized manufacturing, machining conditions are becoming increasingly complex and variable, requiring the system to adjust corresponding process parameter strategies based on dynamic changes in the working conditions. However, existing process parameter optimization methods typically rely on specific machining conditions to determine the optimal process parameters. Once the machining configuration is adjusted or the machining task is changed, the original optimal process parameters are often no longer directly applicable and must be re-optimized, resulting in high optimization costs.

[0005] That is, the optimization of existing process parameters is mostly aimed at optimization under static specific working conditions. Once the processing state changes or the processing conditions are adjusted during the processing, they need to be optimized and solved again, which makes it difficult to achieve the processing goals of high quality, high efficiency and low energy consumption.

[0006] To optimize process parameters under these different working conditions, a migration technique has been proposed. By migrating parameters (i.e., fine-tuning the neural network parameters), the fine-tuned neural network model is used to obtain the optimized process parameters for another working condition. However, due to the lack of correlation between tasks, parameter migration results in poor results. Therefore, existing methods place high demands on the similarity of working conditions. This makes migration only suitable for migrating between similar processing tasks, and is difficult to handle for different processing tasks with significantly different characteristics, making efficient multi-task process parameter optimization challenging. Summary of the Invention

[0007] In order to address the deficiencies of the above-mentioned prior art, the present invention provides a multi-working condition process parameter optimization method and system based on transfer reinforcement learning. By designing a new transfer reinforcement learning (TRL) algorithm, a cross-working condition knowledge transfer mechanism is constructed to achieve adaptive migration optimization of process parameter strategies under different working conditions / different tasks, thereby improving the efficiency of multi-task process parameter optimization, effectively reducing the multi-working condition optimization cost, and ensuring the stability and generalization ability of the optimization process, providing an efficient solution for intelligent and flexible process decision-making in manufacturing systems.

[0008] In a first aspect, the present invention provides a multi-condition process parameter optimization method based on transfer reinforcement learning.

[0009] A multi-condition process parameter optimization method based on transfer reinforcement learning, comprising:

[0010] For subtractive manufacturing systems, a generalized relationship model between process parameters and process indicators under different working conditions is constructed;

[0011] Select different working conditions from multiple working conditions as source working condition domain and target working condition domain respectively;

[0012] The process parameter optimization problem for the source and target working conditions is formalized as a Markov decision process. A transfer reinforcement learning algorithm is used to construct a policy transfer between the source and target working conditions, adaptively learning the optimal process parameter combinations under different working conditions. This policy transfer involves aligning the dynamic characteristics of the machining states under different working conditions and learning a shared strategy among the optimal process parameter strategies for different working conditions.

[0013] Based on the learned optimal process parameter combination under the target working conditions, the actual processing operation of the target working conditions is carried out.

[0014] A further technical solution is that in the transfer reinforcement learning algorithm, the dynamic characteristics of the processing states under different working conditions are aligned, that is, the state transfer alignment is:

[0015] The source and target working condition domains are considered specific domains, any working condition is designated as a shared domain, and a mapping mechanism is designed. The mapping mechanism involves constructing mapping functions and inverse mapping functions between each specific domain and the shared domain, and having each specific domain and the shared domain perform the mapped actions, respectively, so that the specific domain and the shared domain produce the same processing state at the next moment after completing the action, thus ensuring consistency in the state after completing the action. The action refers to the process parameters of each process.

[0016] The mapping function and the inverse mapping function are both obtained by multi-layer perceptron modeling; the input vector of the mapping function consists of state-action pairs in the specific domain, and the output vector is the action after mapping to the shared domain; the input vector of the inverse mapping function consists of state-action pairs in the shared domain, and the output vector is the action after mapping to the specific domain;

[0017] A loss function is constructed to train the mapping function and the inverse mapping function so that the mapping function ensures the consistency of the state after completing the action by mapping the actions between the specific domain and the shared domain; wherein the loss function is the sum of the mapping consistency loss term, the domain cycle loss term, and the adversarial loss term.

[0018] A further technical solution is that in the transfer reinforcement learning algorithm, a shared strategy is learned among the optimal process parameter strategies under different working conditions, i.e., strategy distillation, as follows:

[0019] Extract the empirical knowledge of the optimal process parameter strategy in the source operating domain and distill the optimization empirical knowledge with generalization characteristics into a shared strategy;

[0020] Under the guidance of the shared strategy, the optimal process parameter strategy for the target operating domain is learned.

[0021] A further technical solution is to extract the empirical knowledge of the optimal process parameter strategy in the source operating domain and distill the optimization empirical knowledge with generalization characteristics into a shared strategy, including:

[0022] Build a neural network-based Actor module to learn the sharing policy. The input vector of the Actor module is the state of the shared domain, and the output vector is the action acting on the shared domain.

[0023] Construct the shared strategy loss of the Actor module to extract the optimal process parameter strategy of each source working condition in the source working condition domain as empirical knowledge, so that the shared strategy approaches the optimal process parameter strategy of each source working condition, and completes the distillation of the shared strategy.

[0024] Further technical solutions, guided by the shared strategy, learn the optimal process parameter strategy for the target operating domain, including:

[0025] Based on a neural network, a Critic module and an Actor module are constructed. The input vector of the Critic module is a state-action pair, and the output value is an estimated value of the state-action value. This state-action value is combined with the guidance of the shared strategy to achieve policy learning in the target working domain close to the shared strategy. The input vector of the Actor module is the current processing state, and the output vector is the action, that is, the process parameter combination.

[0026] Based on the shared strategy, the Critic module and the Actor module are trained. During the training process, the parameters in the Critic network are optimized by minimizing the mean squared Bellman error. The parameters in the Actor network are optimized by maximizing the future cumulative reward of the target strategy and its approximation to the shared strategy.

[0027] A further technical solution uses a transfer reinforcement learning algorithm to build a strategy migration between the source and target working conditions, and adaptively learn the optimal process parameter combination under different working conditions, including:

[0028] Determine a set of source operating domains and a set of target operating domains, introduce a shared domain, and collect state transition trajectories in the source operating domains and the target operating domains, as well as the state transition trajectories in the shared domain;

[0029] Based on the collected trajectories, the mapping function and inverse mapping function between the source working condition domain and the shared domain, as well as the mapping function and inverse mapping function between the target working condition domain and the shared domain are learned respectively;

[0030] Based on the optimal process parameter strategy in the source working condition domain, a shared strategy loss is constructed to learn the shared strategy in the optimal process parameter strategy of the source working condition;

[0031] Based on the sharing strategy, the network parameters of the Critic module and the Actor module in each target working domain are optimized to learn the optimal process parameter strategy, that is, the optimal process parameter combination, under each target working domain.

[0032] In the second aspect, the present invention provides a multi-condition process parameter optimization system based on transfer reinforcement learning.

[0033] A multi-condition process parameter optimization system based on transfer reinforcement learning, comprising:

[0034] The generalized relationship module construction module is used to construct a generalized relationship model between process parameters and process indicators under different working conditions for subtractive manufacturing processing systems;

[0035] A working condition domain determination module is used to select different working conditions from multiple working conditions as source working condition domains and target working condition domains respectively;

[0036] The process parameter optimization module is used to formalize the process parameter optimization problem of the source and target working conditions as a Markov decision process. It then uses a transfer reinforcement learning algorithm to establish a policy migration between the source and target working conditions, adaptively learning the optimal process parameter combination under different working conditions. The policy migration construction includes aligning the dynamic characteristics of the machining state under different working conditions and learning a shared strategy among the optimal process parameter strategies under different working conditions.

[0037] The manufacturing and processing execution module is used to perform actual processing operations under the target working conditions based on the optimal process parameter combination under the learned target working conditions.

[0038] In a third aspect, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned multi-condition process parameter optimization method based on transfer reinforcement learning when executing the executable instructions stored in the memory.

[0039] In a fourth aspect, the present invention also provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned multi-condition process parameter optimization method based on transfer reinforcement learning.

[0040] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned multi-condition process parameter optimization method based on transfer reinforcement learning is implemented.

[0041] One or more of the above technical solutions have the following beneficial effects:

[0042] 1. The present invention provides a multi-condition process parameter optimization method and system based on transfer reinforcement learning. By designing a new transfer reinforcement learning algorithm, a cross-condition knowledge transfer mechanism is constructed. Through strategy sharing and migration, adaptive migration optimization of process parameter strategies under different conditions / different tasks is achieved, thereby improving the efficiency of multi-task process parameter optimization. Compared with traditional process parameter optimization algorithms, the present invention can have better generalization capabilities when dealing with changeable and complex processing environments, meet various processing requirements, effectively reduce the cost of repeated training under similar working conditions, effectively reduce the cost of multi-condition optimization, and at the same time ensure the stability and generalization ability of the optimization process. This method can improve the decision-making ability and adaptability of intelligent manufacturing systems to simultaneously optimize the combination of process parameters under multiple working conditions, thereby contributing to the efficient operation of flexible production and intelligent manufacturing.

[0043] 2. The present invention takes into account that in the Markov decision process, the differences between different working conditions are mainly reflected in the changes in state transfer and reward functions. To this end, in response to these differences, the transfer reinforcement learning algorithm proposed in the present invention aligns the dynamic characteristics of the processing state under different working conditions by constructing a strategy migration between the source working condition and the target working condition. On this basis, a shared strategy with generalization characteristics is learned from the optimal process parameter strategies of different working conditions to improve the adaptability of process parameter optimization across working conditions. In this way, an adaptive decision model for process parameter optimization is constructed, which realizes the optimization of process parameters under multiple working conditions and effectively improves the generalization ability of process parameter optimization under multiple working conditions.

[0044] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0046] Figure 1 This is an overall flow chart of the multi-condition process parameter optimization method based on transfer reinforcement learning according to an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of a Markov decision process according to an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the overall structure of the novel transfer reinforcement learning algorithm in an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the convergence curve of the transfer reinforcement learning algorithm proposed in an embodiment of the present invention during the multi-condition process parameter optimization process. DETAILED DESCRIPTION

[0050] It should be noted that the following detailed descriptions are exemplary only and are intended to describe specific embodiments and provide further explanation of the present invention, and are not intended to limit the exemplary embodiments according to the present invention. Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] Example 1

[0052] This embodiment provides a multi-condition process parameter optimization method based on transfer reinforcement learning. Figure 1 As shown, the following steps are included:

[0053] Step S1: For a subtractive manufacturing system, a generalized relationship model between process parameters and process indicators under different working conditions is constructed.

[0054] Specifically, for subtractive manufacturing processing systems such as milling and turning, by analyzing the energy efficiency characteristics of the processing system and combining it with data-driven methods, a generalized relationship model is constructed between process parameters (such as cutting speed, feed rate and cutting depth in the cutting process) and process indicators (such as processing quality, energy consumption, processing time, etc.) under different working conditions.

[0055] In this embodiment, the energy efficiency characteristics of the CNC multi-step face milling process / system are analyzed, and a generalized relationship model between process parameters and key performance indicators such as processing time and power consumption is constructed by combining data-driven methods. Among them, the face milling process follows the relationship of roughing first and finishing later, including m First roughing and last finishing , so a complete face milling process It can be expressed as , and the processing time and total electrical energy consumption in the face milling process are modeled as follows. The relevant parameters of the model are shown in Table 1 below.

[0056] Table 1: Summary of relevant parameters of the machining process

[0057]

[0058] First, the total machining time during face milling (s) can be modeled as:

[0059] ;

[0060] in, 、 They are the startup time and standby time, which are generally constants;

[0061] Cutting time can be expressed as , where Indicates the i Cutting time for the first roughing step, It represents the cutting time of finishing, which can be expressed as ,in L is the workpiece length (mm), D is the tool diameter (mm), is the tool approach stroke (mm), For the i Step roughing or finishing Spindle speed (r / min), For the i Step roughing or finishing The feed per tooth (mm / tooth), z is the number of tool teeth, Indicates cutting width (mm);

[0062] Air cut time It can be expressed as , where Indicates the i The air cutting time of the first rough machining, Indicates the air cutting time of finishing, which can be expressed as , In order to avoid damage to the tool and workpiece during processing, a safe distance of 2-5 mm is usually selected. is the cutting width (mm), which is usually equal to the workpiece width;

[0063] Tool change time It can be expressed as , The time required to replace a blunt tool (min) is generally a constant; according to Taylor's formula for tool life, the tool life (min) of each cutting step can be expressed as , where the coefficients are related to the specific tool material, workpiece material, lubrication method, cutting parameters and other processing conditions; Indicates the i The cutting depth of the roughing or finishing step, Indicates the i Cutting speed for roughing or finishing, represents the time of rough machining in step i, Indicates finishing time.

[0064] Secondly, the energy consumption during face milling (J) can be modeled as:

[0065] ;

[0066] in, 、 、 、 and Respectively represent the starting energy consumption, standby energy consumption, air cutting energy consumption, processing energy consumption and tool changing energy consumption; starting energy consumption Generally a constant; standby energy consumption Energy consumption of tool change It can be expressed as 、 , Standby power during CNC machining; Processing power consumption It can be expressed as: Where It is the standby power during CNC machining. Indicates the no-load power (W) of each cutting step, which is determined by the no-load power of the machine tool spindle. and no-load power of the feed system movement Composition, using experimental data for nonlinear fitting, we can get: , is the spindle speed of each cutting step (r / min), is the feed angular velocity of each cutting step (rad / s), which can be expressed as , S is the thread pitch (mm), u is the number of spirals; is the cutting power of each cutting step, which can be expressed as , is the cutting speed (m / min), according to the empirical formula, the cutting force (N) can be calculated as: , where the coefficient is related to the tool material, workpiece material, lubrication method, and cutting layer thickness; Indicates the power of auxiliary systems that maintain the operation of coolant pumps, oil mist separators, etc. during the cutting process, usually a constant; It represents the additional load loss power of the machine tool system caused by the cutting load during each cutting process. Based on the experimental data, it can be expressed by nonlinear fitting as ; Air cutting energy consumption It can be expressed as: , where It is the standby power during CNC machining; Indicates the no-load power (W) of each cutting step; It represents the power of auxiliary systems such as coolant pumps and oil mist separators that maintain the operation of the cutting process, which is generally a constant. Therefore, the specific energy consumption (SEC) in the face milling process can be expressed as , MRV is the volume of material removed ( ).

[0067] According to the above definition, the multi-objective optimization mathematical model of the multi-step face milling process can be expressed as:

[0068] ;

[0069] Among them, the decision variables It is composed of the process parameter combination and the number of steps of each process step, which can be expressed as .

[0070] In addition, the model sets several processing constraints, which are:

[0071] 1) Process parameter range requirements: , , ;

[0072] 2) Machining allowance requirements: , is the total cutting depth;

[0073] 3) Processing power requirements: Ensure that the power required for the cutting operation does not exceed the limited power. , is the spindle rated power, is the overall efficiency of the machine tool spindle;

[0074] 4) Tool life requirements: The tool life during cutting cannot be less than the user-set tool life ;

[0075] 5) Roughness requirements: , ensuring that the machined surface formed by each cutting process meets the minimum surface roughness requirements, including and Represent the tool rake angle and clearance angle respectively;

[0076] 6) Finishing requirements: Finishing must be performed in the last step. .

[0077] Step S2: Select different working conditions from multiple working conditions as the source working condition domain and the target working condition domain respectively.

[0078] Specifically, from a variety of working conditions, a set of working conditions that have been verified and adopt the optimal process parameter strategy (or optimal process parameter combination) is selected as the source working condition domain, and a set of working conditions that have not yet implemented optimal process parameter optimization is selected as the target working condition domain. Considering that different working conditions may have different factors such as machine tool configuration, workpiece material, tool size, and machining allowance, based on the generalized relational model of the machining system, the relationship between process parameters and process indicators in each specific working condition in the domain is specifically modeled to adjust and refine the relational model.

[0079] In this example, working conditions with different machining allowances were selected as cases for multi-condition process parameter optimization to verify the adaptability and optimization performance of the proposed algorithm under these conditions. The machining allowances in the generalized relational model were adjusted, and three working conditions with different machining allowances (6mm, 10mm, and 14mm) were selected as source working condition domains, while three working conditions with machining allowances (8mm, 9mm, and 12mm) were selected as target working condition domains.

[0080] Step S3: Formalize the process parameter optimization problem of the source working condition and the target working condition into a Markov decision process, and adopt the transfer reinforcement learning algorithm to construct the strategy migration between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions; wherein, the construction of the strategy migration includes: aligning the dynamic characteristics of the processing state under different working conditions, and learning the shared strategy among the optimal process parameter strategies of different working conditions.

[0081] Step S3.1: The process parameter optimization problem of the source and target working conditions is formalized into a Markov decision process (MDP), denoted as Among them, the source operating domain is defined as , the target operating domain is defined as Taking cutting technology as an example, each cutting task needs to be completed through multiple cutting steps. The Markov decision process of each working condition is as follows: Figure 2 As shown, it can be understood as: the process agent (Agent) according to the processing status of the current step , according to a strategy Select a combination of process parameters and perform cutting operations to obtain the processing status after cutting and key performance indicators such as power consumption and processing time (i.e. rewards ), and then proceed to the next step of cutting until the cutting task is completed. The process parameter optimization goal during the entire machining process is defined as: continuously optimizing the process parameter combination and the number of steps for each step to maximize the cumulative reward of the machining process, thereby improving machining efficiency, reducing power consumption, and ensuring machining quality. This optimization goal can be specifically expressed as: ; Among them, the entropy regularization term is introduced , in order to improve the algorithm exploration ability and robustness during the optimization process, and The temperature coefficient is used to control the degree of entropy influence.

[0082] The Markov decision process consists of four tuples: Describe:

[0083] 1) State Space S Represents the processing status, a vector consisting of process parameters, processing allowances, and process performance indicators: .

[0084] 2) Action Space A Indicates the process parameter combination selected for each cutting step, according to the current state st Select an action , which can be expressed as: .

[0085] 3) Conversion function The current state and corresponding actions Mapping to the next state .

[0086] 4) Reward Function Designed according to performance indicators and processing constraints.

[0087] In this embodiment, the action Designed for , including cutting speed (m / min), feed rate (mm / tooth), cutting depth (mm); the state Designed for , Indicates the process parameter combination at the previous moment Indicates the current machining allowance. represents the current processing type (roughing or finishing); the reward function is designed to be composed of two optimization objectives: specific energy consumption and processing time, and the processing constraint is introduced as a penalty term, which can be expressed as: Where, is the normalization function, and is the trade-off coefficient, is the penalty coefficient; is the penalty term, which can be expressed as:

[0088] ;

[0089] In the above formula, the first and fifth terms is an indicator function, which is 1 when the entire process is completed.

[0090] Step S3.2: Use the transfer reinforcement learning algorithm to construct the strategy migration between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions.

[0091] Specifically, to adapt to the changes in different working conditions, a novel transfer reinforcement learning algorithm is designed to construct an adaptive decision-making model for process parameter optimization, thereby improving the generalization ability of process parameter optimization under multiple working conditions. Considering that the differences between different working conditions in the Markov decision process are mainly reflected in changes in state transitions and reward functions, the proposed transfer reinforcement learning algorithm addresses these differences by constructing a policy transfer between the source and target working conditions, aligning the dynamic characteristics of the machining state under different working conditions. On this basis, the algorithm learns a shared policy with generalization characteristics from the optimal process parameter policies of different working conditions, thereby improving the adaptability of process parameter optimization across working conditions.

[0092] Among them, the overall structure of the new transfer reinforcement learning algorithm is as follows Figure 3 As shown in the above transfer reinforcement learning algorithm, the dynamic characteristics of the processing states under different working conditions are aligned, that is, the state transfer alignment is:

[0093] To reduce the source operating range and target operating domain The algorithm specifies any working condition as a shared domain. , and design a mapping mechanism to construct a mapping function to align the state transfer between the source working condition domain, the target working condition domain (all regarded as specific domains) and the shared domain. In the mapping mechanism, it is necessary to construct a mapping function between the source working condition domain, the target working condition domain and the shared domain. H and the inverse mapping function By mapping the actions between the specific domain and the shared domain, each specific domain and the shared domain are allowed to execute the mapped actions respectively, so as to ensure that the specific domain and the shared domain produce the same processing state at the next moment after completing the action (i.e., the process parameters of each process), that is, to ensure that the state after completing the action is consistent. Further, the state transition trajectory in the specific domain is defined as , and the state transition trajectory in the shared domain is defined as .

[0094] The above mapping function H and the inverse mapping function They are all modeled by Multilayer Perceptron (MLP). Among them, the mapping function H The input vector of is composed of state-action pairs in a specific domain, and the output vector is the action after mapping to the shared domain; the inverse mapping function On the contrary, the input vector consists of state-action pairs in the shared domain, and the output vector is the corresponding action after mapping to the specific domain. To ensure that the next moment state between the specific domain and the shared domain is consistent after action mapping, the proposed algorithm designs the following loss function to train the mapping function and the inverse mapping function, which can be expressed as:

[0095] ;

[0096] in, and is the balancing factor.

[0097] The above loss function includes three loss terms, namely: mapping consistency loss term , domain cycle loss term and adversarial loss , respectively expressed as:

[0098] (1) Mapping consistency loss: When two domains perform mutually mapped actions in the same state, they can produce the same state at the next moment, which can be expressed as:

[0099] ;

[0100] in, Represents the state transition function in the shared domain.

[0101] (2) Domain cycle loss term: Under the same state conditions, the actions in a specific domain are lost after the mapping function H and the inverse mapping function P After that, it remains unchanged and can be expressed as:

[0102] .

[0103] (3) Adversarial loss term: To ensure that actions within a specific domain pass through the mapping function H After effectively embedding the action distribution within the shared domain, the algorithm introduces an adversarial mechanism, which requires training a discriminator to distinguish between the true action distribution of the shared domain and the mapped actions. The discriminator is modeled as a multi-layer perceptron. The input vector is a certain action, and the output value is a probability value indicating the confidence that the input action originated from the shared domain. One of the optimization goals of the mapping function is to make the mapped actions as close as possible to the true action distribution of the shared domain, thereby maximizing the uncertainty of the discriminator.

[0104] Specifically, the discriminator and the mapping function are trained alternately based on the following adversarial loss term, which is:

[0105] ;

[0106] in, is the discriminator of the shared domain action distribution; the first term represents the difference between the discriminator's prediction of the mapped action and the real action; in order to ensure the convergence stability of training based on this goal, the gradient norm is introduced R 1. R 2, R 1 is the real action distribution The gradient norm can be expressed as ; R 2 is the action distribution after mapping The gradient norm can be expressed as .

[0107] The inverse mapping function P The adversarial training process and mapping function H Similarly, its adversarial loss term is:

[0108] ;

[0109] in, A discriminator for domain-specific action distribution; R 1 is the real action distribution The gradient norm can be expressed as ; R 2 is the action distribution after mapping The gradient norm can be expressed as ; In addition, the above-mentioned adversarial losses Can be selected as .

[0110] In the above transfer reinforcement learning algorithm, the shared strategy is learned among the optimal process parameter strategies under different working conditions, namely strategy distillation, which is:

[0111] Taking into account the fact that the impact of process parameters on process performance varies to varying degrees between different working conditions, that is, the difference in reward functions, the optimal process parameter strategies under different working conditions are different. To solve this problem, a policy distillation mechanism is introduced to achieve adaptive optimization of multi-task process parameters. This mechanism can effectively extract the empirical knowledge of the optimal process parameter strategy in the source working condition domain, and distill the optimized empirical knowledge with generalization characteristics into a shared strategy; then, under the guidance of the shared strategy, the process parameter optimization knowledge can be efficiently transferred to other target working conditions, thereby learning the optimal process parameter strategy for the target working condition domain. The above process is mainly constrained by the generalized policy distillation loss, so as to efficiently optimize the process parameter combination under multiple working conditions. The generalized policy distillation loss is:

[0112] ;

[0113] in, is the balance factor, Accumulate rewards for the future, The term uses KL divergence to minimize the shared policy With specific strategies (including the source working condition strategy and the target working condition strategy), The term uses the entropy regularization term to encourage the exploration of specific strategies. The entire strategy distillation process is optimized in two steps based on this loss function, including:

[0114] First, the sharing strategy is learned from the source working domain. Specifically, given a set of source working domains, the optimal process parameter strategy obtained by conventional deep reinforcement learning algorithm training is ; The shared strategy is usually modeled using the Actor module in the deep reinforcement learning algorithm. The input vector of the Actor module is the state of the shared domain, and the output vector is the action acting on the shared domain. In order to obtain a shared strategy with generalization characteristics, the algorithm distills the loss of the generalized strategy Fixed, thus obtaining the shared strategy loss function: , that is, ensuring shared policies The optimal strategy to approach each source condition as closely as possible , training is performed based on the shared strategy loss, thereby distilling the optimization experience knowledge with generalization characteristics into the shared strategy. It represents the mapping function that transforms the action of the source working condition domain into the shared domain, ensuring the consistency of the state changes of the two domains.

[0115] Secondly, under the guidance of the shared strategy, the optimal process parameter strategy (i.e., the optimal strategy or optimal process parameter combination) for the target operating domain is learned. The optimal strategy for the target operating domain is usually learned using a conventional deep reinforcement learning algorithm, which requires the construction of a Critic module and an Actor module.

[0116] Specifically, the Critic module Typically modeled using a neural network, the input vector is a state-action pair, and the output is an estimate of the state-action value (Q-value). This is primarily used to evaluate the future cumulative reward after using a specific combination of process parameters under different processing states. Based on this, the state-action value introduces the aforementioned generalized policy distillation loss to guide policy learning in the target operating domain toward a shared policy, which can be expressed as:

[0117] ;

[0118] in, Items represent future cumulative rewards, The term is the cross entropy term, which is used to ensure that the strategy learning in the target operating domain is as close as possible to the shared strategy; The term represents the entropy regularization term, which is used to explore the target strategy; It represents the mapping function that transforms the target working condition domain action to the shared domain, ensuring the consistency of the state changes in the two domains.

[0119] Furthermore, in order to accurately estimate the value of different states and actions, the critic network is trained by minimizing the mean square Bellman error. Parameters in Optimization can be expressed as:

[0120] ;

[0121] in, Indicates that the experience replay poolD (Replay Buffer) samples a batch of state-action pairs ; Target estimate Defined as:

[0122] ;

[0123] In the above formula, is the target Critic network, its parameters Through the Critic network parameters The exponential moving average is obtained, that is: , is the smoothing update coefficient.

[0124] Specifically, the Actor module It is usually modeled by a neural network, with the input vector being the current processing state and the output vector being the action, i.e., the process parameter combination. In order to efficiently output the optimal process parameter combination, the Actor network parameters are adjusted by maximizing the future cumulative reward of the target strategy and the degree of approximation to the shared strategy. Optimization can be expressed as:

[0125] ;

[0126] in, It represents the mapping function that transforms the target working condition domain action to the shared domain, ensuring the consistency of the state changes in the two domains.

[0127] As an implementation method, the specific process of multi-condition process parameter optimization is as follows:

[0128] Step S3.2.1: Determine a set of source operating conditions With a set of target operating conditions , introduce the shared domain, and collect the state transition trajectories in the source operating domain and the target operating domain respectively and the state transition trajectory in the shared domain ;

[0129] Step S3.2.2: Based on the collected trajectories, learn the source working condition domains Mapping function between action distributions and shared domains and the inverse mapping function , and target operating domain With shared domain The mapping function of the action distribution between and the inverse mapping function .

[0130] Step S3.2.3: Use the optimal process parameter strategy in the source working condition domain to construct a shared strategy loss function. learning shared strategies;

[0131] Step S3.2.4, optimization experience based on shared strategy, based on and To optimize the Critic module and Actor module in each target working domain, so as to efficiently learn the optimal process parameter strategy under each target working domain.

[0132] The above-mentioned solution proposed in this embodiment is different from the traditional solution based on neural network as an optimizer, which only optimizes static process parameters. This embodiment is based on reinforcement learning as an optimizer, which can realize dynamic optimization of process parameters, that is, adjust parameters according to different passes; moreover, different from the traditional network parameter migration and fine-tuning method, the migration method proposed in this embodiment can effectively handle tasks with different processing dynamic characteristics and different processing requirements.

[0133] In this embodiment, during the state transfer alignment process, the mapping function H , inverse mapping function P And the corresponding discriminator and They are all modeled by multi-layer perceptrons, where the input vector of the mapping function is the state-action pair in a specific domain, and the output vector is the action mapped to the shared domain. It consists of two hidden layers, and the number of neurons in each network layer is [8, 256, 256, 3]. The activation function after the output layer is the Tanh function, and the activation function after each hidden layer is the ReLU function; the inverse mapping function P Network structure and mapping function H Consistent, the input vector is the state-action pair in the shared domain, and the output vector is the action mapped to a specific domain; the discriminator and The input vector is a specific action, and the output value is the probability that the action belongs to the current working condition domain. It consists of two hidden layers, and the neurons in each network layer are [3, 256, 64, 1]. The activation function after each hidden layer is the ReLU function. All networks are based on the loss function Perform gradient optimization and use Adam optimizer (Adaptive Moment Estimation) for parameter update.

[0134] During the policy distillation process, both the shared policy and the target working condition policy are constructed using a new deep reinforcement learning algorithm. The network structure and hyperparameter settings in the algorithm can be set according to the specific situation. Preferably, in this embodiment, the network structure of the Critic module is mainly based on the Kolmogorov-Arnold Network (KAN) network layer. The KAN network layer can replace the linear weight transformation and fixed activation function in the traditional neural network structure by constructing a learnable adaptive activation function. Among them, each KAN network layer can be specifically expressed as:

[0135] ;

[0136] In the above formula, for Layer and Adaptive activation function between layers, total indivual, is the neuron of each layer, and each adaptive activation function is composed of a spline function and a basis function: , the basis function can be expressed as , and the spline function can be obtained by parameterizing the linear combination of B-spline functions , are learnable parameters. In the high-dimensional, continuous space of process parameters, this network structure possesses superior nonlinear representation capabilities, and its B-spline functions provide enhanced smoothness and approximation. Therefore, the KAN network layer can accurately estimate the complex nonlinear influence of various process parameters on the optimization objective under different processing conditions. This can effectively mitigate the estimation error of the critic network, especially in locally unexplored regions, thereby improving its evaluation accuracy and generalization capabilities.

[0137] In order to further enhance the ability to evaluate process parameters, a new energy update layer is introduced in the middle layer of the Critic network, which can be specifically expressed as follows:

[0138] ;

[0139] in, , , , X is the input feature, Z is the output feature, and 、 and are all trainable weight matrices. Compared with the traditional self-attention mechanism, the energy update layer adopts multiple iterative updates, and its update rules follow , and output after the final update is completed .

[0140] The above-mentioned update mechanism can enhance the feature extraction capability of processing state information and more efficiently capture the local effective features in processing state and process parameters, thereby further improving the value estimation capability of the critic network in the state-action space.

[0141] Accordingly, the Actor Network employs a similar network structure to the Critic Network, with a KAN network layer as the backbone and an energy update layer added in the middle. Through a learnable adaptive activation function, the KAN network layer effectively identifies the complex nonlinear relationships between different machining states and process parameter combinations, enhancing the generalization performance of the Actor Network in outputting optimal process parameters. The energy update layer, by optimizing the local feature representation of the current machining state, improves the Actor Network's ability to identify key process parameters, further enhancing the optimality of process parameter decisions.

[0142] In this embodiment, in the first stage, the sharing strategy is modeled by the Actor module in the new deep reinforcement learning algorithm based on the loss function The Actor module is gradient optimized, and the Adam optimizer is used to update the parameters during the optimization process. In the second stage, the target working condition strategy learning needs to use the Actor module and Critic module in the new deep reinforcement learning algorithm, based on the loss function. and Gradient optimization was performed on both networks, and the Adam optimizer was used to update parameters during the optimization process. The hyperparameters involved in the entire transfer reinforcement learning algorithm are shown in Table 2 below.

[0143] Table 2: Algorithm hyperparameter settings

[0144]

[0145] like Figure 4 As shown in Figure 3, it records the convergence diagram of the new transfer reinforcement learning algorithm for process parameter optimization in each target operating condition domain. It can be seen that under the guidance of the sharing strategy, the cumulative reward obtained by the target operating condition strategy continues to increase and converges quickly. This shows that the algorithm proposed in this embodiment can efficiently learn the optimal process parameter strategy for multiple operating conditions. By comparing the optimization results with the traditional transfer reinforcement learning algorithm, namely the generalized policy improvement algorithm (GPI), as shown in Table 3 below, it can be seen that the proposed algorithm can reduce processing energy consumption by at least 49.6% and processing time by 49.7%.

[0146] Table 3: Comparison of optimization results with traditional transfer reinforcement learning algorithms

[0147]

[0148] Step S4: Perform actual processing under the target working condition according to the learned optimal process parameter combination under the target working condition.

[0149] Specifically, the optimal process parameter combination obtained by the new transfer reinforcement learning algorithm is applied to the corresponding target working conditions for actual processing.

[0150] Example 2

[0151] This embodiment provides a multi-condition process parameter optimization system based on transfer reinforcement learning, including:

[0152] The generalized relationship module construction module is used to construct a generalized relationship model between process parameters and process indicators under different working conditions for subtractive manufacturing processing systems;

[0153] A working condition domain determination module is used to select different working conditions from multiple working conditions as source working condition domains and target working condition domains respectively;

[0154] The process parameter optimization module is used to formalize the process parameter optimization problem of the source and target working conditions as a Markov decision process. It then uses a transfer reinforcement learning algorithm to establish a policy migration between the source and target working conditions, adaptively learning the optimal process parameter combination under different working conditions. The policy migration construction includes aligning the dynamic characteristics of the machining state under different working conditions and learning a shared strategy among the optimal process parameter strategies under different working conditions.

[0155] The manufacturing and processing execution module is used to perform actual processing operations under the target working conditions based on the optimal process parameter combination under the learned target working conditions.

[0156] Example 3

[0157] This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for implementing the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0158] Example 4

[0159] This embodiment further provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0160] Example 5

[0161] This embodiment provides a computer program product including executable instructions, which are computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method provided in this embodiment.

[0162] The steps involved in the above embodiments 2 to 5 correspond to those in embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0163] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0164] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention is described in conjunction with the accompanying drawings, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A multi-condition process parameter optimization method based on transfer reinforcement learning, characterized in that: include: For subtractive manufacturing systems, a generalized relationship model between process parameters and process indicators under different working conditions is constructed; Select different working conditions from multiple working conditions as source working condition domain and target working condition domain respectively; The process parameter optimization problem for the source and target working conditions is formalized as a Markov decision process. A transfer reinforcement learning algorithm is used to construct a policy transfer between the source and target working conditions, adaptively learning the optimal process parameter combinations under different working conditions. This policy transfer involves aligning the dynamic characteristics of the machining states under different working conditions and learning a shared strategy among the optimal process parameter strategies for different working conditions. Perform actual processing operations under the target working conditions based on the learned optimal process parameter combination under the target working conditions; In the transfer reinforcement learning algorithm, the dynamic characteristics of the machining states under different working conditions are aligned, that is, the state transition alignment is: The source and target working condition domains are considered specific domains, any working condition is designated as a shared domain, and a mapping mechanism is designed. The mapping mechanism involves constructing mapping functions and inverse mapping functions between each specific domain and the shared domain, and having each specific domain and the shared domain perform the mapped actions, respectively, so that the specific domain and the shared domain produce the same processing state at the next moment after completing the action, thus ensuring consistency in the state after completing the action. The action refers to the process parameters of each process. The mapping function and the inverse mapping function are both obtained by multi-layer perceptron modeling; the input vector of the mapping function consists of state-action pairs in the specific domain, and the output vector is the action after mapping to the shared domain; the input vector of the inverse mapping function consists of state-action pairs in the shared domain, and the output vector is the action after mapping to the specific domain; Construct a loss function to train the mapping function and the inverse mapping function so that the mapping function ensures the consistency of the state after completing the action by mapping the actions between the specific domain and the shared domain. The loss function is the sum of the mapping consistency loss term, the domain cycle loss term, and the adversarial loss term. The transfer reinforcement learning algorithm is used to build a strategy migration between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions, including: Determine a set of source operating domains and a set of target operating domains, introduce a shared domain, and collect state transition trajectories in the source operating domains and the target operating domains, as well as the state transition trajectories in the shared domain; Based on the collected trajectories, the mapping function and inverse mapping function between the source working condition domain and the shared domain, as well as the mapping function and inverse mapping function between the target working condition domain and the shared domain are learned respectively; Based on the optimal process parameter strategy in the source working condition domain, a shared strategy loss is constructed to learn the shared strategy in the optimal process parameter strategy of the source working condition; Based on the sharing strategy, the network parameters of the Critic module and the Actor module in each target working domain are optimized to learn the optimal process parameter strategy, that is, the optimal process parameter combination, under each target working domain.

2. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 1, characterized in that: In the transfer reinforcement learning algorithm, the shared strategy is learned among the optimal process parameter strategies under different working conditions, namely strategy distillation, as follows: Extract the empirical knowledge of the optimal process parameter strategy in the source operating domain and distill the optimization empirical knowledge with generalization characteristics into a shared strategy; Under the guidance of the shared strategy, the optimal process parameter strategy for the target operating domain is learned.

3. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 2, characterized in that: Extract the empirical knowledge of the optimal process parameter strategy in the source operating domain and distill the optimization empirical knowledge with generalization characteristics into a shared strategy, including: Build a neural network-based Actor module to learn the sharing policy. The input vector of the Actor module is the state of the shared domain, and the output vector is the action acting on the shared domain. Construct the shared strategy loss of the Actor module to extract the optimal process parameter strategy of each source working condition in the source working condition domain as empirical knowledge, so that the shared strategy approaches the optimal process parameter strategy of each source working condition, and completes the distillation of the shared strategy.

4. The multi-condition process parameter optimization method based on transfer reinforcement learning according to claim 2 is characterized in that: Under the guidance of the shared strategy, the optimal process parameter strategy for the target operating domain is learned, including: Based on a neural network, a Critic module and an Actor module are constructed. The input vector of the Critic module is a state-action pair, and the output value is an estimated value of the state-action value. This state-action value is combined with the guidance of the shared strategy to achieve policy learning in the target working domain close to the shared strategy. The input vector of the Actor module is the current processing state, and the output vector is the action, that is, the process parameter combination. Based on the shared strategy, the Critic module and the Actor module are trained. During the training process, the parameters in the Critic network are optimized by minimizing the mean squared Bellman error. The parameters in the Actor network are optimized by maximizing the future cumulative reward of the target strategy and its approximation to the shared strategy.

5. A multi-condition process parameter optimization system based on transfer reinforcement learning, characterized in that: include: The generalized relationship module construction module is used to construct a generalized relationship model between process parameters and process indicators under different working conditions for subtractive manufacturing processing systems; A working condition domain determination module is used to select different working conditions from multiple working conditions as source working condition domains and target working condition domains respectively; The process parameter optimization module is used to formalize the process parameter optimization problem of the source and target working conditions as a Markov decision process. It then uses a transfer reinforcement learning algorithm to establish a policy migration between the source and target working conditions, adaptively learning the optimal process parameter combination under different working conditions. The policy migration construction includes aligning the dynamic characteristics of the machining state under different working conditions and learning a shared strategy among the optimal process parameter strategies under different working conditions. The manufacturing execution module is used to perform the actual processing of the target working conditions based on the optimal process parameter combination learned under the target working conditions; In the transfer reinforcement learning algorithm, the dynamic characteristics of the machining states under different working conditions are aligned, that is, the state transition alignment is: The source and target working condition domains are considered specific domains, any working condition is designated as a shared domain, and a mapping mechanism is designed. The mapping mechanism involves constructing mapping functions and inverse mapping functions between each specific domain and the shared domain, and having each specific domain and the shared domain perform the mapped actions, respectively, so that the specific domain and the shared domain produce the same processing state at the next moment after completing the action, thus ensuring consistency in the state after completing the action. The action refers to the process parameters of each process. The mapping function and the inverse mapping function are both obtained by multi-layer perceptron modeling; the input vector of the mapping function consists of state-action pairs in the specific domain, and the output vector is the action after mapping to the shared domain; the input vector of the inverse mapping function consists of state-action pairs in the shared domain, and the output vector is the action after mapping to the specific domain; Construct a loss function to train the mapping function and the inverse mapping function so that the mapping function ensures the consistency of the state after completing the action by mapping the actions between the specific domain and the shared domain. The loss function is the sum of the mapping consistency loss term, the domain cycle loss term, and the adversarial loss term. The transfer reinforcement learning algorithm is used to build a strategy migration between the source working condition and the target working condition, and adaptively learn the optimal process parameter combination under different working conditions, including: Determine a set of source operating domains and a set of target operating domains, introduce a shared domain, and collect state transition trajectories in the source operating domains and the target operating domains, as well as the state transition trajectories in the shared domain; Based on the collected trajectories, the mapping function and inverse mapping function between the source working condition domain and the shared domain, as well as the mapping function and inverse mapping function between the target working condition domain and the shared domain are learned respectively; Based on the optimal process parameter strategy in the source working condition domain, a shared strategy loss is constructed to learn the shared strategy in the optimal process parameter strategy of the source working condition; Based on the sharing strategy, the network parameters of the Critic module and the Actor module in each target working domain are optimized to learn the optimal process parameter strategy, that is, the optimal process parameter combination, under each target working domain.

6. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is used to implement the multi-condition process parameter optimization method based on transfer reinforcement learning as described in any one of claims 1 to 4 when executing the executable instructions stored in the memory.

7. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause the processor to execute the executable instructions to implement the multi-condition process parameter optimization method based on transfer reinforcement learning as described in any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the multi-condition process parameter optimization method based on transfer reinforcement learning according to any one of claims 1 to 4 is implemented.

Images