Cutting process parameter optimization method and system based on deep reinforcement learning

Abstract

This invention discloses a method and system for optimizing cutting process parameters based on deep reinforcement learning, relating to the field of intelligent manufacturing technology. The method comprises: constructing a relationship model between process parameters and process indicators for multi-step cutting tasks; constructing a cutting environment based on the relationship model, obtaining the machining state under the current environment, and formalizing the process parameter optimization problem as a Markov decision process; using an improved soft decision-evaluation algorithm as a process parameter optimizer within the Markov decision process to generate a process agent, which, after training, outputs the optimal process parameter combination for multiple steps; wherein the improved soft decision-evaluation algorithm includes: introducing a neural network into the Critic module and the Actor module; and executing the actual cutting task based on the optimal number of steps and process parameter combination output by the process agent. This invention can improve process parameter optimization performance and enhance part machining quality.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing mechanical processing technology, and specifically to a cutting process parameter optimization method and system based on deep reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In manufacturing processes, especially subtractive manufacturing processes, the selection of process parameters is crucial, directly affecting product quality and processing efficiency. In actual machining, due to the large cutting allowance of the workpiece and the rigidity of the machining process system, multiple passes are often required to obtain the final formed part surface that meets the machining requirements. Compared with single-step cutting, the process parameter optimization problem of this type of multi-step cutting is more complex: on the one hand, the process parameters are decision variables, and their quantity scale (i.e., dimension) will change dynamically with the change of the number of steps. For example, when the number of machining steps is a dynamic parameter to be optimized, the total number of process parameters will change accordingly, and the number of variables in the entire process is uncertain. On the other hand, throughout the entire machining process, the process parameters of the previous step will affect the machining status of the subsequent step, thereby generating a chain reaction on the process parameter strategy of the subsequent step, and the process parameters of the previous and next steps have a time-dependent relationship. However, traditional process parameter optimization schemes based on meta-heuristic optimization algorithms are mostly proposed for single-step cutting, which usually requires the assumption that the number of decision variables (i.e., process parameters) is fixed. This makes it impossible to directly optimize the number of steps and the process parameter combinations of each step in a multi-step machining process.

[0004] In addition, with the continuous emergence of new manufacturing processes, the number of process parameters involved in intelligent manufacturing has also increased significantly. At the same time, the nonlinear relationship between process parameters and processing performance has become more complex, which has also increased the difficulty of optimizing process parameters using traditional optimization methods. Summary of the Invention

[0005] In order to address one or more deficiencies in the above-mentioned prior art, the present invention provides a cutting process parameter optimization method and system based on deep reinforcement learning. On the basis of the traditional deep reinforcement learning framework, a deep reinforcement learning algorithm is proposed through an improved decision-evaluation module (i.e., Actor-Critic module), so as to optimize the process parameter combination of multiple steps, which can effectively improve the performance of process parameter optimization, thereby ensuring the quality of part processing, improving processing efficiency and effectively reducing energy consumption, and can meet the needs of complex process parameter optimization.

[0006] In order to achieve the above objectives, the present invention adopts one or more of the following technical solutions:

[0007] In a first aspect, the present invention provides a cutting process parameter optimization method based on deep reinforcement learning.

[0008] A cutting process parameter optimization method based on deep reinforcement learning, comprising:

[0009] For multi-step cutting tasks, a relationship model between process parameters and process indicators is constructed;

[0010] The cutting process environment is constructed based on the relational model to obtain the processing status under the current environment, and the process parameter optimization problem is formalized as a Markov decision process;

[0011] In the Markov decision process, an improved soft decision-evaluation algorithm is used as a process parameter optimizer to generate a process agent, which is trained to output the optimal process parameter combination for multiple steps. The improved soft decision-evaluation algorithm introduces a neural network into the Critic module and the Actor module. Through parameterized modeling and training of the neural network, the nonlinear influence of each process parameter on the optimized processing target under different processing states and the optimal action distribution corresponding to different processing states are learned.

[0012] The actual cutting task is performed according to the optimal number of steps and process parameter combination output by the process intelligence body.

[0013] In the second aspect, the present invention also provides a cutting process parameter optimization system based on deep reinforcement learning.

[0014] A cutting process parameter optimization system based on deep reinforcement learning, comprising:

[0015] The relationship model building module is used for multi-step cutting tasks to build the relationship model between process parameters and process indicators;

[0016] The state acquisition and processing module is used to build a cutting processing environment based on the relational model, obtain the processing status under the current environment, and formalize the process parameter optimization problem into a Markov decision process;

[0017] The process parameter optimization module uses an improved soft decision-evaluation algorithm as a process parameter optimizer in a Markov decision process to generate a process agent, which is then trained to output the optimal process parameter combination for multiple steps. The improved soft decision-evaluation algorithm introduces a neural network into the Critic module and the Actor module. Through parameterized modeling and training of the neural network, it learns the nonlinear impact of each process parameter on the optimized processing target under different processing states, as well as the optimal action distribution corresponding to different processing states.

[0018] The cutting processing execution module is used to execute the actual cutting processing task according to the optimal number of working steps and process parameter combination output by the process intelligence body.

[0019] In a third aspect, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned cutting process parameter optimization method based on deep reinforcement learning when executing the executable instructions stored in the memory.

[0020] In a fourth aspect, the present invention also provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned cutting process parameter optimization method based on deep reinforcement learning.

[0021] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned cutting process parameter optimization method based on deep reinforcement learning is implemented.

[0022] By adopting the above technical solution, the beneficial effects of the present invention are as follows:

[0023] 1. The present invention proposes a cutting process parameter optimization method and system based on deep reinforcement learning. Based on the traditional deep reinforcement learning framework, a new deep reinforcement learning algorithm (DRL) is proposed through an improved decision-evaluation module (i.e., Actor-Critic module) to optimize the process parameter combination of multiple steps. Compared with traditional meta-heuristic optimization algorithms, the deep reinforcement learning algorithm proposed in this invention has excellent dynamic decision-making capabilities and can efficiently solve the optimal process parameter strategy in complex processing scenarios. It can be applied to complex processing scenarios such as multi-step processing, effectively improving process parameter optimization performance, thereby improving part processing quality, optimizing process steps, and improving processing efficiency.

[0024] 2. This paper improves the structure of the Critic module and the Actor module, and adopts a combined structure of the KAN layer and the energy update layer, which effectively enhances the value estimation ability and strategy decision-making ability in the learning process, thereby further improving the performance of process parameter optimization.

[0025] 3. The present invention has good generalization capabilities and can be applied to various manufacturing processes such as milling, turning, and grinding. It can adapt to complex and changeable production conditions and processing requirements, ensure the continuous and efficient operation of the production process, and reduce the uncertainty and additional costs brought about by relying on empirical parameter tuning.

[0026] 4. The present invention is not only applicable to the complex process parameter optimization problems of a single manufacturing process, but can also be expanded to complex manufacturing scenarios such as additive and subtractive composite manufacturing and multi-process mixed manufacturing, meeting the complex process parameter optimization needs of various manufacturing modes, and providing an efficient and highly generalized process parameter optimization solution for the field of intelligent manufacturing.

[0027] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 Schematic diagram of the process flow of a cutting process parameter optimization method based on deep reinforcement learning in an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of a Markov decision process according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the structure of the deep reinforcement learning algorithm in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the structure of the Critic module and the Actor module in an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of the convergence curve in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that the following detailed descriptions are exemplary only and are intended to describe specific embodiments and provide further explanation of the present invention, and are not intended to limit the exemplary embodiments according to the present invention. Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0035] Example 1

[0036] This embodiment provides a cutting process parameter optimization method based on deep reinforcement learning, such as Figure 1 As shown, the specific steps include:

[0037] Step S1: For multi-step cutting tasks, a relationship model between process parameters and process indicators is constructed. Specifically, by analyzing the energy efficiency characteristics of the machining system and combining data-driven methods, a relationship model is constructed between process parameters such as cutting speed, feed rate, and depth of cut, and key process indicators such as machining quality, machining time, and power consumption.

[0038] In this embodiment, CNC multi-step face milling is used as an example to introduce that in order to reduce the processing time and power consumption of the cutting process, the cutting speed is achieved through the corresponding process parameter optimization method. , feed rate and cutting depth Optimization of other process parameters.

[0039] Specifically, a relationship model between process parameters and process indicators is constructed. Considering that the face milling process needs to obey the relationship of roughing first and then finishing, it includes First roughing and last finishing step , so a complete face milling process It can be expressed as: Based on the face milling process, a relationship model between process parameters and process indicators is constructed, where the relevant parameters of the model during the machining process are shown in Table 1 below.

[0040] Table 1: Summary of relevant parameters of the machining process

[0041]

[0042] Based on the above face milling process, a relationship model between process parameters, processing time and total power consumption is constructed. First, the total processing time in the face milling process is (s) can be modeled as:

[0043] ;

[0044] in, and They are the start-up time and standby time, which are usually constants; is the cutting time; It is the empty cutting time; Indicates tool change time; cutting time It can be expressed as , where Indicates the i Cutting time for the first roughing step, It represents the cutting time of finishing, which can be expressed as , L Indicates the length of the workpiece (mm), D Indicates the tool diameter (mm), Indicates the tool approach stroke (mm), For the i Step roughing or finishing Spindle speed (r / min), For the i Step roughing or finishing Feed per tooth (mm / tooth), z is the number of tool teeth, Indicates cutting width (mm); air cutting time It can be expressed as , where Indicates the i The air cutting time of step rough machining; Indicates the air cutting time of finishing, which can be expressed as , Indicates the safe distance to avoid damage to the tool and workpiece during machining, usually 2-5mm; Indicates cutting width (mm), usually equal to the workpiece width; tool change time It can be expressed as , Indicates the time required to replace a blunt tool (min), which is usually a constant; according to Taylor's formula for tool life, the tool life (min) of each cutting step can be expressed as The coefficients in this formula are related to the specific tool material, the material of the workpiece, the lubrication method, the cutting parameters and other processing conditions. Indicates the i Step roughing or finishing The cutting depth, Indicates the i Step roughing or finishing Cutting speed; represents the time of rough machining in step i, Indicates finishing time.

[0045] Secondly, the energy consumption during face milling ( J ) can be modeled as:

[0046] ;

[0047] in, 、 、 、 and Respectively represent the starting energy consumption, standby energy consumption, air cutting energy consumption, processing power consumption and tool changing energy consumption; starting energy consumption Usually constant; standby power consumption Energy consumption of tool change It can be expressed as: and , is the standby power during CNC machining, which is usually a constant; the machining power consumption can be expressed as , where It is the standby power during CNC machining; Indicates the no-load power (W) of each cutting step, which is determined by the no-load power of the machine tool spindle. and no-load power of the feed system movement Composition, nonlinear fitting using experimental data can be obtained , where is the spindle speed of each cutting step (r / min), is the feed angular velocity of each cutting step (rad / s), which can be expressed as ; S is the pitch (mm); u Indicates the number of spirals; It represents the cutting power of each cutting step and can be expressed as: , is the cutting speed (m / min); according to the empirical formula, the cutting force (N) can be calculated as: , the coefficients in the formula are related to the tool material, workpiece material, lubrication method, and cutting layer thickness; Indicates the power of auxiliary systems that maintain the operation of coolant pumps, oil mist separators, etc. during the cutting process, usually a constant; It represents the additional load loss power of the machine tool system caused by the cutting load during each cutting process. Based on experimental data, it can be expressed by nonlinear fitting as follows: ; The air-cut energy consumption can be expressed as: , where It is the standby power during CNC machining; Indicates the no-load power (W) of each cutting step; It represents the power of auxiliary systems such as coolant pump and oil mist separator to maintain the operation of cutting process. Therefore, the specific energy consumption (SEC) in the face milling process can be expressed as , MRV is the volume of material removed ( ).

[0048] Finally, according to the above definition, the multi-objective optimization mathematical model of the multi-step face milling process can be expressed as:

[0049] ;

[0050] Among them, the decision variables It is composed of the process parameter combination and the number of steps of each process step, which can be expressed as:

[0051] .

[0052] In addition, the above model sets several processing constraints, which are:

[0053] 1) Process parameter range requirements: , , ;

[0054] 2) Machining allowance requirements: , is the total cutting depth;

[0055] 3) Processing power requirements: Ensure that the power required for the cutting operation does not exceed the limited power. , is the spindle rated power, is the overall efficiency of the machine tool spindle;

[0056] 4) Tool life requirements: Tool life during cutting Cannot be less than the user preset tool life ;

[0057] 5) Roughness requirements: , ensuring that the machined surface formed by each cutting process meets the minimum surface roughness requirements, including and Represent the tool rake angle and clearance angle respectively;

[0058] 6) Finishing requirements: Finishing must be performed in the last pass, i.e. .

[0059] Step S2: construct a cutting process environment based on the relational model, obtain the processing status under the current environment, and formalize the process parameter optimization problem into a Markov decision process (MDP).

[0060] Furthermore, the Markov decision process is described and defined by the quaternion (S, A, P, R), where:

[0061] (1) The state space S represents the processing state, which is a vector composed of process parameters, machining allowances, and process performance indicators, and can be expressed as: ;

[0062] (2) The action space A represents the combination of process parameters selected for each cutting step, according to the current state Select an action , which can be expressed as: ;

[0063] (3) Conversion function The current state and corresponding actions Mapping to the next state ;

[0064] (4) Reward Function It represents the reward corresponding to the action calculated based on the key performance indicators and processing constraints, that is, the processing decision value.

[0065] like Figure 2 As shown in the figure, for the cutting process, each cutting task needs to be completed through multiple cutting steps. In each step, the process agent (Agent) is based on the processing status of the current step. , according to a strategy Select a combination of process parameters and perform cutting operations to obtain the processing status after cutting and key performance indicators such as power consumption and processing time (i.e. rewards ). After that, the next step of cutting is carried out until the cutting task is completed.

[0066] In this embodiment, the goal of process parameter optimization during the entire cutting process is to continuously optimize the process parameter combination and the number of steps in each process step to maximize the cumulative reward during the processing, thereby improving processing efficiency, reducing power consumption, and ensuring processing quality. Specifically, it can be expressed as:

[0067] ;

[0068] in, Accumulate rewards for the future, strategy entropy Used to encourage strategic exploration, it can be defined as , The temperature coefficient is used to control the degree of entropy influence.

[0069] Based on the above process parameter optimization goals, the Markov decision process in this embodiment is as follows:

[0070] 1) Move Designed for , which includes cutting speed (m / min), feed rate (mm / tooth), cutting depth (mm);

[0071] 2) Status Designed for ,in Indicates the process parameter combination at the previous moment, Indicates the current machining allowance. Indicates the current machining type (i.e., roughing or finishing);

[0072] 3) The reward function is designed by combining the two optimization objectives of energy consumption and processing time, while introducing the processing constraint as a penalty term, which can be expressed as:

[0073] ;

[0074] in, is the normalization function; and is the trade-off coefficient; is the penalty coefficient; is the penalty term, which can be expressed as:

[0075] ;

[0076] In the above formula, the first and fifth terms is an indicator function, which takes the value 1 when the entire process is completed.

[0077] Step S3: In the Markov decision process, an improved soft actor-critic (SAC) algorithm is used as a process parameter optimizer to generate a process intelligent agent, which outputs the optimal process parameter combination for multiple steps after training.

[0078] Specifically, the reinforcement learning algorithm mainly consists of two modules:

[0079] 1) Critic module, which is used to evaluate the impact of specific process parameter combinations under different processing states on the final processing objectives (such as cutting efficiency and energy consumption), that is, the processing decision value or state-action value (Q-value).

[0080] Among them, the state-action value is related to the cumulative reward, which can be expressed as:

[0081] ;

[0082] In the above formula, is the policy entropy, The temperature coefficient is used to control the degree of entropy influence.

[0083] 2) Actor module, which is used to learn the optimal action (i.e., process parameter combination) and execute the cutting action based on the evaluation results of the Critic module.

[0084] However, when solving process parameter optimization problems, traditional reinforcement learning has certain limitations, namely: the Critic module is prone to produce biased value estimates for unexplored process parameter spaces, thereby affecting the convergence of policy learning and the acquisition of the global optimal solution; due to the relatively simple network structure, the Actor module may fall into a local optimal solution in a complex and high-dimensional process parameter space. To address the above problems, in order to improve the generalization ability and optimization performance of the overall algorithm and achieve the optimization goal, this embodiment makes targeted structural improvements to the Actor module and the Critic module, and proposes a new deep reinforcement learning algorithm, namely the improved Soft Actor-Critic (SAC) algorithm. This algorithm is used as a process parameter optimizer to generate a process agent and obtain the optimal process parameter strategy or combination for multiple steps through training.

[0085] Among them, the improved soft decision-evaluation algorithm is as follows Figure 3 As shown in the figure, it is mainly composed of the Critic module and the Actor module, and a neural network is introduced into the Critic module and the Actor module. By parameterizing the neural network modeling and training, the nonlinear influence of each process parameter on the optimized processing target under different processing states and the optimal action distribution corresponding to different processing states are learned respectively.

[0086] Furthermore, the Critic module is parameterized by neural network modeling. The input vector consists of the processing state and process parameter combination (i.e. action), and the output value is the processing decision value. In order to reduce the overestimation bias in the value estimation, the module adopts a dual Critic network mechanism to improve the optimization stability, that is, to build two Critic networks. 、 , and take the minimum value of the two as the estimated value, such as Figure 4 As shown. Among them, the parameters of the Critic network It is learned by minimizing the mean squared Bellman error (MSBE) and is:

[0087] ;

[0088] In the above formula, Indicates sampling a batch of state-action pairs from the experience replay pool D (Replay Buffer) ; Target estimate Defined as , is the target Critic network, its parameters Through the Critic network parameters The exponential moving average is obtained, that is: , is the smoothing update coefficient.

[0089] In this embodiment, to improve the evaluation accuracy of the critic network, the network structure is mainly based on the Kolmogorov-Arnold Network (KAN) network layer. The KAN network layer can replace the linear weight transformation and fixed activation function in the traditional neural network structure by constructing a learnable adaptive activation function. Each KAN network layer can be specifically expressed as:

[0090] ;

[0091] In the above formula, for Layer and Adaptive activation function between layers, total indivual, is the neuron of each layer, and each adaptive activation function is composed of a spline function and a basis function: , the basis function can be expressed as , and the spline function can be obtained by parameterizing the linear combination of B-spline functions , are learnable parameters. In the high-dimensional, continuous space of process parameters, this network structure possesses superior nonlinear representation capabilities, and its B-spline functions provide enhanced smoothness and approximation. Therefore, the KAN network layer can accurately estimate the complex nonlinear influence of various process parameters on the optimization objective under different processing conditions. This can effectively mitigate the estimation error of the critic network, especially in locally unexplored regions, thereby improving its evaluation accuracy and generalization capabilities.

[0092] In order to further enhance the ability to evaluate process parameters, a new energy update layer is introduced in the middle layer of the Critic network, which can be specifically expressed as follows:

[0093] ;

[0094] in, , , , X is the input feature, Z is the output feature, and 、 and are all trainable weight matrices. Compared with the traditional self-attention mechanism, the energy update layer adopts multiple iterative updates, and its update rules follow , and output after the final update is completed .

[0095] The above-mentioned update mechanism can enhance the feature extraction capability of processing state information and more efficiently capture the local effective features in processing state and process parameters, thereby further improving the value estimation capability of the critic network in the state-action space.

[0096] The Actor module first passes through a neural network, the Actor network The input vector is the current processing state, and the output vector is the action mean. and the standard deviation of the logarithm of the action Composition, forming a Gaussian distribution , as the action distribution related to process parameters. Actor network parameters By minimizing the loss function, we can get:

[0097] ;

[0098] In the above formula, Indicates sampling state from the experience replay pool D , is the policy entropy, The temperature coefficient is used to control the influence of entropy. This loss function can be understood as optimizing the Actor network to obtain a process parameter strategy that maximizes processing value. At the same time, the entropy regularization term is added to increase the exploratory nature of the strategy and avoid premature convergence to a suboptimal solution.

[0099] Furthermore, the Actor Network employs a similar network structure to the Critic Network, with a KAN network layer as the backbone and an energy update layer added in the middle. Through a learnable adaptive activation function, the KAN network layer effectively identifies the complex nonlinear relationships between different machining states and process parameter combinations, enhancing the generalization performance of the Actor Network in outputting optimal process parameters. The energy update layer, by optimizing the local feature representation of the current machining state, improves the Actor Network's ability to identify key process parameters, further enhancing the optimality of process parameter decisions.

[0100] However, given the limited expressive power of the Actor Network, which only outputs process parameter combinations in the form of a unimodal Gaussian distribution, high-value process parameter combinations in complex manufacturing environments often exhibit multimodal distributions, meaning multiple local optimal solutions may exist, rather than a single solution. Therefore, to further enhance the Actor Module's policy optimization capabilities under complex working conditions, the module introduces a new policy update method to dynamically adjust the action distribution after the Actor Network outputs it, enabling more efficient exploration and convergence towards high-value areas.

[0101] During the update process, the action distribution output by the Actor network is used as the initial distribution, and a set of action particles are sampled from it. , these particles are updated through multiple iterations to obtain the optimized action distribution , whose update rule is designed based on Stein variational gradient descent, which can be expressed as:

[0102] ;

[0103] ;

[0104] In the above formula, is the update rate, and is the RBF kernel function , Represents the kernel variance, the first term shifts the action particles to the high probability area in the optimal action distribution, and the second term represents the repulsive force that encourages the action particles to disperse from the unimodal action distribution. The updated policy entropy can be specifically expressed as:

[0105] .

[0106] Further, such as Figure 3 As shown in FIG, the training process of the above process agent includes:

[0107] Step S3.1, initialize network parameters, the network includes the Critic network 、 and Actor Network and the target critic network .

[0108] Step S3.2, each time step t , execute the current action strategy in the environment , calculate the new state according to the transfer function and reward function and rewards .

[0109] Step S3.3, the experience ( , , , ) is stored in the experience replay pool for subsequent training.

[0110] Step S3.4: Update the network parameters every d steps, that is:

[0111] 1) Update the critic network parameters: ;

[0112] 2) Update Actor network parameters: ;

[0113] 3) Update the target critic network parameters: ;

[0114] Step S3.5: Repeat the above steps S3.2 to S3.4 until the optimal process parameter combination is learned through training.

[0115] Combine Figure 4 As shown in Figure 2, in this embodiment, the critic network consists of three network layers, each passing through a KAN layer, an energy update layer, and a KAN layer. The units in each network layer are [8, 64, 64, 1]. The actor network consists of four network layers, each passing through a KAN layer, an energy update layer, and a KAN layer. Finally, two KAN layers are passed in parallel, outputting the mean and logarithmic standard deviation of the action distribution. The units in each network layer are [5, 64, 64, 64, 6]. Network weights are optimized using the Adam learner with a learning rate of 3e-4, and the number of experience batches required for training is 100. Other hyperparameters are shown in Table 2.

[0116] Table 2: Algorithm hyperparameter settings

[0117]

[0118] During training, the process agent continuously interacts with the process parameter optimization environment, accumulating experience and continuously optimizing the process parameter strategy using a novel deep reinforcement learning algorithm. To explore diverse experiences, the agent performs random actions for the first 500 steps, learning how different process parameter combinations affect total energy consumption and machining time during the cutting process. After this random exploration, the agent outputs specific actions based on the Actor module and periodically updates the parameters in the Critic and Actor modules to continuously optimize the process parameter strategy. The entire training process takes 5,000 steps.

[0119] like Figure 5As shown in Table 3, after 500 steps of random exploration, the process agent's cumulative reward continues to increase as it interacts with the environment and accumulates experience, while the randomness of its strategy continues to decrease, ultimately stabilizing. This demonstrates that the proposed algorithm can continuously learn and converge to the optimal parameter combination. By comparing the optimization results with other deep reinforcement learning algorithms, such as Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft Actor-Critic (SAC), as shown in Table 3 below, we can see that the proposed algorithm can reduce processing energy consumption by at least 15.4% and processing time by 14.8%.

[0120] Table 3: Optimization results of different deep reinforcement learning algorithms

[0121]

[0122] Step S4: Execute the actual cutting task according to the optimal number of process steps and process parameter combination output by the process intelligent agent.

[0123] Example 2

[0124] This embodiment provides a cutting process parameter optimization system based on deep reinforcement learning, including:

[0125] The relationship model building module is used for multi-step cutting tasks to build the relationship model between process parameters and process indicators;

[0126] The state acquisition and processing module is used to build a cutting processing environment based on the relational model, obtain the processing status under the current environment, and formalize the process parameter optimization problem into a Markov decision process;

[0127] The process parameter optimization module uses an improved soft decision-evaluation algorithm as a process parameter optimizer in a Markov decision process to generate a process agent, which is then trained to output the optimal process parameter combination for multiple steps. The improved soft decision-evaluation algorithm introduces a neural network into the Critic module and the Actor module. Through parameterized modeling and training of the neural network, it learns the nonlinear impact of each process parameter on the optimized processing target under different processing states, as well as the optimal action distribution corresponding to different processing states.

[0128] The cutting processing execution module is used to execute the actual cutting processing task according to the optimal number of working steps and process parameter combination output by the process intelligence body.

[0129] Example 3

[0130] This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for implementing the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0131] Example 4

[0132] This embodiment further provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0133] Example 5

[0134] This embodiment provides a computer program product including executable instructions, which are computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method provided in this embodiment.

[0135] The steps involved in the above embodiments 2 to 5 correspond to those in embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0136] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0137] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention is described in conjunction with the accompanying drawings, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A cutting process parameter optimization method based on deep reinforcement learning, characterized in that: include: For multi-step cutting tasks, a relationship model between process parameters and process indicators is constructed; Based on the relational model, a cutting environment is constructed to obtain the machining status in the current environment. The process parameter optimization problem is formalized as a Markov decision process. The decision variables of the process parameter optimization problem include the process parameter combination of each process step and the number of process steps, which is the sum of m roughing steps and the last finishing step. In the Markov decision process, an improved soft decision-evaluation algorithm is used as a process parameter optimizer to generate a process agent, which is trained to output the optimal process parameter combination for multiple steps. The improved soft decision-evaluation algorithm introduces a neural network into the Critic module and the Actor module. Through parameterized modeling and training of the neural network, the nonlinear influence of each process parameter on the optimized processing target under different processing states and the optimal action distribution corresponding to different processing states are learned. The Critic module is used to evaluate the impact of a specific process parameter combination on the final processing target under different processing conditions, that is, the processing decision value; The Critic module is parameterized by neural network modeling. The input vector is the processing state and action of the current step. The action is the combination of process parameters. The output value is the processing decision value. The critic module uses a dual critic network mechanism, which constructs two critic networks and takes the minimum value of the two as the value estimate. The critic network consists of a KAN layer and an energy update layer. The KAN layer constructs a learnable nonlinear function to estimate the nonlinear impact of various process parameters on the optimized processing target under different processing states. The energy update layer is used to capture local effective features in the processing state and process parameters. The Actor module is used to learn the optimal action, i.e., the process parameter combination, and execute the cutting action based on the evaluation results of the Critic module; The Actor module includes a neural network consisting of a KAN layer and an energy update layer. The input vector of the Actor module is the machining state of the current step, and the output vector is the distribution of actions corresponding to the process parameter combination. The action distribution output by the Actor network is used as the initial distribution, and a set of action particles are sampled from it. Based on the update rule, the particles are iteratively updated multiple times to obtain the final action, that is, the process parameter combination required for cutting. The actual cutting task is performed according to the optimal number of steps and process parameter combination output by the process intelligence body.

2. The cutting process parameter optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The process parameters include cutting speed, feed rate and cutting depth; the process indicators include processing quality, power consumption and processing time.

3. The cutting process parameter optimization method based on deep reinforcement learning according to claim 1 is characterized in that: In the Markov decision process, each cutting task in the current cutting environment is completed through multiple cutting steps. The Markov decision process is described and defined by the four-tuple (S, A, P, R). The state space S represents the machining state, including process parameters, machining allowances, and process performance indicators. The action space A represents the actions selected for each cutting step, that is, the process parameter combination. The transformation function P maps the current machining state and corresponding actions to the next machining state. The reward function R represents the reward corresponding to the action, calculated based on the key performance indicators and machining constraints, that is, the machining decision value. Among them, for each step, the process agent is based on the processing status of the current step , select the action strategy To determine the process parameter combination, execute the action, and obtain the processing status after cutting and rewards ; Each machining step is executed in a continuous and iterative cycle until the multi-step cutting task is completed.

4. The cutting process parameter optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The training process of the improved soft decision-evaluation algorithm is: Initialize network parameters, including the Critic network 、 and Actor Network and the target critic network ; At each time step t, execute the current action policy in the environment , calculate the new state according to the transfer function and reward function and rewards ; The experience Store the experience in the replay pool for subsequent training; Every d Step 1, update the network parameters; The above steps are repeated until the optimal process parameter combination is learned through training.

5. A cutting process parameter optimization system based on deep reinforcement learning, characterized in that: include: The relationship model building module is used for multi-step cutting tasks to build the relationship model between process parameters and process indicators; The state acquisition and processing module is used to build a cutting environment based on the relational model, obtain the machining state in the current environment, and formalize the process parameter optimization problem as a Markov decision process. The decision variables of the process parameter optimization problem include the process parameter combination of each process step and the number of process steps, which is the sum of m roughing steps and the last finishing step. The process parameter optimization module uses an improved soft decision-evaluation algorithm as a process parameter optimizer in a Markov decision process to generate a process agent, which is then trained to output the optimal process parameter combination for multiple steps. The improved soft decision-evaluation algorithm introduces a neural network into the Critic module and the Actor module. Through parameterized modeling and training of the neural network, it learns the nonlinear impact of each process parameter on the optimized processing target under different processing states, as well as the optimal action distribution corresponding to different processing states. The Critic module is used to evaluate the impact of a specific process parameter combination on the final processing target under different processing conditions, that is, the processing decision value; The Critic module is parameterized by neural network modeling. The input vector is the processing state and action of the current step. The action is the combination of process parameters. The output value is the processing decision value. The critic module uses a dual critic network mechanism, which constructs two critic networks and takes the minimum value of the two as the value estimate. The critic network consists of a KAN layer and an energy update layer. The KAN layer constructs a learnable nonlinear function to estimate the nonlinear impact of various process parameters on the optimized processing target under different processing states. The energy update layer is used to capture local effective features in the processing state and process parameters. The Actor module is used to learn the optimal action, i.e., the process parameter combination, and execute the cutting action based on the evaluation results of the Critic module; The Actor module includes a neural network consisting of a KAN layer and an energy update layer. The input vector of the Actor module is the machining state of the current step, and the output vector is the distribution of actions corresponding to the process parameter combination. The action distribution output by the Actor network is used as the initial distribution, and a set of action particles are sampled from it. Based on the update rule, the particles are iteratively updated multiple times to obtain the final action, that is, the process parameter combination required for cutting. The cutting processing execution module is used to execute the actual cutting processing task according to the optimal number of working steps and process parameter combination output by the process intelligence body.

6. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the cutting process parameter optimization method based on deep reinforcement learning as described in any one of claims 1 to 4 when executing the executable instructions stored in the memory.

7. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause the processor to execute the executable instructions to implement the cutting process parameter optimization method based on deep reinforcement learning as described in any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the cutting process parameter optimization method based on deep reinforcement learning according to any one of claims 1 to 4 is implemented.

Images