Cutting process parameter optimization method and system based on deep reinforcement learning
By applying deep reinforcement learning to optimize multi-step cutting processing methods in intelligent manufacturing, the problem that traditional optimization algorithms are difficult to deal with complex process parameters combinations is solved, and more efficient processing quality and efficiency are achieved.
Patent Information
- Application Number
- CN202510694140.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-28
- Publication Date
- 2025-06-27
- Estimated Expiration
- 2045-05-28
AI Technical Summary
Traditional meta-heuristic optimization algorithms are difficult to effectively optimize the number of work steps and the combination of process parameters of each work step in multi-step cutting processing, especially in complex intelligent manufacturing scenarios.
Using the cutting processing process parameter optimization method based on deep reinforcement learning, the relationship model between process parameters and process indicators is constructed through the improved Actor-Critic module, and the process parameter optimization problem is formalized into the Markov decision-making process, and the process agent is generated to output the optimal process parameter combination of multiple steps.
It significantly improves the performance of process parameter optimization, improves the processing quality and processing efficiency of parts, reduces energy consumption, and is suitable for complex and variable manufacturing conditions.
Smart Images

Figure CN120215452A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent manufacturing machining, and particularly relates to a method and system for optimizing cutting process parameters based on deep reinforcement learning. Background Art
[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.
[0003] In manufacturing processes, especially in subtractive manufacturing processes, the selection of process parameters is crucial for the manufacturing process, which directly affects product quality and processing efficiency. In the actual machining process, due to the large cutting allowance of the workpiece and the influence of the rigidity of the machining process system, it is often necessary to perform multiple passes to obtain the final formed part surface that meets the machining requirements. Compared with single-step cutting, the complexity of the process parameter optimization problem for such multi-step cutting machining is higher: on the one hand, as decision variables, the quantity scale (i.e., dimension) of process parameters will change dynamically with the number of machining steps. For example, when the number of machining steps is a dynamic parameter to be optimized, the entire number of process parameters will change accordingly, and the number of variables in the entire process is uncertain; on the other hand, during the entire machining process, the process parameters of the previous machining step will affect the machining state of the subsequent machining step, thus having a chain reaction on the process parameter strategy of the subsequent machining step, and the process parameters of the front and rear machining steps have temporal dependence. However, traditional process parameter optimization schemes based on meta-heuristic optimization algorithms are mostly proposed for single-step cutting, and they usually need to assume that the number of decision variables (i.e., process parameters) is fixed, which makes it impossible to directly optimize the number of machining steps and the combination of process parameters for each machining step in the multi-step machining process.
[0004] In addition, with the continuous emergence of new manufacturing processes, nowadays, the number of process parameters involved in intelligent manufacturing has also increased significantly, and the non-linear relationship between process parameters and machining performance has become more complex, which also increases the difficulty of optimizing process parameters using traditional optimization methods. Summary of the Invention
[0005] To address one or more of the above deficiencies in the existing technology, the present invention provides a method and system for optimizing cutting process parameters based on deep reinforcement learning. Based on the traditional deep reinforcement learning framework, through an improved decision-evaluation module (i.e., Actor-Critic module), a deep reinforcement learning algorithm is proposed to optimize the combination of process parameters for multiple machining steps, which can effectively improve the performance of process parameter optimization, thereby ensuring the machining quality of parts, improving machining efficiency and effectively reducing energy consumption, and meeting the requirements of complex process parameter optimization.
[0006] To achieve the above objectives, the present invention adopts one or more of the following technical solutions: In a first aspect, the present invention provides a method for optimizing cutting process parameters based on deep reinforcement learning.
[0007] A method for optimizing cutting process parameters based on deep reinforcement learning, comprising: For a cutting processing task with multiple working steps, construct a relationship model between process parameters and process indicators; Based on the relationship model, construct a cutting processing environment, obtain the processing state in the current environment, and formalize the process parameter optimization problem as a Markov decision process; In the Markov decision process, an improved soft decision-evaluation algorithm is used as a process parameter optimizer to generate a process agent, and after training, an optimal process parameter combination for multiple working steps is output; wherein, the improved soft decision-evaluation algorithm is: introduce neural networks into the Critic module and the Actor module, and through parametric modeling and training of the neural networks, respectively learn the non-linear influence of each process parameter on the optimized processing target in different processing states and the optimal action distribution corresponding to different processing states; According to the optimal number of working steps and process parameter combination output by the process agent, execute the actual cutting processing task.
[0008] In a second aspect, the present invention further provides a system for optimizing cutting process parameters based on deep reinforcement learning.
[0009] A system for optimizing cutting process parameters based on deep reinforcement learning, comprising: A relationship model construction module, configured to construct a relationship model between process parameters and process indicators for a cutting processing task with multiple working steps; A state acquisition and processing module, configured to construct a cutting processing environment based on the relationship model, obtain the processing state in the current environment, and formalize the process parameter optimization problem as a Markov decision process; A process parameter optimization module, configured to use an improved soft decision-evaluation algorithm as a process parameter optimizer in the Markov decision process to generate a process agent, and after training, output an optimal process parameter combination for multiple working steps; wherein, the improved soft decision-evaluation algorithm is: introduce neural networks into the Critic module and the Actor module, and through parametric modeling and training of the neural networks, respectively learn the non-linear influence of each process parameter on the optimized processing target in different processing states and the optimal action distribution corresponding to different processing states; A cutting processing execution module, configured to execute an actual cutting processing task according to the optimal number of working steps and process parameter combination output by the process agent.
[0010] In a third aspect, the present invention further provides an electronic device, including: a memory for storing executable instructions; a processor for implementing the above-mentioned optimization method for cutting process parameters based on deep reinforcement learning when executing the executable instructions stored in the memory.
[0011] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to implement the above-mentioned optimization method for cutting process parameters based on deep reinforcement learning when executing the executable instructions.
[0012] In a fifth aspect, the present invention further provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads and executes the executable instructions from the computer-readable storage medium, the above-mentioned optimization method for cutting process parameters based on deep reinforcement learning is implemented.
[0013] By adopting the above technical solutions, the beneficial effects of the present invention are as follows: 1. The optimization method and system for cutting process parameters based on deep reinforcement learning proposed by the present invention, on the basis of the traditional deep reinforcement learning framework, through an improved decision-evaluation module (i.e., the Actor-Critic module), proposes a new deep reinforcement learning algorithm (Deep Reinforcement Learning, DRL) to optimize the process parameter combinations of multiple machining steps. Compared with the traditional meta-heuristic optimization algorithm, the deep reinforcement learning algorithm proposed by the present invention has good dynamic decision-making ability, can efficiently solve the optimal process parameter strategy in complex machining scenarios, can be applied to complex machining scenarios such as multiple machining steps, effectively improves the performance of process parameter optimization, thereby improving the machining quality of parts, optimizing the machining steps, and enhancing the machining efficiency.
[0014] 2. The present invention improves the structures of the Critic module and the Actor module, adopts a combined structure of the KAN layer and the energy update layer, effectively enhances the value estimation ability and policy decision-making ability in the learning process, and further improves the performance of process parameter optimization.
[0015] 3. The present invention has good generalization ability, can be applied to various manufacturing processes such as milling, turning, and grinding, can adapt to complex and changeable production conditions and machining requirements, ensure the continuous and efficient operation of the production process, and reduce the uncertainty and additional costs brought by relying on empirical parameter tuning.
[0016] 4. The present invention is not only applicable to the optimization of complex process parameters in a single manufacturing process, but also can be extended to complex manufacturing scenarios such as additive-subtractive hybrid manufacturing and multi-process hybrid manufacturing, meeting the complex process parameter optimization requirements of various manufacturing modes, and providing an efficient and highly generalized process parameter optimization solution for the field of intelligent manufacturing.
[0017] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention.
[0019] Figure 1 is a schematic flow chart of the cutting process parameter optimization method based on deep reinforcement learning in an embodiment of the present invention; Figure 2 is a schematic diagram of the Markov decision process in an embodiment of the present invention; Figure 3 is a schematic structural diagram of the deep reinforcement learning algorithm in an embodiment of the present invention; Figure 4 is a schematic structural diagram of the Critic module and the Actor module in an embodiment of the present invention; Figure 5 is a schematic diagram of the convergence curve in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0020] It should be noted that the following detailed description is exemplary only and is for the purpose of describing specific embodiments, aiming to provide further explanation of the present invention and not intended to limit the exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0021] Embodiment 1 This embodiment provides a cutting process parameter optimization method based on deep reinforcement learning, as Figure 1 shown, and specifically includes the following steps: Step S1: For a multi-step cutting machining task, construct a relationship model between process parameters and process indicators. Specifically, by analyzing the energy efficiency characteristics of the machining system and combining data-driven methods, construct a relationship model between process parameters such as cutting speed, feed rate, and cutting depth and key process indicators such as machining quality, machining time, and power consumption.
[0022] In this embodiment, the numerical control multi-step face milling machining is taken as an example for introduction. That is, in order to reduce the machining time and power consumption during the cutting process, through the corresponding process parameter optimization method, the cutting speed , feed rate and cutting depth and other process parameters are optimized.
[0023] Specifically, construct a relationship model between process parameters and process indicators. Among them, considering that the face milling process needs to follow the roughing-first and finishing-second relationship, it includes roughing passes and the last finishing pass , so a complete face milling process can be expressed as: . Based on this face milling process, construct a relationship model between process parameters and process indicators, and the relevant parameters of the model during the machining process are shown in Table 1 below.
[0024] Table 1: Summary of relevant parameters in the machining process
[0025] Based on the above face milling machining process, construct a relationship model between process parameters and machining time and total power consumption. First, the total machining time (s) in the face milling process can be modeled as: ; Among them, and are the start-up time and standby time respectively, usually constants; is the cutting time; is the non-cutting time; represents the tool change time; the cutting time can be expressed as , in the formula, represents the cutting time of the i th roughing pass, represents the cutting time of the finishing pass, which can be expressed as , L represents the workpiece length (mm), D represents the tool diameter (mm), represents the tool approach travel (mm), is the spindle speed (r / min) for the i th roughing or finishing step, and is the feed per tooth (mm / tooth) for the i th roughing or finishing step, is the number of tool teeth, z and represents the cutting width (mm); the non-cutting time can be expressed as , where represents the non-cutting time of the i th roughing step, represents the non-cutting time of the finishing step, which can be expressed as , represents the safety distance to avoid damaging the tool and workpiece during machining, usually selected as 2 - 5 mm; represents the cutting width (mm), usually equal to the workpiece width; the tool change time can be expressed as , represents the time required to replace a worn tool (min), usually a constant; according to the Taylor formula for tool life, the tool life (min) for each cutting step can be expressed as , and the coefficients in this formula are related to machining conditions such as specific tool materials, workpiece materials, lubrication methods, and cutting parameters. represents the i th roughing or finishing step cutting depth, represents the i th roughing or finishing step cutting speed; represents the time of the th roughing step, represents the time of the
[0026] Secondly, the electrical energy consumption during face milling ( J ) can be modeled as: ; where , , , and represent the start-up energy consumption, standby energy consumption, non-cutting energy consumption, machining electrical energy consumption, and tool change energy consumption respectively; the start-up energy consumption is usually a constant; the standby energy consumption and the tool change energy consumption can be expressed as: and , is the standby power during CNC machining, usually a constant; the machining electrical energy consumption can be expressed as , where is the standby power during CNC machining; represents the no-load power (W) of each cutting step, which consists of the no-load power for driving the spindle of the machine tool to rotate and the no-load power for the movement of the feed system. Nonlinear fitting using experimental data gives , where is the spindle speed (r / min) of each cutting step, is the feed angular velocity (rad / s) of each cutting step, which can be expressed as ; S is the pitch (mm); u represents the number of helical lines; represents the cutting power of each cutting step, which can be expressed as: , is the cutting speed (m / min); according to the empirical formula, the cutting force (N) can be calculated as: , and the coefficients in the formula are related to the tool material, the material of the workpiece being machined, the lubrication method, and the cutting layer thickness; represents the power of the auxiliary system for maintaining the operation of the coolant pump, oil mist separator, etc. during the cutting process, usually a constant; represents the additional load loss power of the machine tool system caused by the cutting load during each cutting step. Based on experimental data, it can be expressed by nonlinear fitting as: ; the no-cutting energy consumption can be expressed as: , where is the standby power during CNC machining; represents the no-load power (W) of each cutting step; represents the power of the auxiliary system for maintaining the operation of the coolant pump, oil mist separator, etc. during the cutting process. Therefore, the specific energy consumption (SEC) during face milling can be expressed as , MRV is the volume of the removed material ( ).
[0027] Finally, according to the above definitions, the multi-objective optimization mathematical model for the multi-step face milling process can be expressed as: ; where the decision variable consists of the combination of process parameters and the number of steps for each process step, and can be expressed as: 。
[0028] In addition, the above model sets several processing constraints as follows: 1) Requirements for process parameter range: , , ; 2) Requirements for machining allowance: , is the total cutting depth; 3) Requirements for machining power: Ensure that the power required for the cutting operation does not exceed the specified power, , is the rated power of the spindle, is the overall efficiency of the machine tool spindle; 4) Requirements for tool life: The tool life during the cutting process shall not be less than the user-specified tool life ; 5) Requirements for roughness: , Ensure that the machined surface formed by each cutting meets the minimum surface roughness requirement, where and represent the rake angle and clearance angle of the tool respectively; 6) Requirements for finish machining: Finish machining must be performed in the last step, i.e., 。
[0029] Step S2: Construct a cutting machining environment based on the relational model, obtain the machining state in the current environment, and formalize the process parameter optimization problem as a Markov Decision Process (MDP).
[0030] Furthermore, the Markov Decision Process is described and defined by a quadruple (S, A, P, R), where: (1) The state space S represents the machining state, a vector composed of process parameters, machining allowance, and process performance indicators, which can be expressed as: ; (2) The action space A represents the combination of process parameters selected for each cutting step. Select an action according to the current state , which can be expressed as: ; (3) The transition function maps the current state and the corresponding action to the next state ; (4) The reward function It represents the action - corresponding reward calculated according to key performance indicators and processing constraints, that is, the processing decision value.
[0031] As Figure 2 shown, for the cutting - machining process, each cutting task needs to be completed through multi - step cutting. In each step, the process agent selects a process - parameter combination according to a certain strategy based on the machining state of the current step and performs a cutting operation, so as to obtain the machining state after cutting as well as key performance indicators such as electrical - energy consumption and machining time (i.e., the reward ). Then, the cutting for the next step is carried out until the cutting task is completed. )
[0032] In this embodiment, during the entire cutting - machining process, the process - parameter optimization objective is: continuously optimize the process - parameter combination and the number of process steps in each step to maximize the cumulative reward during the machining process, so as to improve the machining efficiency, reduce the electrical - energy consumption, and ensure the machining quality. Specifically, it can be expressed as: ; where is the future cumulative reward, and the policy entropy is used to encourage policy exploration and can be defined as , and
[0033] is the temperature coefficient to control the influence degree of entropy. 1) The action is designed as , which includes the cutting speed (m / min), the feed rate (mm / tooth), and the cutting depth (mm); 2) The state is designed as , where represents the process - parameter combination at the previous moment, represents the current machining allowance, and represents the current machining type (i.e., rough machining or finish machining); 3) The reward function is designed from the two optimization objectives of electrical - energy consumption and machining time, and at the same time, machining constraints are introduced as penalty terms, and can be expressed as: ; where is the normalization function; and are the trade - off coefficients; is the penalty coefficient; is the penalty term, which can be expressed as: ; In the above formula, in the first term and the fifth term is the indicator function, which takes 1 when the whole processing is completed.
[0034] Step S3, in the Markov decision process, use the improved Soft Actor-Critic (SAC) algorithm as the process parameter optimizer to generate a process agent, and through training, output the optimal process parameter combination for multiple processing steps.
[0035] Specifically, the reinforcement learning algorithm mainly consists of two modules: 1) The Critic module is used to evaluate the impact of a specific process parameter combination on the final processing goals (such as cutting efficiency and energy consumption) in different processing states, that is, the processing decision value or state-action value (Q-value).
[0036] Among them, the state-action value is related to the cumulative reward, and can be specifically expressed as: ; In the above formula, is the policy entropy, is the temperature coefficient to control the influence degree of entropy.
[0037] 2) The Actor module is used to learn the optimal action (that is, the process parameter combination) and execute the cutting action according to the evaluation results of the Critic module.
[0038] However, when solving the process parameter optimization problem, traditional reinforcement learning has certain limitations, that is: the Critic module is prone to generate a biased value estimate for the un-explored process parameter space, thus affecting the convergence of policy learning and the acquisition of the global optimal solution; due to the relatively simple network structure, the Actor module may fall into a local optimal solution in the complex and high-dimensional process parameter space. To address the above problems, in order to improve the generalization ability and optimization performance of the overall algorithm and achieve the optimization goal, in this embodiment, the Actor module and the Critic module are structurally improved in a targeted manner, and a new deep reinforcement learning algorithm, that is, the improved Soft Actor-Critic (SAC) algorithm, is proposed. This algorithm is used as the process parameter optimizer to generate a process agent, and through training, obtain the optimal process parameter strategy or combination for multiple processing steps.
[0039] Among them, the improved Soft Actor-Critic algorithm is as Figure 3As shown in the figure, it mainly consists of a Critic module and an Actor module. A neural network is introduced into the Critic module and the Actor module. By parametrically modeling and training the neural network, the non-linear effects of process parameters on the optimization processing objective in different processing states and the optimal action distribution corresponding to different processing states are learned respectively.
[0040] Furthermore, the Critic module is parametrically modeled through a neural network. The input vector consists of the combination of the processing state and process parameters (i.e., the action), and the output value is the processing decision value. To reduce the overestimation bias in value estimation, this module adopts a dual-Critic network mechanism to improve the optimization stability, that is, two Critic networks are constructed 、 ,and the minimum value of the two is taken as the value estimation, as shown in Figure 4 . Among them, the parameters of the Critic network are learned by minimizing the Mean Squared Bellman Error (MSBE), which is: ; In the above formula, represents sampling a batch of state-action pairs from the experience replay pool D (Replay Buffer) ; the target estimate is defined as , is the target Critic network, and its parameters are obtained by the exponential moving average of the Critic network parameters , that is: , is the smoothing update coefficient.
[0041] In this embodiment, to improve the evaluation accuracy of the Critic network, the network structure is mainly based on the KAN network layer (Kolmogorov-Arnold Network). The KAN network layer can replace the linear weight transformation and fixed activation function in the traditional neural network structure by constructing a learnable adaptive activation function. Among them, each KAN network layer can be specifically expressed as: ; In the above formula, is the layer and the layer's adaptive activation function, with a total of ones, is the neuron of each layer, and each adaptive activation function consists of a spline function and a basis function: , and the basis function can be expressed as , and the spline function can be parameterized by the linear combination of B-spline functions , are learnable parameters. In the high-dimensional continuous process parameter space, this network structure has excellent non-linear expression ability, and the B-spline function in its structure can provide better smoothness and approximation ability. Therefore, the KAN network layer can accurately estimate the complex non-linear influence relationship of each process parameter on the optimization target under different processing states. Especially for the locally unexplored area, it can effectively alleviate the estimation error of the Critic network and improve the evaluation accuracy and generalization ability of the Critic network.
[0042] To further strengthen the evaluation ability of process parameters, a new type of energy update layer is introduced in the middle layer of the Critic network, which can be specifically expressed as: ; where, , , , X is the input feature, Z is the output feature, and , and are all trainable weight matrices. Compared with the traditional self-attention mechanism, this energy update layer adopts a multi-iteration update method, and its update rule follows , and outputs after the final update ends.
[0043] The above update mechanism can enhance the feature extraction ability of processing state information, capture local effective features in processing state and process parameters more efficiently, and thus further improve the value estimation ability of the Critic network for the state-action space.
[0044] The Actor module first passes through a neural network, namely the Actor network . The input vector is the current processing state, and the output vector consists of the action mean and the action log standard deviation , forming a Gaussian distribution , as the action distribution related to process parameters. The Actor network parameters are obtained by minimizing the following loss function: ; In the above formula, represents sampling the state from the experience replay pool D, is the policy entropy, Use the temperature coefficient to control the degree of influence of entropy. This loss function can be understood as optimizing the Actor network to obtain a process parameter strategy that maximizes the processing value, while adding an entropy regularization term to increase the exploration of the strategy and avoid premature convergence to suboptimal solutions.
[0045] Furthermore, the Actor network adopts a network structure similar to that of the Critic network, with the KAN network layer as the backbone and an energy update layer added to the middle layer. Through a learnable adaptive activation function, the KAN network layer can effectively identify the complex non-linear relationships between different processing states and process parameter combinations, enhancing the generalization performance of the Actor network when outputting optimal process parameters; while the energy update layer can improve the Actor network's ability to identify key process parameters by optimizing the local feature representation of the current processing state, thereby further improving the optimality of process parameter decision-making.
[0046] However, considering the limited expressive ability of the Actor network, which only outputs process parameter combinations in the form of a unimodal Gaussian distribution, in an actual complex manufacturing environment, high-value process parameter combinations often exhibit multimodal distributions, that is, there may be multiple local optimal solutions rather than a single solution. Therefore, to further improve the strategy optimization ability of the Actor module under complex working conditions, after the Actor network outputs the action distribution, the module introduces a new strategy update method to dynamically adjust the action distribution, making it more effectively explore and converge to high-value regions.
[0047] During the update process, the action distribution output by the Actor network is used as the initial distribution, and a set of action particles is sampled from it , and these particles are updated iteratively multiple times to obtain an optimized action distribution , and its update rule is designed based on Stein variational gradient descent, which can be specifically expressed as: ; ; In the above formula, is the update rate, and is the RBF kernel function , represents the kernel variance. The first term of the formula offsets the action particles to the high-probability region in the optimal action distribution, while the second term represents the repulsive force that encourages the action particles to disperse from the unimodal action distribution. The updated policy entropy can be specifically expressed as: .
[0048] Furthermore, as Figure 3 shown, the training process of the above process intelligent agent includes: Step S3.1, Initialize the network parameters. This network includes the Critic network , and the Actor network and the target Critic network .
[0049] Step S3.2: At each time step t , execute the current action policy in the environment , calculate the new state according to the transition function and the reward function and the reward .
[0050] Step S3.3: Store the experience ( , , , ) in the experience replay pool for subsequent training.
[0051] Step S3.4: Every d steps, update the network parameters, that is:[[]] 1) Update the parameters of the Critic network: ; 2) Update the parameters of the Actor network: ; 3) Update the parameters of the target Critic network: ; Step S3.5: Loop the above steps S3.2 to S3.4 until the optimal process parameter combination is learned through training.
[0052] Combined with Figure 4 shown, in this embodiment, the Critic network consists of three network layers, passing through a KAN layer, an energy update layer, and a KAN layer respectively, and the units of each network layer are [8, 64, 64, 1]. The Actor network consists of four network layers, passing through a KAN layer, an energy update layer, and a KAN layer respectively, and finally passing through two KAN layers in parallel, outputting the mean and logarithmic standard deviation of the action distribution respectively, and the units of each network layer are [5, 64, 64, 64, 6]. The network weight optimization all adopts the Adam optimizer with a learning rate of 3e-4, and the required experience batch for training is 100. Other hyperparameters are shown in Table 2.
[0053] Table 2: Algorithm hyperparameter settings
[0054] During the training process, the process agent continuously interacts with the process parameter optimization environment, accumulates experience, and uses a new deep reinforcement learning algorithm to continuously optimize the process parameter strategy. To explore diverse experiences, the agent performs random actions in the first 500 steps to learn the impact of different process parameter combinations on the total energy consumption and machining time during the cutting process. After the random exploration, the agent outputs specific actions based on the Actor module and updates the parameters in the Critic module and Actor module at regular intervals to continuously optimize the process parameter strategy. The entire training process requires 5000 steps.
[0055] As Figure 5 shown, after 500 steps of random exploration, during the process of the process agent interacting with the environment and accumulating experience, the cumulative reward continuously increases, and the policy randomness continuously decreases and finally stabilizes. This indicates that the proposed algorithm can continuously learn and converge to the optimal parameter combination. By comparing with the optimization results of other deep reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft Actor-Critic (SAC), as shown in Table 3 below, it can be seen that the proposed algorithm can reduce the machining energy consumption by at least 15.4% and the machining time by 14.8%.
[0056] Table 3: Optimization Results of Different Deep Reinforcement Learning Algorithms
[0057] Step S4: According to the optimal number of machining steps and process parameter combination output by the process agent, perform the actual cutting machining task.
[0058] Embodiment 2 This embodiment provides a cutting machining process parameter optimization system based on deep reinforcement learning, including: A relationship model construction module, used to construct a relationship model between process parameters and process indicators for a multi-step cutting machining task; A state acquisition and processing module, used to construct a cutting machining environment based on the relationship model, acquire the machining state in the current environment, and formalize the process parameter optimization problem as a Markov decision process; The process parameter optimization module is used to adopt an improved soft decision-evaluation algorithm as a process parameter optimizer in the Markov decision process to generate a process agent, and through training, output the optimal process parameter combination for multiple process steps. The improved soft decision-evaluation algorithm is as follows: Neural networks are introduced into the Critic module and the Actor module. By parametric modeling and training of the neural networks, the non-linear influence of each process parameter on the optimized machining target under different machining states and the optimal action distribution corresponding to different machining states are respectively learned. The cutting machining execution module is used to execute the actual cutting machining task according to the optimal number of process steps and the process parameter combination output by the process agent.
[0059] Embodiment III This embodiment provides an electronic device, including: a memory for storing executable instructions; a processor for implementing the above method provided in this embodiment when executing the executable instructions stored in the memory.
[0060] Embodiment IV This embodiment also provides a computer-readable storage medium storing executable instructions, which when executed by a processor, will cause the processor to execute the above method provided in this embodiment.
[0061] Embodiment V This embodiment provides a computer program product, which includes executable instructions, and the executable instructions are a kind of computer instructions; the executable instructions are stored in a computer-readable storage medium. When the processor of an electronic device reads the executable instructions from the computer-readable storage medium and the processor executes the executable instructions, the electronic device is caused to execute the above method provided in this embodiment.
[0062] The steps involved in Embodiments II to V above correspond to those in Method Embodiment I. For specific implementation manners, reference may be made to the relevant description part of Embodiment I. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to execute any method in the present invention.
[0063] Those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0064] The above are only the preferred embodiments of the present invention. Although the specific implementation manners of the present invention have been described in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solution of the present invention are still within the protection scope of the present invention.
Claims
1. A method for optimizing cutting process parameters based on deep reinforcement learning, characterized in that, Including: Construct a relationship model between process parameters and process indicators for multi-step cutting processing tasks; Based on the relationship model, construct a cutting processing environment, obtain the processing state in the current environment, and formalize the process parameter optimization problem as a Markov decision process; In the Markov decision process, an improved soft decision-evaluation algorithm is used as the process parameter optimizer to generate a process agent, which outputs the optimal process parameter combination for multi-steps after training; among them, the improved soft decision-evaluation algorithm is: introduce neural networks into the Critic module and the Actor module, and through parametric modeling and training of the neural networks, learn the non-linear influence of each process parameter on the optimization processing target and the optimal action distribution corresponding to different processing states respectively; According to the optimal number of processing steps and the process parameter combination output by the process agent, execute the actual cutting processing task.
2. The optimization method for cutting process parameters based on deep reinforcement learning according to claim 1, characterized in that The process parameters include cutting speed, feed rate, and cutting depth; the process indicators include machining quality, power consumption, and machining time.
3. The optimization method for cutting process parameters based on deep reinforcement learning according to claim 1, wherein In the Markov decision process, each cutting task in the current cutting processing environment is completed through multi-step cutting; among them, the Markov decision process is described and defined by the quadruple (S, A, P, R), the state space S represents the processing state, including process parameters, machining allowance, and process performance indicators; the action space A represents the actions selected for each cutting step, that is, the process parameter combination; the transition function P represents mapping the current processing state and the corresponding action to the next processing state; the reward function R represents the action corresponding reward calculated according to the key performance indicators and processing constraints, that is, the processing decision value; Among them, for each working step, the process intelligent agent selects an action strategy according to the processing state of the current working step to determine the process parameter combination, execute the action, and obtain the processing state after cutting and the reward ; Continuously loop and iterate through each processing step until the multi-step cutting processing task is completed.
4. The optimization method of cutting process parameters based on deep reinforcement learning according to claim 1, characterized in that The Critic module is used to evaluate the influence of a specific process parameter combination on the final processing target in different processing states, that is, the processing decision value; The Critic module is parametrically modeled through a neural network, the input vector is the processing state and action of the current step, and this action is the process parameter combination, and the output value is the processing decision value; Among them, the Critic module adopts a dual-Critic network mechanism, that is, two Critic networks are constructed, and the minimum value of the two is taken as the value estimate; the Critic network consists of a KAN layer and an energy update layer. The KAN layer is used to estimate the non-linear influence of each process parameter on the optimization processing target in different processing states by constructing a learnable non-linear function, and the energy update layer is used to capture the local effective features in the processing state and process parameters.
5. The optimization method of cutting process parameters based on deep reinforcement learning according to claim 1, characterized in that The Actor module is used to learn the optimal action, that is, the process parameter combination and execute the cutting action according to the evaluation result of the Critic module; The Actor module contains a neural network, which consists of a KAN layer and an energy update layer. The input vector of the Actor module is the machining state of the current working step, and the output vector is the distribution of actions corresponding to the process parameter combination. Among them, the action distribution output by the Actor network is used as the initial distribution, and a set of action particles are sampled from it. Based on the update rule, after multiple iterations of updating, the final action, that is, the process parameter combination required for cutting, is obtained.
6. The optimization method of cutting process parameters based on deep reinforcement learning according to claim 1, characterized in that, The training process of the improved soft decision-evaluation algorithm is as follows: Initialize the network parameters, where the network includes a Critic network , and an Actor network and a target Critic network ; At each time step t, execute the current action policy in the environment , calculate the new state according to the transition function and the reward function and the reward ; Store the experience in the experience replay pool for subsequent training; Every d step, perform network parameter update; Loop the above steps until the optimal process parameter combination is learned through training.
7. A cutting process parameter optimization system based on deep reinforcement learning, characterized in that, It includes: A relationship model construction module, which is used to construct a relationship model between process parameters and process indicators for the cutting machining task of multiple working steps. A state acquisition and processing module, which is used to construct a cutting machining environment based on the relationship model, acquire the machining state in the current environment, and formalize the process parameter optimization problem as a Markov decision process. A process parameter optimization module, which is used to use the improved soft decision-evaluation algorithm as a process parameter optimizer in the Markov decision process to generate a process agent. After training, it outputs the optimal process parameter combination for multiple working steps. Among them, the improved soft decision-evaluation algorithm is: introducing neural networks into the Critic module and the Actor module, and through parametric modeling and training of the neural networks, respectively learning the non-linear influence of each process parameter on the optimized machining target under different machining states and the optimal action distribution corresponding to different machining states. A cutting machining execution module, which is used to execute the actual cutting machining task according to the optimal number of working steps and process parameter combination output by the process agent.
8. An electronic device, characterized in that, It includes: A memory, which is used to store executable instructions. A processor, when executing the executable instructions stored in the memory, implements the method for optimizing cutting machining process parameters based on deep reinforcement learning according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, Stored with executable instructions, which are used to cause the processor to implement the method for optimizing cutting machining process parameters based on deep reinforcement learning according to any one of claims 1-6 when executing the executable instructions.
10. A computer program product, characterized in that, The computer program product includes executable instructions, and the executable instructions are stored in a computer-readable storage medium. When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the method for optimizing cutting machining process parameters based on deep reinforcement learning according to any one of claims 1-6.
Citation Information
Patent Citations
Energy-saving optimization method for machining parameters of numerical control lathe in cutting stage
CN113050543A
Process route multi-objective optimization method based on deep reinforcement learning
CN115016405A
Vertical lathe operation task scheduling method and system based on deep reinforcement learning
CN116384666A
Reactive power optimization method based on double-delay depth deterministic strategy gradient
CN116468159A
Oscillation scanning laser arc hybrid welding process parameter optimization method and system
CN117483956A
Cited By
Metal cutting multi-axis linkage control method based on reinforcement learning
CN121050365A
Abrasive water jet machining parameter real-time reverse optimization method based on reinforcement learning
CN122287329A
A numerical control machine tool machining parameter self-optimization method based on deep reinforcement learning
CN122755771A
