Machine tool machining parameter online optimization method and equipment based on reinforcement learning

Abstract

The invention belongs to the related technical field of intelligent manufacturing and numerical control machining, and discloses a machine tool machining parameter online optimization method and device based on reinforcement learning, and the method comprises the steps: S1, calculating a stability discrimination label and a stability margin based on a lobe graph database corresponding to a to-be-optimized machine tool, and generating a stability confidence coefficient; extracting features capable of representing flutter and load change based on the multivariate process signals in different processing states to form feature vectors; training a lightweight neural network based on the feature vector and the original signal window segment to obtain a processing state classifier; s2, performing off-line training on the reinforcement learning model based on the real processing data, the processing state classifier, the stability discrimination label and the stability confidence coefficient; and S3, based on the real-time machining data, the machining state classifier and the reinforcement learning model, the rotating speed of the main shaft and the feeding amount of each tooth are obtained. Parameters can be adjusted in real time according to states.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent manufacturing and CNC machining, and more specifically, relates to a method and device for online optimization of machine tool machining parameters based on reinforcement learning. Background Technology

[0002] In modern high-end equipment manufacturing, CNC milling is a key machining process, and its efficiency and quality directly affect product performance. However, in actual machining processes, due to factors such as tool wear, material inhomogeneity, environmental disturbances, and changes in the stiffness, frequency, and damping ratio of parts during material removal, the dynamic characteristics of the system continuously change. This makes traditional cutting parameter setting methods based on static chatter stability lobe diagrams or empirical formulas difficult to adapt to dynamic working conditions. On the one hand, excessively conservative reduction of cutting parameters to avoid chatter severely restricts machining efficiency; on the other hand, blindly increasing the feed rate or rotational speed can easily trigger severe chatter, causing tool breakage, surface quality deterioration, and even equipment damage.

[0003] Existing adaptive machining methods largely rely on online flutter detection (such as spectral analysis and wavelet transform) combined with rule bases for parameter adjustment, but suffer from problems such as response lag, poor rule generalization ability, and inability to handle multi-objective optimization. In recent years, reinforcement learning has been introduced into the field of machining parameter optimization due to its advantages in sequential decision-making and multi-objective trade-offs. However, conducting online exploratory training directly on real machine tools faces bottlenecks such as low sample efficiency, high safety risks, and convergence difficulties. Furthermore, a significant domain gap exists between the simulation environment (Sim) and real machining (Real), leading to policy transfer failures.

[0004] Therefore, there is an urgent need for an online optimization method that integrates prior knowledge, signal-driven perception, and edge intelligent execution to achieve closed-loop intelligent control from "offline modeling - online fine-tuning - real-time control" and overcome the challenge of adaptive processing parameters in highly dynamic and highly interference scenarios. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and device for online optimization of machine tool processing parameters based on reinforcement learning, which aims to solve the problem that the existing parameter optimization methods cannot adjust the parameters in real time according to the state.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for online optimization of machine tool machining parameters based on reinforcement learning is provided, comprising an offline stage and an online stage: Offline phase: S1. Based on the position of any combination of machining parameters in the lobe diagram database corresponding to the machine tool to be optimized relative to the stability boundary of the lobe diagram, calculate the stability discrimination label and stability margin, and generate the stability confidence based on the obtained stability margin; at the same time, extract the features that can characterize chatter and load changes based on the multivariate process signals under different machining states to form a feature vector; train the lightweight neural network based on the feature vector and the original signal window segment corresponding to the feature vector to obtain the machining state classifier. S2, the reinforcement learning model is trained offline based on real processing data and processing state classifiers, stability discrimination labels and stability confidence; Online phase: S3, input the real-time machining data into the machining state classifier, and then input the machining state category and corresponding confidence level output by the machining classifier into the reinforcement learning model. The reinforcement learning model outputs the spindle speed and feed per tooth.

[0007] Furthermore, within the control period, the reinforcement learning model is updated online using a small-step incremental update method based on real processing data.

[0008] Furthermore, the reward function of the reinforcement learning model is a normalized and smooth composite reward function constructed from multidimensional indicators.

[0009] Furthermore, the expression for the composite reward function is:

[0010] In the formula, Reward for efficiency; To smooth out the penalty; It is a positive incentive or negative reward in the form of exponential decay.

[0011] Furthermore, negative rewards :

[0012] Exponentially decaying positive stimulus :

[0013] In the formula These are the weighting coefficients; This refers to the axial depth of cut. The critical stable axial cutting depth.

[0014] Furthermore, the cutting force is characterized based on a discrete micro-element force model that identifies the instantaneous cutting thickness, and its numerical expression is as follows:

[0015] In the formula, , , These are the tangential, radial, and axial milling forces of the cutting edge element, respectively; h is the instantaneous cutting thickness of the cutting edge element in the feed direction; and dz is the axial length of the cutting edge element. , , These are the cutting force coefficients of the milling cutter in the tangential, radial, and axial directions, respectively. , , These are the cutting edge force coefficients of the milling cutter in the tangential, radial, and axial directions, respectively.

[0016] Furthermore, the cutting force coefficient is calibrated using the average cutting force method; the cutting force signals from each group of experiments are periodically averaged to obtain the average cutting force in the x and y directions, establishing a linear relationship between the average cutting force and the feed per tooth, and based on the obtained linear relationship, the least squares method is used to... A regression solution is performed, where the slope term of the linear regression is used to determine the cutting term coefficient, and the intercept term is used to determine the cutting edge force coefficient.

[0017] Furthermore, real processing data is input into a processing state classifier, which outputs processing state categories and corresponding confidence scores. Then, the reinforcement learning model is trained offline based on the output processing state categories and corresponding confidence scores, as well as stability discrimination labels and stability confidence scores.

[0018] The present invention also provides an online optimization system for machine tool machining parameters based on reinforcement learning. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the online optimization method for machine tool machining parameters based on reinforcement learning as described above.

[0019] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the reinforcement learning-based online optimization method for machine tool machining parameters as described above.

[0020] In summary, compared with the prior art, the online optimization method and equipment for machine tool machining parameters based on reinforcement learning provided by this invention have the following beneficial effects: 1. In the optimization process, this invention incorporates stability labels and confidence levels as core factors in decision-making. This allows parameter optimization to improve efficiency indicators such as material removal rate while actively avoiding unstable regions, reducing the probability of chatter, thereby improving the quality of machined surfaces, suppressing abnormal vibrations, and extending tool life.

[0021] 2. To address the training instability caused by multi-objective and multi-dimensional parameters, this invention uses a composite reward function to reasonably scale / normalize the efficiency index, avoiding the reward being dominated by a single large-scale quantity; at the same time, it introduces a control quantity smoothing mechanism to suppress frequent large-scale parameter tuning, reduce machine tool servo shock and machining process fluctuations, make the output parameters more in line with actual execution constraints, and improve the stability and reliability of the machining process.

[0022] 3. This invention utilizes the time and frequency domain characteristics of online signals such as vibration / current to form state input, and eliminates invalid interference data such as idling and transition sections through processing section identification, thereby increasing the proportion of effective information, reducing the probability of misjudgment and false updates, and making online decision-making less sensitive to noise and non-processing disturbances.

[0023] 4. Compared to solutions that rely solely on offline training and direct application in the field, which can lead to performance degradation, this invention integrates prior strategies and real-time data to update the model in small steps during field operation. This allows the model to adapt to dynamic drift caused by clamping changes, tool wear, temperature drift, etc., and to continuously maintain stability and efficiency indicators.

[0024] 5. This invention uses edge computing and a PLC closed-loop link to achieve low-latency inference and control. While ensuring the deterministic timing of data acquisition and execution, it can respond promptly to changes in processing status, thereby improving the real-time performance, stability, and field availability of the system.

[0025] 6. This invention enables the strategy to be continuously corrected and optimized as working conditions change over a long period of time through online experience accumulation and playback updates, avoiding performance degradation caused by data distribution drift, thereby improving long-term stable processing capability and overall economic efficiency. Attached Figure Description

[0026] Figure 1 This is a flowchart of an online optimization method for machine tool machining parameters based on reinforcement learning, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the division of processing signals according to an embodiment of the present invention; Figure 3 This is a schematic diagram of processing signal feature extraction according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] This invention provides an online optimization method for machine tool machining parameters based on reinforcement learning. The optimization method can achieve adaptive, intelligent and multi-objective collaborative optimization of machining parameters, significantly improve machining stability and efficiency while ensuring surface quality, effectively suppress chatter, and has good generalization ability for working conditions. It is suitable for intelligent CNC machining scenarios with high flexibility and high reliability.

[0029] The optimization method constructs a flutter stability lobe diagram database covering stochastic dynamic parameters, trains a reinforcement learning model offline based on the lobe diagram database, and updates the reinforcement learning model according to actual data. In this way, the reinforcement learning model can achieve real-time adaptive adjustment of spindle speed and feed per tooth, effectively balancing machining efficiency, surface quality and process stability.

[0030] Please see Figure 1 The optimization method includes an offline phase and an online phase. The offline phase includes the following steps: S1. Calculate the stability discrimination label and stability margin based on the position of any combination of machining parameters in the lobe diagram database corresponding to the machine tool to be optimized relative to the stability boundary of the lobe diagram, and generate the stability confidence score based on the obtained stability margin.

[0031] The leaflet diagram database covers randomized dynamic parameters such as system structural stiffness, natural frequency, and damping ratio.

[0032] S1 includes the following sub-steps: S11. Establish a two-degree-of-freedom regenerative chatter dynamic model. Based on the two-degree-of-freedom regenerative chatter dynamic model, obtain high-precision dynamic parameters of the static machine tool machining system through experimental modal analysis (EMA) and cutting force coefficient calibration.

[0033] In one implementation, the process of performing dynamic modeling and obtaining the tool tip frequency response function of a machine tool machining system includes the following steps: (1) Under no-load conditions, clamp the target tool onto the machine tool spindle to ensure that the clamping rigidity is consistent with the actual machining conditions; (2) Apply broadband pulse excitation to the tool tip using an impact hammer, and simultaneously use a triaxial accelerometer or a non-contact laser vibration meter to collect the vibration response signals of the tool tip in the X and Y directions; (3) Based on the collected excitation force signal and vibration response signal, the frequency response function at the blade tip is calculated through frequency domain analysis, i.e. , Where X(ω) and Y(ω) are the displacement response spectra, , The spectrum of excitation force; (4) Modal parameters are identified on the curve corresponding to the frequency response function, and the natural frequencies of the dominant modes are extracted using frequency domain curve fitting methods (such as the complex exponential method or the multi-reference least squares complex frequency domain method). Damping ratio ξ and modal stiffness k; (5) Based on the identified parameters, a two-degree-of-freedom regenerative flutter dynamic model with the blade tip FRF as the core is established for the subsequent construction of flutter stability lobe diagram.

[0034] S12, based on the identified dynamic parameters, a semi-discrete method is used to generate a leaf lobe diagram database covering the entire range of changes in the processing.

[0035] S13. Based on the obtained leaf lobe diagram database, calculate the position of any combination of processing parameters in the leaf lobe diagram database relative to the stability boundary, obtain the stability disk discrimination label and stability margin, and generate the stability confidence based on the obtained stability margin, forming a stability prior output that can be consistently called for offline training and online control.

[0036] Specifically, firstly, a two-degree-of-freedom regenerative flutter dynamic model is established with the frequency response function (FRF) at the knife tip as its core. Its equations of motion are expressed as follows:

[0037] In the formula, , , , , and These are the mass, stiffness, and damping coefficients in the x and y directions, respectively.

[0038] The key modal parameters affecting the dynamic stiffness and damping characteristics of the system were then identified, including the natural frequency, modal stiffness, modal mass, and modal damping ratio of the dominant mode. For parameter identification, experimental modal analysis (EMA) with impact hammer excitation was used to obtain the frequency response function (FRF), and high-precision identification of the modal parameters was achieved through FRF fitting, ensuring the acquisition of highly robust modal parameters under actual working conditions.

[0039] To achieve accurate construction of the Stability Leaf Lobe Diagram (SLD), this invention also identifies the cutting force coefficient during the cutting process. The cutting force is characterized based on a discrete micro-element force model that identifies the instantaneous cutting thickness, and its numerical expression is as follows:

[0040] In the formula, , , These are the tangential, radial, and axial milling forces of the cutting edge element, respectively; h is the instantaneous cutting thickness of the cutting edge element in the feed direction; and dz is the axial length of the cutting edge element. , , These are the milling force coefficients of the milling cutter in the tangential, radial, and axial directions, respectively; , , These are the cutting edge force coefficients of the milling cutter in the tangential, radial, and axial directions, respectively.

[0041] For simplified two-dimensional models, we typically only focus on the dynamic response of the tool tip in the two principal vibration directions. Since the contributions of axial vibration and axial force to regenerative chatter are relatively small, it is only necessary to identify the tangential and radial cutting force coefficients. .

[0042] The cutting force coefficient was calibrated using the average cutting force method: a series of calibration cutting experiments were designed, with the axial depth of cut and radial width of cut fixed, while only the feed per tooth was varied. The cutting force signals from each group of experiments were periodically averaged to obtain the average cutting force in the x and y directions. A linear relationship between the average cutting force and the feed per tooth was established. Based on this linear relationship, the least squares method was used to... A regression solution is performed, where the slope term of the linear regression is used to determine the cutting term coefficient, and the intercept term is used to determine the cutting edge force coefficient. The specific average milling force can be written as:

[0043] In the formula, and denoted by z, represents the average cutting force component within a stable spindle speed range; z represents the number of tool teeth. The axial cutting depth, This refers to the feed per tooth. and These are the geometric mean coefficients determined by the entry and exit angles, respectively, and can be calculated analytically using machining geometry parameters.

[0044] The structural stiffness k and natural frequency obtained from the identification The damping ratio ξ is subjected to a uniform random disturbance of ±20% to generate no less than 8000 sets of parameter combination samples; for each set of samples (dynamic parameter combination), the actual tool geometry parameters and the calibrated cutting force coefficient are combined. , The corresponding milling dynamics delay differential equation is established; the milling dynamics delay differential equation is solved using the semi-discrete time-domain method, and the critical stable axial depth of cut at different spindle speeds n is calculated. ; with radial cutting width As an external parameter, it is divided into several levels according to typical processing scenarios, and for each... Repeat the above steps to generate the corresponding stability leaflet curves; organize all calculation results into a multidimensional query database, with the input index being... The output is a mapping relationship. It supports online fast interpolation query of the maximum permissible axial cutting depth under any working condition, and finally constructs a high-dimensional flutter stability leaf lobe diagram (SLD) database. This leaf lobe diagram database is based on ( k , , ξ , ) as the input index, with ( n , ) is the output mapping, comprehensively covering typical roughing / finishing conditions; where The radial cut width remains unchanged within a single machining operation.

[0045] In one implementation, the cutting force coefficient , and cutting edge force coefficient , The calibration is performed using the average cutting force method. The specific procedure is as follows: (1) Under steady-state cutting conditions, design multiple sets (no less than 4 sets) of different axial depths of cut. With feed per tooth The calibration experiment kept the spindle speed n and radial tangent width constant. The process should be constant, ensuring no significant chatter during manufacturing.

[0046] (2) The cutting force signals during the stable cutting stage of each group of experiments were collected synchronously using a three-dimensional force measuring instrument, and then the average cutting force in the X and Y directions was calculated. , .

[0047] (3) Determine the angle of entry based on the tool geometry parameters and machining method (such as slot milling, climb milling). and the cutting angle Then the geometric mean coefficient is calculated. , .

[0048] (4) The average cutting force formula is transformed into a system of linear equations about the coefficients to be identified. The least squares method is used to solve the obtained system of linear equations to obtain high-precision solutions. , , , parameter.

[0049] Based on the stability leaflobe diagram database obtained from S12, in each set of randomized dynamic parameter samples and given speed n Under these conditions, the corresponding critical stable axial cutting depth can be found using SLD:

[0050] That This represents the stability boundary under the given stochastic dynamic parameters and rotational speed conditions. This mapping relationship, in the form of a database index, can be written as... .

[0051] To generate a stable prior output consistent with the online output, for any candidate combination of processing parameters... Define its normalized boundary ratio relative to the stability boundary:

[0052] in >0 is a minimal constant to prevent the denominator from being zero. Based on Provide stability discrimination labels :

[0053] Where d>0 indicates that the region is in a stable region and the farther away from the boundary, the more stable it is; d<0 indicates that the region has crossed the boundary and the degree of instability is higher.

[0054] Stability confidence is generated from stability margin. Using a monotonic mapping:

[0055] In the formula, This is the slope coefficient.

[0056] S2, based on the multivariate process signal extraction under different processing conditions, can characterize the features of flutter and load changes to form feature vectors; based on the feature vectors and the original signal window segments corresponding to the feature vectors, a lightweight neural network is trained to obtain a processing state classifier.

[0057] S2 includes the following sub-steps: S21. Window analysis is performed on vibration and current signals collected under different processing conditions. Energy features such as root mean square (RMS) are extracted as indicators of rapid and interpretable state differences, and the samples are preliminarily segmented and screened accordingly.

[0058] S22 preprocesses the obtained windowed signal, including smoothing, low-pass filtering, and downsampling, to reduce noise interference and unify the data caliber for offline training and online inference.

[0059] S23, perform time-domain / frequency-domain / time-frequency-domain feature extraction on the preprocessed window data, and construct a supervised sample set by combining manual annotation. Input the feature vector and the original signal window segment corresponding to the feature vector into the training lightweight neural network (MLP) for training to obtain the processing state classifier. When the processing state classifier runs online, it can directly output the current processing state category and confidence level, and use it as the state input or constraint signal of the subsequent reinforcement learning model.

[0060] Specifically, windowing processing is performed on the vibration and current signals acquired online. Let the vibration signal be... The current signal is Based on sampling rate Discretized after collection The data of the k-th window for:

[0061] In the formula, N is the number of sampling points corresponding to the window length, and H is the number of sampling points corresponding to the window step size.

[0062] Within each time window, energy indices such as RMS are calculated to characterize the amplitude variation trend of the machining process from idling, tool feed, stable cutting to tool exit, thus providing a basis for sample construction and manual annotation. In the supervised sample set construction stage, window segments of different stages can be manually annotated according to preset rules to form a correspondence between "signal window - state label" to achieve supervised learning of the machining state.

[0063] Considering that the signals acquired in actual processing often contain a large amount of interference and exhibit non-stationarity, a sliding window is preferably used to smooth the original signal, and low-pass filtering is applied to suppress high-frequency noise. Since most cutting process energy is distributed in the low-frequency range, a 2000Hz low-pass filter is required for the original signal. The sampling rates of the current and vibration signals are determined by the PLC's scan cycle and the acquisition port. In one embodiment, the PLC scan cycle is 1ms, the current signal sampling frequency is 1kHz, and the vibration signal sampling frequency is 10kHz. The above processing is used to unify the feature calculation criteria and improve the robustness and transferability of the subsequent reinforcement learning model.

[0064] In terms of feature construction, it is preferable to extract time-domain, frequency-domain, and time-frequency-domain features simultaneously to enhance the observability of flutter-related dynamic changes and load fluctuations: time-domain features include standard deviation, variance, kurtosis, RMS, etc.; frequency-domain features include FFT amplitude mean, standard deviation, maximum value, total power, and dominant frequency position, etc.; time-frequency-domain features include instantaneous frequency variance, time energy variance, frequency center standard deviation, and flutter-specific features such as flutter frequency band energy ratio, harmonic features, number of spectral peaks, envelope spectrum standard deviation, wavelet capability, etc.

[0065] In one implementation, the process of constructing a state representation aligned with simulation based on online sensing signals is as follows: (1) The processing state is explicitly distinguished by extracting the RMS value of the signal. The signals of different stages are segmented online. Only the data of the effective part is analyzed and the features are extracted. The signals in the idling, entry and exit stages need to be ignored.

[0066] (2) The original signal is smoothed using a sliding window. Most of the energy in the common cutting process is distributed in the low frequency band, so the original signal needs to be low-pass filtered at 2000Hz. At the same time, due to the high sampling frequency, the original signal needs to be downsampled, with the current signal downsampled to 500Hz and the vibration signal downsampled to 5000Hz.

[0067] (3) Constructing state sequences by combining time-frequency analysis and deep learning: Perform time-domain analysis, frequency-domain analysis and time-frequency domain analysis on the collected time-domain signals to screen out signal features that are sensitive to flutter. Specifically: time-domain features include standard deviation, variance, kurtosis, RMS, etc.; frequency-domain features include FFT amplitude mean, standard deviation, maximum value, total power, dominant frequency position, etc.; time-frequency domain features include instantaneous frequency variance, time energy variance, frequency center standard deviation, and specific flutter features: flutter frequency band energy ratio, harmonic features, number of spectral peaks, envelope spectrum standard deviation, wavelet capability analysis, etc.

[0068] Based on the aforementioned feature vectors and corresponding original signal window segments, a supervised sample set is constructed, and a lightweight neural network (MLP) is trained to obtain a processing state classifier. During online operation, the real-time acquired signals are preprocessed and feature extracted using the same standard. The obtained feature vectors and their corresponding original window segments are then input into the processing state classifier, which directly outputs the current processing state category and confidence level. The output of the processing state classifier is used as the state input or constraint signal for subsequent reinforcement learning models, enabling reinforcement learning to explicitly perceive the current processing stage and instability risk during decision-making, thereby achieving a closed-loop linkage of "state recognition—parameter decision-making".

[0069] S3, The reinforcement learning model is trained offline based on real processing data and processing state classifiers, stability discrimination labels and stability confidence; wherein, the reward function of the reinforcement learning model is a normalized and smooth composite reward function constructed by multi-dimensional indicators.

[0070] Real processing data is input into the processing state classifier, which outputs the processing state category and the corresponding confidence level. Then, the reinforcement learning model is trained offline based on the output processing state category and the corresponding confidence level, as well as the stability discrimination label and stability confidence level.

[0071] S3 includes the following sub-steps: S31, based on the feature vectors and processing parameters determined in S2, processes the data collected during the actual processing to form an offline interactive dataset. Stability discrimination labels and confidence scores serve as risk supervision signals for offline training.

[0072] S32 uses the Soft Actor-Critic (SAC) maximum entropy reinforcement learning algorithm as its training framework.

[0073] S33, design a composite reward function that includes three objectives: efficiency, stability and smoothness, to improve policy robustness and generalization ability, and conduct offline training on the reinforcement learning model based on a preset set of working conditions, and obtain transferable offline policy priors based on the reinforcement learning model.

[0074] Please see Figure 2 and Figure 3 To improve the strategy's generalization ability and coverage of typical boundary conditions, several preset working condition sets, including heavy cutting, roughing, semi-finishing, and finishing, are set based on the SLD database. These working condition combinations are defined by parameter ranges: heavy cutting conditions correspond to larger... and Rough machining conditions correspond to medium to large sizes. With medium Semi-finishing conditions correspond to medium With smaller The finishing process corresponds to a smaller size. and This allows offline training data to cover typical boundaries from "low-load, high-speed finishing" to "high-load / medium-speed heavy cutting".

[0075] Maximum entropy reinforcement learning, while maximizing cumulative reward, introduces a policy entropy regularization term to encourage exploration and improve policy robustness. Its optimization objective can be written as:

[0076] In the formula, and Represent the state and action at time t, respectively; For instant rewards; For a given state, a random policy; For policy entropy; is the entropy regularization coefficient, used to balance "maximizing reward" and "exploration level".

[0077] SAC employs a dual-Q network to mitigate the problem of overestimation of Q-values, i.e., using the smaller of the two values ​​during policy updates. For continuous action spaces, policies are often modeled as Gaussian distributions and output the mean and standard deviation. To make the sampling process differentiable, a reparameterization technique is used to rewrite the action sampling as follows:

[0078] In the formula, This represents the Hadamard product (element-by-element multiplication). The noise vector is sampled from a standard normal distribution. I represents a matrix with a mean of 0 and a covariance of 0.

[0079] In scenarios where actions are bounded, further methods can be adopted. The mapping yields the final action:

[0080] In the formula, For unlimited motion variables; This refers to actual actions.

[0081] The optimization objective of the policy network can be written as:

[0082] In the formula, R represents experience playback (offline dataset); For the j-th network; Let be the logarithmic probability of the policy with respect to the current action.

[0083] To enable adaptive adjustment of exploration intensity under different states, SAC uses the entropy regularization coefficient. Automatic learning is achieved through constrained optimization, with the target entropy set as... ,get The loss function is:

[0084] When the policy entropy is lower than the target value (i.e.) (Too small, insufficient exploration) At that time, the above formula is promoted. Increase, thus in Increasing the weight of the entropy term enhances exploration; when the policy entropy is higher than the target value... The reduced size allows training to focus more on value enhancement and stable convergence.

[0085] Material Removal Rate (MRR) is defined as:

[0086] Considering The order of magnitude is ,and n The order of magnitude is usually To avoid the reward being dominated by large-scale variables and to enhance training stability, logarithmic compression normalization is applied to MRR, and an efficiency reward is defined. for:

[0087] Based on this, a motion smoothness penalty term is introduced to suppress drastic fluctuations in control commands. Let the continuous action vector output by the reinforcement learning model at time t be... The smoothing penalty is defined as the average amplitude of the change in action at adjacent time points:

[0088] In the formula d represents the weighting coefficient, and d represents the dimension of the action vector.

[0089] By combining a chatter stability lobe diagram (SLD) database constructed based on the semi-discrete method, the current cutting parameter combination is determined in real time under given stochastic dynamic parameters. Is it stable? If it is determined to be unstable, then a significant negative reward is imposed. :

[0090] If it is determined to be stable, then based on the current cutting depth... Maximum permissible stable depth of cut at the corresponding rotational speed The degree of proximity provides a positive incentive in the form of exponential decay. :

[0091] In the formula Weighting coefficients (can be taken as follows) ).

[0092] Final reward function Defined as:

[0093] In offline reinforcement learning, the state vector includes dynamic parameters. and milling machining parameters ( The agent's actions are... In one processing step, maintain ( Unchanged, output online by the reinforcement learning model This aims to achieve a comprehensive optimization of efficiency and stability. Specifically, it takes into account the specific characteristics of different processing stages. The parameters are combined, so several specific combinations of process parameters are set, such as heavy cutting, roughing, semi-finishing and finishing.

[0094] The steps in the online phase are as follows: S4, input the real-time machining data into the machining state classifier, and then input the machining state category and corresponding confidence level output by the machining classifier into the reinforcement learning model. The reinforcement learning model outputs the spindle speed and feed per tooth. Within the control cycle, the reinforcement learning model is updated online using a small-step incremental update method based on the real machining data.

[0095] S4 includes the following sub-steps: S41, deploy the reinforcement learning model obtained by offline training based on the SLD database in S3 to the actual processing environment and establish an online data input link: acquire vibration signals and spindle current signals in real time during the processing through the data acquisition module, and construct the model input according to the preprocessing, windowing and time-frequency analysis caliber determined in S2; the processing state recognition classifier (MLP) obtained by step S2 outputs the current processing state category and its confidence level for each control cycle, and uses the recognition result as the online state input of the reinforcement learning model, thereby ensuring that the real processing scene and the simulation environment are consistent at the level of "signal caliber - state expression - control cycle".

[0096] S42 integrates offline policy priors and real processing data within the online control cycle to perform small-step incremental updates on the reinforcement learning model, achieving Sim2Real transfer: a time-scale decay mechanism for prior weights is preferably introduced, so that the policy follows the leaf lobe stability prior more in the early stage of online deployment, and gradually increases its dependence on real signals after accumulating sufficient online samples.

[0097] Specifically, a time-decaying weight can be set for the "SLD-based stability constraint / prior reward term". Furthermore, a time-increasing weight was assigned to "state recognition results and risk evidence driven by online signals". :

[0098] In the formula, It is a time constant; accordingly .

[0099] Evidence of risk can be provided by the instability probability, confidence level, or optional stability margin estimate output by the MLP. Based on the aforementioned fusion weights, incremental updates are performed on the online playback data, enabling the strategy to dynamically adjust the spindle speed n and feed per tooth online while ensuring stability constraints. This enables a smooth transition from "leaf-shaped prior driving" to "signal-driven control automation optimization," and adapts to actual working condition changes such as clamping differences, position-dependent dynamic drift, and tool wear.

[0100] In practical use, edge computing units are deployed to achieve low-latency execution of the control strategy, and a real-time closed loop is achieved through the acquisition-processing-distribution link between the PLC and the host computer. The PLC's scanning cycle and sampling / uploading time are used as data caliber constraints to guide the real data windowing, incremental update and feature calculation parameter configuration in step S2, as well as the calculation frequency and refresh time of the stability prior output of S1 within the control cycle. The reinforcement learning strategy is continuously optimized through an online experience playback mechanism to improve the long-term adaptability to changes in operating conditions.

[0101] The PLC's scanning cycle and sampling / uploading time are used as data caliber constraints to guide the real data windowing, incremental updates, and feature calculation parameter configuration of S2, as well as the calculation frequency of S1's stability prior output within the control cycle. Meanwhile, the edge continuously updates samples and optimizes reinforcement learning strategies through an online experience replay mechanism to improve the long-term adaptability to changes in operating conditions.

[0102] Specifically, it includes the following sub-steps: S51: Deterministic Acquisition and Cyclic Reference Establishment. A Beckhoff PLC is used as the underlying real-time reference, with its scan cycle (1ms) completing deterministic acquisition of multi-source signals, buffer updates, and execution interface maintenance. Vibration and spindle current signals can be connected through corresponding EtherCAT terminal modules, and the equivalent sampling rate is configured based on the terminal's sampling capability within a single scan cycle, thereby ensuring observability while controlling the online data volume and processing burden.

[0103] In one implementation, a Beckhoff C6030 series ultra-compact industrial control cabinet PC is used as an edge device. Communication is achieved through an EK1100 EtherCAT coupler. The EL3104 module is used to acquire the three-phase current of the spindle, the EL3632 module is used to acquire the three-dimensional vibration acceleration signal, the EL1018 is used to read the feed rate inside the machine tool, and the EL2008 is used to write the speed and feed rates to be set into the machine tool. The TwinCAT3 is used inside the C6030 to complete the data acquisition and preprocessing operations.

[0104] S52: Edge-end incremental processing and strategy inference deployment. An edge intelligent node is constructed using a Beckhoff C6030 industrial PC, integrating signal preprocessing, feature extraction, and reinforcement learning strategy inference functions. It interacts periodically with the PLC via ADS / EtherCAT and other methods. Within each control cycle, the edge performs time-frequency analysis / feature calculation and state construction on the latest window data provided by the PLC. The obtained state input is then fed into the offline-trained reinforcement learning model for strategy inference, outputting action parameters, which are then written back to the PLC. The PLC performs deterministic writing to the machine tool interface, ensuring that the closed-loop control cycle meets real-time requirements.

[0105] S53: The PLC packages and uploads the collected data to the edge terminal / host computer at a fixed pace, enabling the edge terminal to complete windowing and incremental updates at a stable refresh rhythm. The "scanning cycle - sampling rate - upload pace" together limit the online available window length, step size and feature refresh frequency. Based on this, the sliding window division, incremental update strategy and feature calculation parameters of the real data are configured in S2. In S1, a matching stability label and confidence level are generated to avoid input distribution offset caused by the inconsistency between offline and online standards.

[0106] S54: At the edge, without affecting the real-time closed loop, the "state-action-report-next state" interaction samples in the online machining process are written into the experience replay cache, and the replay samples are periodically extracted in the background / low priority mode to update the reinforcement learning strategy in small steps or fine-tune the parameters. By continuously supplementing the samples, the system covers working condition samples such as new tool wear, clamping stiffness changes, and speed range changes, so as to achieve long-term adaptation and robustness improvement of the strategy to working condition drift.

[0107] The present invention also provides an online optimization system for machine tool machining parameters based on reinforcement learning. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the online optimization method for machine tool machining parameters based on reinforcement learning as described above.

[0108] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the reinforcement learning-based online optimization method for machine tool machining parameters as described above.

[0109] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for online optimization of machine tool machining parameters based on reinforcement learning, characterized in that, Includes offline and online phases: Offline phase: S1. Calculate the stability discrimination label and stability margin based on the position of any combination of machining parameters in the lobe diagram database corresponding to the machine tool to be optimized relative to the stability boundary of the lobe diagram. Generate the stability confidence based on the obtained stability margin. At the same time, extract the features that can characterize chatter and load changes based on the multivariate process signals under different machining states to form a feature vector. A lightweight neural network is trained based on feature vectors and the corresponding original signal window segments to obtain a processing state classifier; S2, the reinforcement learning model is trained offline based on real processing data and processing state classifiers, stability discrimination labels and stability confidence; Online phase: S3, input the real-time machining data into the machining state classifier, and then input the machining state category and corresponding confidence level output by the machining classifier into the reinforcement learning model. The reinforcement learning model outputs the spindle speed and feed per tooth.

2. The online optimization method for machine tool machining parameters based on reinforcement learning as described in claim 1, characterized in that: Within the control period, the reinforcement learning model is updated online using a small-step incremental update method based on real processing data.

3. The online optimization method for machine tool machining parameters based on reinforcement learning as described in claim 1, characterized in that: The reward function of the reinforcement learning model is a normalized and smooth composite reward function constructed from multidimensional indicators.

4. The online optimization method for machine tool machining parameters based on reinforcement learning as described in claim 3, characterized in that: The expression for the composite reward function is: In the formula, Reward for efficiency; To smooth out the penalty; It is a positive incentive or negative reward in the form of exponential decay.

5. The online optimization method for machine tool machining parameters based on reinforcement learning as described in claim 4, characterized in that: Negative Rewards : positive exponentially decaying form of exponentially decaying excitation : In the formula These are the weighting coefficients; This refers to the axial depth of cut. The critical stable axial cutting depth.

6. The online optimization method for machine tool machining parameters based on reinforcement learning as described in any one of claims 1-5, characterized in that: The cutting force is characterized by a discrete micro-element force model based on the instantaneous cutting thickness, and its numerical expression is as follows: In the formula, , , These are the tangential, radial, and axial milling forces of the cutting edge micro-element; h is the instantaneous cutting thickness of the cutting edge element in the feed direction; dz is the axial length of the cutting edge element. , , These are the cutting force coefficients of the milling cutter in the tangential, radial, and axial directions, respectively. , , These are the cutting edge force coefficients of the milling cutter in the tangential, radial, and axial directions, respectively.

7. The online optimization method for machine tool machining parameters based on reinforcement learning as described in claim 6, characterized in that: The cutting force coefficient was calibrated using the average cutting force method. The cutting force signals from each group of experiments were periodically averaged to obtain the average cutting force in the x and y directions. A linear relationship between the average cutting force and the feed per tooth was established. Based on this linear relationship, the least squares method was used to... A regression solution is performed, where the slope term of the linear regression is used to determine the cutting term coefficient, and the intercept term is used to determine the cutting edge force coefficient.

8. The online optimization method for machine tool machining parameters based on reinforcement learning as described in any one of claims 1-5, characterized in that: Real processing data is input into the processing state classifier, which outputs the processing state category and the corresponding confidence level. Then, the reinforcement learning model is trained offline based on the output processing state category and the corresponding confidence level, as well as the stability discrimination label and stability confidence level.

9. An online optimization system for machine tool machining parameters based on reinforcement learning, characterized in that: The system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it performs the online optimization method for machine tool machining parameters based on reinforcement learning as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the online optimization method for machine tool machining parameters based on reinforcement learning as described in any one of claims 1-8.

Images