Policy neural network output path smoothing method based on Kalman filtering

Abstract

The invention discloses a strategy neural network output path smoothing method based on Kalman filtering, and relates to the technical field of numerical control machining and intelligent control, and the method comprises the following steps: S1, constructing a state vector containing a tool space position and a motion speed; s2, generating a corresponding observation vector based on a path point sequence output by the strategy neural network; s3, establishing a system state prediction equation and an observation equation based on the state vector and the observation vector; and S4, based on the system state prediction equation and the observation equation, recursive estimation is performed on the path state through a Kalman filtering algorithm so as to realize smoothing processing of the tool path. The method has the advantages of effectively restraining path jitter and speed sudden change, improving machining precision and stability and avoiding overload of a machine tool servo system and abnormal abrasion of a tool.

Description

Technical Field

[0001] This invention relates to the fields of CNC machining and intelligent control technology, and more specifically, to a method for smoothing the output path of a strategy neural network based on Kalman filtering. Background Technology

[0002] In the field of intelligent manufacturing, automatic generation of milling machining paths based on reinforcement learning has become a research hotspot. Policy neural networks, through interaction with the environment, can output machining paths that meet material removal rates and physical constraints. However, these methods face significant challenges in practical applications: due to the inherent exploratory nature of reinforcement learning algorithms, differences in network parameter initialization, and uneven distribution of training data, the generated path point sequences often exhibit irregular local jitter and abrupt speed changes. This path smoothness problem can lead to a series of process defects in high-speed, high-precision machining scenarios, including machine tool servo system overload, cutting chatter, deterioration of machined surface quality, and abnormal tool wear.

[0003] Traditional path smoothing techniques primarily employ post-processing methods, such as B-spline interpolation and moving average filtering. While B-spline methods can achieve global path smoothing, their strong dependence on control points makes them ill-suited for unstructured path sequences output by reinforcement learning. Moving average filtering, although computationally simple, introduces significant phase lag and fails to effectively handle velocity discontinuities. More critically, existing methods generally lack explicit modeling capabilities for path dynamics, failing to effectively suppress high-frequency noise while maintaining geometric accuracy. This results in smoothed paths potentially still violating machine tool dynamics constraints.

[0004] In the field of robot control, existing research has attempted to apply state estimation methods such as Kalman filtering to policy output smoothing. However, these methods are mainly aimed at scenarios such as navigation or game AI, and their system models and parameter settings are difficult to directly transfer to the field of CNC machining. In particular, milling has strict requirements for path accuracy and dynamic performance, requiring the establishment of a dedicated state-space model to accurately describe the tool motion characteristics. Current technology has not yet proposed a systematic solution to combine Kalman filtering with policy neural networks to achieve real-time smoothing of machining paths.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] In view of this, the present invention provides a method for smoothing the output path of a strategy neural network based on Kalman filtering, which has the advantages of effectively suppressing path jitter and speed abrupt changes, improving machining accuracy and stability, and avoiding machine tool servo system overload and abnormal tool wear.

[0007] This invention provides a method for smoothing the output path of a policy neural network based on Kalman filtering, comprising the following steps: S1. Construct a state vector containing the tool's spatial position and velocity; S2. Generate the corresponding observation vector based on the path point sequence output by the policy neural network; S3. Based on the state vector and observation vector, establish the system state prediction equation and observation equation; S4. Based on the system state prediction equation and observation equation, the path state is recursively estimated using the Kalman filter algorithm to achieve smoothing of the tool path.

[0008] In one optional implementation, S1 specifically includes: Will t The true three-dimensional coordinates of the tool center point at all times and three-axis true velocity components Combined to form a six-dimensional state vector : .

[0009] In one optional implementation, S2 specifically includes: The tool is obtained through a policy neural network. t The six-dimensional observation vector output at time step The observation vector Three-dimensional coordinates of the tool and triaxial velocity estimation components ; ; Among them, the triaxial velocity estimation component is achieved by... t Three-dimensional coordinates of time 3D coordinates compared to the previous time step Perform difference operations to obtain the triaxial velocity estimation components. ; The calculation formula for the triaxial velocity estimation components is as follows: .

[0010] In one optional implementation, S3 specifically includes: S31. Using a linear constant velocity model, construct the state transition matrix. : ; in, For time step; S32, Based on the state transition matrix and state vector Establish the system state prediction equation: ; in, for t The state vector at any given time; for t The state vector at time -1; This is process noise; S33. Construct the observation matrix : ; S34, Based on observation matrix and observation vector Establish the observation equation: ; in, To observe noise.

[0011] In one optional implementation, S4 specifically includes: S41. Prediction Phase: State prediction: based on t State estimate updated at time -1 and state transition matrix ,calculate t Predicted state estimate at time 1 ; ; Covariance prediction: Synchronous update t Time prediction error covariance matrix Incorporation process noise characteristics: ; in, For process noise The covariance matrix; for t The error covariance matrix updated at time -1; for t The prediction error covariance matrix at time t; S42, Kalman gain calculation: based on t Prediction error covariance matrix at time step Observation matrix Observe the noise characteristics and calculate the Kalman gain; ; in, To observe noise The covariance matrix.

[0012] S43, Update Phase: State update: based on Kalman gain and fused t Observation vector at time Compared with the predicted state estimate Obtain the smoothed state estimate. : ; Covariance Update: Adjustment t The prediction error covariance matrix at time step 1 is used for the next iteration: ; in, for t The error covariance matrix updated at each time step; S44. The smoothed state estimate Extracting the three-dimensional coordinate components as a smoothed path point sequence for CNC machine tool trajectory control; S45. Repeat steps S41 to S44 to perform real-time smoothing of the path point sequence output by the policy neural network.

[0013] As can be seen from the above, the policy neural network output path smoothing method provided in this application, by constructing a state space model and applying the Kalman filtering algorithm to recursively estimate the path point sequence output by the policy neural network, effectively solves the problems of path jitter and speed change. It has the advantages of effectively suppressing path jitter and speed change, improving machining accuracy and stability, and avoiding machine tool servo system overload and abnormal tool wear. Attached Figure Description

[0014] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a method for smoothing the output path of a policy neural network based on Kalman filtering, according to an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] like Figure 1 As shown in the figure, this application proposes a method for smoothing the output path of a policy neural network based on Kalman filtering, including the following steps: S1. Construct a state vector containing the tool's spatial position and motion velocity.

[0018] S2. Generate the corresponding observation vector based on the path point sequence output by the policy neural network.

[0019] S3. Based on the state vector and observation vector, establish the system state prediction equation and observation equation.

[0020] S4. Based on the system state prediction equation and observation equation, the path state is recursively estimated using the Kalman filter algorithm to achieve smooth tool path processing.

[0021] In practical applications, constructing a state vector that includes the tool's spatial position and velocity can be understood as defining a mathematical model describing the tool's motion characteristics, which can be achieved in various ways. For example, the state vector can be composed of the tool's coordinate components in three-dimensional space and their corresponding velocity components. Specifically, position and velocity information in a Cartesian coordinate system can be chosen as state variables, or equivalent expressions in other coordinate systems (such as polar or cylindrical coordinate systems) can be used. Furthermore, the dimension of the state vector can be expanded or reduced according to actual needs, for example, including only position information or adding acceleration components, mainly to achieve a comprehensive description of the tool's motion state.

[0022] Generating observation vectors from the pathpoint sequence output by a policy neural network can be understood as transforming the pathpoints output by the neural network into observation data that can be used for filtering. Specifically, observation vectors can be generated by directly extracting the geometric coordinates of the pathpoints and combining this with difference operations to calculate the velocity components.

[0023] Recursive path state estimation using the Kalman filter algorithm can be understood as using a recursive algorithm to update and optimize the path state in real time. Specifically, the Kalman filter algorithm can be implemented through standard Kalman filtering, extended Kalman filtering, or unscented Kalman filtering, primarily to achieve dynamic estimation of the path state and noise suppression.

[0024] The innovation of this application lies in introducing Kalman filtering technology to transform the path point sequence output by the policy neural network into a state vector containing the tool's spatial position and velocity. This vector is then combined with system state prediction and observation equations to achieve recursive estimation of the path state. This method not only effectively eliminates local jitter and velocity discontinuities in the path point sequence but also improves the dynamic characteristics of the path while maintaining its geometric accuracy, thus meeting the trajectory control requirements of high-speed and high-precision machining scenarios.

[0025] The working principle of this application embodiment is as follows: A Kalman filter-based strategy neural network output path smoothing method is proposed. By constructing a state vector containing the tool's spatial position and motion velocity, it combines the geometric information and dynamic characteristics of path points, thereby providing a complete state description for subsequent recursive estimation. Specifically, in step S1, the state vector not only captures the tool's coordinates in three-dimensional space but also explicitly introduces three-axis velocity components. This design effectively solves the technical deficiency of relying solely on position information and failing to constrain acceleration changes, ensuring the completeness of the path state description.

[0026] Furthermore, in step S2, a corresponding observation vector is generated based on the path point sequence output by the policy neural network, wherein velocity estimation components are extracted from adjacent path points through differential operations. This approach fully utilizes the temporal correlation of the path point sequence, transforming the static coordinate output of the neural network into dynamic observation data with implicit motion trends, thereby compensating for the velocity discontinuity problem caused by random exploration and making the observation data closer to the real physical process.

[0027] In step S3, based on the state vector and observation vector, the system state prediction equation and observation equation are established. Specifically, a linear constant velocity model is used to construct the state transition matrix to achieve a natural transition of path changes; simultaneously, an observation matrix is ​​designed to achieve a direct mapping between state and observation. This modeling approach can distinguish between process noise and observation noise, providing a precise mathematical framework for Kalman filtering and avoiding the excessive dependence on the global path in traditional interpolation methods.

[0028] Finally, in step S4, the path state is recursively estimated using the Kalman filter algorithm based on the system state prediction equation and the observation equation. This process dynamically fuses the prediction model with real-time observation data, adjusting the estimation weights according to the noise covariance in each iteration, thus suppressing high-frequency jitter while preserving key path features. This achieves smooth toolpath processing, solves the problems of local jitter and speed discontinuity in the output path point sequence of the strategy neural network, and avoids process defects such as excessive acceleration of the machine tool servo system, cutting chatter, and decreased surface quality.

[0029] Through the coordinated operation of the above steps, this method can significantly improve the continuity of the path and the dynamic response performance of the machine tool while ensuring the geometric accuracy of the path, thus meeting the actual needs of high-speed and high-precision machining scenarios.

[0030] In one optional implementation, step S1 specifically includes: Will t The true three-dimensional coordinates of the tool center point at all times and three-axis true velocity components Combined to form a six-dimensional state vector : .

[0031] Specifically, the six-dimensional state vector This refers to a complete description containing the tool's position and velocity information in three-dimensional space, which can be achieved by directly acquiring sensor data or by generating it through numerical calculations. Among these, the true three-dimensional coordinates... This can be understood as the actual spatial position of the tool at a certain moment during the machining process, its purpose being to provide the geometric basis required for path smoothing; three-axis true velocity components It refers to the instantaneous velocity of the tool in the three coordinate axes, which can be obtained through differential calculation or real-time measurement, and its purpose is to reflect the dynamic characteristics of the tool's motion.

[0032] In detail, the above scheme solves the problem of ambiguity in state vector construction by clearly defining the specific components of the state vector, thereby ensuring the accuracy and consistency of the path smoothing process. Based on this, integrating the tool's true three-dimensional coordinates and three-axis velocity components into a unified input comprehensively reflects the tool's dynamic behavior, avoiding state estimation bias caused by missing information. Simultaneously, this scheme, combined with the path point sequence output by the policy neural network, provides a complete and consistent state foundation for Kalman filtering, effectively suppressing path jitter and discontinuities. Furthermore, the well-defined six-dimensional state vector structure makes the establishment of state prediction and observation equations more accurate, thereby improving the reliability of the Kalman filter algorithm in recursively estimating the path state.

[0033] In one optional implementation, step S2 specifically includes: The tool is obtained through a policy neural network. t The six-dimensional observation vector output at time step The observation vector Three-dimensional coordinates of the tool and triaxial velocity estimation components ; ; Among them, the triaxial velocity estimation component is achieved by...t Three-dimensional coordinates of time 3D coordinates compared to the previous time step Perform difference operations to obtain the triaxial velocity estimation components. ; The formula for calculating the triaxial velocity estimation components is as follows: .

[0034] Specifically, observation vector This refers to a six-dimensional vector composed of the tool's three-dimensional coordinates and three-axis velocity estimation components, which can be implemented using real-time acquired path point sequence data. In practical applications, the three-axis velocity estimation components refer to the values ​​of the coordinate differences between adjacent path points and the time step. dt The velocity value calculated using the ratio is intended to reduce velocity fluctuations caused by the randomness of neural network output. Furthermore, the specific implementation of the difference operation can be based on numerical calculation methods for discrete-time systems, such as forward differencing or central differencing algorithms, to ensure the accuracy of the velocity estimation.

[0035] In detail, the above scheme effectively solves the noise problem in velocity estimation in the observation vector by introducing differential operations to generate velocity components. First, the three-dimensional coordinate data output by the policy neural network serves as the basic input, which, combined with the velocity components generated by the differential operations, constitutes the observation vector. This design leverages the continuity of the path point sequence, avoiding the high-frequency noise amplification problem that might arise from directly relying on the output speed of the neural network. Secondly, a time step is introduced into the difference operation formula. dt This ensures that velocity calculations conform to the laws of physical motion, thereby enhancing the real-time performance and accuracy of velocity estimation. Based on this, Kalman filtering can utilize high-quality observation vectors... Recursive estimation significantly improves path smoothing performance and dynamic characteristic modeling capabilities.

[0036] In summary, the above technical solutions not only optimize the generation process of observation vectors but also significantly improve the input data quality of Kalman filtering, thereby achieving efficient smoothing of the output path of the policy neural network.

[0037] In one optional implementation, step S3 specifically includes: S31. Using a linear constant velocity model, construct the state transition matrix. : ; in, For time step; S32, Based on the state transition matrix and state vector Establish the system state prediction equation: ; in, for t The state vector at any given time; for t The state vector at time -1; This is process noise; S33. Construct the observation matrix : ; S34, Based on observation matrix and observation vector Establish the observation equation: ; in, To observe noise.

[0038] In practical applications, state transition matrix This refers to a mathematical tool used to describe the changing state of a cutting tool between adjacent moments. It can be implemented using a linear constant-speed model. This model assumes that the tool moves at a constant speed over a short period, thus simplifying the state transition process and avoiding additional errors introduced by complex models. Its purpose is to improve the efficiency and accuracy of state prediction. The system state prediction equation refers to the equation derived from the state transition matrix. The mathematical expression for predicting the state vector at the previous moment can be implemented using recursive matrix operations. Its purpose is to handle uncertainties during motion and ensure the stability of state estimation in dynamic changes. The observation matrix H is a mathematical tool used to describe the relationship between the observation vector and the state vector. It can be implemented using an identity matrix to simplify the observation process, reduce computational complexity, and ensure that observation data can be directly used for state estimation. The observation equation refers to the equation derived from the observation matrix. The mathematical expression that maps the state vector to the observation space can be implemented using a linear transformation. Its purpose is to handle measurement errors and improve the accuracy of state estimation.

[0039] Specifically, the above technical solution addresses the prediction inaccuracies caused by ambiguous model definitions during path smoothing by employing a clearly defined Kalman filter framework. First, a linear constant velocity model is used to construct the state transition matrix. This model assumes that the tool's movement speed is constant over a short period of time, thus simplifying the state transition process and making the prediction equations more operable and efficient. Secondly, based on the state transition matrix... and state vector Establish system state prediction equations Including process noise This effectively handles uncertainties during motion, improves the robustness of predictions, and ensures the stability of state estimates during dynamic changes. Furthermore, the observation matrix H is constructed as an identity matrix. , representing the observation vector Directly corresponding to the state vector This simplifies the observation process, avoids complex transformation calculations, ensures that the observation data can be directly used for state estimation, and reduces processing latency. Finally, based on the observation matrix H and the observation vector... Establish observation equations Introducing observation noise It can handle measurement errors, enabling Kalman filtering to more accurately fuse observation data during the update phase and improve the accuracy of state estimation. These features work together to ensure the accuracy and smoothness of path state estimation.

[0040] The above technical solution successfully solves the problem of inaccurate or unstable state prediction caused by the lack of specific model definition and equation construction methods in the path smoothing process, and significantly improves the path smoothing effect.

[0041] In one optional implementation, step S4 specifically includes: S41. Prediction Phase: State prediction: based on t State estimate updated at time -1 and state transition matrix ,calculate t Predicted state estimate at time 1 ; ; Covariance prediction: Synchronous update t Time prediction error covariance matrix Incorporation process noise characteristics: ; in, For process noise The covariance matrix; for t The error covariance matrix updated at time -1; for t The prediction error covariance matrix at time t; S42, Kalman gain calculation: based on t Prediction error covariance matrix at time step Observation matrix Observe the noise characteristics and calculate the Kalman gain; ; in, To observe noise The covariance matrix.

[0042] S43, Update Phase: State update: based on Kalman gain and fused t Observation vector at time Compared with the predicted state estimate Obtain the smoothed state estimate. : ; Covariance Update: Adjustment t The prediction error covariance matrix at time step 1 is used for the next iteration: ; in, for t The error covariance matrix updated at each time step; S44. The smoothed state estimate Extracting the three-dimensional coordinate components as a smoothed path point sequence for CNC machine tool trajectory control; S45. Repeat steps S41 to S44 to perform real-time smoothing of the path point sequence output by the policy neural network.

[0043] Specifically, the prediction phase refers to the process of predicting the tool state using a system model. This can be implemented using a linear constant speed model to ensure the accuracy of the state prediction. Covariance prediction considers the uncertainties in the system dynamics during the prediction process. This can be achieved by updating the prediction error covariance matrix and incorporating process noise characteristics, aiming to improve the robustness of the prediction. Kalman gain calculation optimizes the weight of observation data in the state update. This can be achieved by calculating the Kalman gain based on the prediction error covariance, the observation matrix, and the observation noise characteristics, aiming to enable the algorithm to adaptively fuse prediction and observation information and reduce the impact of noise. The update phase refers to fusing the observation vector and the predicted state estimate through Kalman gain. This can be achieved through state updates and covariance updates, aiming to ensure the continuity and stability of the recursive estimation.

[0044] Specifically, this scheme achieves real-time smoothing of the output path point sequence of the policy neural network by explicitly defining the execution flow of the Kalman filter algorithm. First, in the prediction phase, the state transition matrix is ​​used... The state estimate at the previous time step Calculate the predicted state estimate at the current time. Simultaneously update the prediction error covariance matrix. The influence of process noise is taken into account. This process ensures the accuracy of state prediction and improves the adaptability to system dynamic uncertainties. Next, in the Kalman gain calculation stage, based on the prediction error covariance matrix... Based on the observation matrix H and the observation noise characteristics, the optimal Kalman gain is calculated. This optimizes the weighting of observation data in state updates, effectively reducing noise interference with the smoothing results. Subsequently, in the update phase, the observation vectors are fused using Kalman gain. Compared with the predicted state estimate Obtain the smoothed state estimate. And update the error covariance matrix simultaneously. This prepares the system for the next iteration. Finally, the smoothed state estimates are extracted. The three-dimensional coordinate components are directly output as smooth path points for machine tool control. By repeatedly executing the entire process, real-time smoothing of the entire path point sequence is achieved. This complete process not only solves the problems of path jitter and speed discontinuity, but also significantly improves the path smoothing effect and machining quality.

[0045] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for smoothing the output path of a policy neural network based on Kalman filtering, characterized in that, Includes the following steps: S1. Construct a state vector containing the tool's spatial position and velocity; S2. Generate the corresponding observation vector based on the path point sequence output by the policy neural network; S3. Based on the state vector and observation vector, establish the system state prediction equation and observation equation; S4. Based on the system state prediction equation and observation equation, the path state is recursively estimated using the Kalman filter algorithm to achieve smoothing of the tool path.

2. The method according to claim 1, characterized in that, S1 specifically includes: Will t The true three-dimensional coordinates of the tool center point at all times and three-axis true velocity components Combined to form a six-dimensional state vector : 。 3. The method according to claim 2, characterized in that, S2 specifically includes: The tool is obtained through a policy neural network. t The six-dimensional observation vector output at time step The observation vector Three-dimensional coordinates of the tool and triaxial velocity estimation components ; ； Among them, the triaxial velocity estimation component is achieved by... t Three-dimensional coordinates of time 3D coordinates compared to the previous time step Perform difference operations to obtain the triaxial velocity estimation components. ; The calculation formula for the triaxial velocity estimation components is as follows: 。 4. The method according to claim 3, characterized in that, S3 specifically includes: S31. Using a linear constant velocity model, construct the state transition matrix. : ； in, For time step; S32, Based on the state transition matrix and state vector Establish the system state prediction equation: ； in, for t The state vector at any given time; for t The state vector at time -1; This is process noise; S33. Construct the observation matrix : ； S34, Based on observation matrix and observation vector Establish the observation equation: ； in, To observe noise.

5. The method according to claim 4, characterized in that, S4 specifically includes: S41. Prediction Phase: State prediction: based on t State estimate updated at time -1 and state transition matrix ,calculate t Predicted state estimate at time 1 ; ； Covariance prediction: Synchronous update t Time prediction error covariance matrix Incorporation process noise characteristics: ； in, For process noise The covariance matrix; for t The error covariance matrix updated at time -1; for t The prediction error covariance matrix at time t; S42, Kalman gain calculation: based on t Prediction error covariance matrix at time step Observation matrix Observe the noise characteristics and calculate the Kalman gain; ； in, To observe noise The covariance matrix; S43, Update Phase: State update: based on Kalman gain and fused t Observation vector at time Compared with the predicted state estimate Obtain the smoothed state estimate. : ； Covariance Update: Adjustment t The prediction error covariance matrix at time step 1 is used for the next iteration: ； in, for t The error covariance matrix updated at each time step; S44. The smoothed state estimate Extracting the three-dimensional coordinate components as a smoothed path point sequence for CNC machine tool trajectory control; S45. Repeat steps S41 to S44 to perform real-time smoothing of the path point sequence output by the policy neural network.

Images