A reinforcement learning-based online chatter suppression method for robot polishing

Abstract

The present application relates to an online chatter suppression method for robot polishing based on reinforcement learning, and relates to the technical field of robot vibration control and regulation. The method comprises: starting from the vibration signal and force error signal in the robot polishing process, constructing a highly recognizable multi-source feature vector through discrete wavelet transform, energy entropy calculation and statistical feature extraction, and after normalization, using a convolutional neural network and a multi-head attention module to perform feature fusion and dimensionality reduction to obtain a low-dimensional feature representation, and then constructing a timing-enhanced strategy model through a DDPG framework embedded with LSTM. In the strategy model, the Actor network outputs impedance parameters based on the timing state, and the Critic network achieves stable update of the strategy by minimizing the loss function and combining entropy regularization, thereby regulating the robot end effector. The present application realizes real-time suppression and efficient control of complex chatter mechanisms in the process of robot polishing thin-walled parts, significantly improving the system's adaptability and control accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of robot vibration control and regulation, and in particular to a method for suppressing online chatter during robot polishing based on reinforcement learning. Background Art

[0002] In related technologies, thin-walled structures are widely used in aerospace, automotive, electronic communications, and marine engineering due to their superior properties, such as light weight and high specific strength. However, thin-walled structures are often bulky, complex, have uneven wall thickness, and exhibit weak rigidity. These characteristics make them prone to vibration and deformation during machining, leading to reduced surface quality and large machining errors, which directly affect final performance.

[0003] Force-controlled robotic grinding technology has become a crucial component of advanced manufacturing due to its ability to adapt to complex surfaces, irregular shapes, and achieve high-quality machining. For example, robotic grinding of thin-walled structures is crucial. However, chatter often occurs during robotic grinding due to the dynamic contact between the tool and the workpiece and fluctuations in machining parameters. Chatter is an unstable, self-excited vibration that can lead to reduced surface quality, increased tool wear, and increased energy consumption, severely impacting machining efficiency and finished product quality. Therefore, robotic control strategies must monitor and suppress vibrations during the machining process.

[0004] Existing methods for achieving vibration suppression during robotic grinding and polishing processes are relatively simple and suffer from various drawbacks. For example, a Chinese patent (patent number CN20201155164.5) proposes a contact vibration suppression method for robotic grinding and polishing. This prior art solution primarily uses a six-dimensional force sensor to collect the normal contact force of the workpiece in real time, performs signal decomposition on this information, calculates the permutation entropy, and then adjusts the robot based on the threshold of the indicator to achieve vibration suppression. However, the robotic grinding process is subject to complex working conditions and environmental changes. Existing technical solutions rely on threshold settings, cannot adapt to changing working conditions, lack adaptability, and have insufficient control accuracy. Summary of the Invention

[0005] This application provides an online chatter suppression method for robot polishing based on reinforcement learning. Starting from the vibration signal and force error signal in the robot polishing process, a model output suppression strategy is constructed to control the robot end effector, realizing real-time suppression and efficient control of complex chatter mechanisms in the process of robot polishing thin-walled parts, significantly improving the system's adaptability and control accuracy, and solving the problems of insufficient adaptability and insufficient control accuracy in the existing technology.

[0006] In a first aspect, the present application provides a method for suppressing online chatter in robot polishing based on reinforcement learning, comprising:

[0007] Collecting discrete vibration signals during the robot grinding process, and determining a force error signal during the grinding process, wherein the force error signal is used to characterize a force control deviation during the grinding process;

[0008] The discrete vibration signal and the force error signal are treated as multi-source signals for preprocessing and normalized feature extraction and fusion to obtain a normalized feature vector with high recognition;

[0009] A convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation.

[0010] Temporal encoding is performed on the low-dimensional feature representation to obtain a temporal state, and a temporal enhanced policy model is constructed through a deep reinforcement learning framework embedded with LSTM, wherein the policy model includes an Actor network and a Critic network of a deep deterministic policy gradient algorithm;

[0011] Using the timing state as input to the Actor network to obtain an output impedance parameter, and using the timing state and the impedance parameter as input to the Critic network to obtain an output quality evaluation value;

[0012] Based on the quality evaluation value, the robot control parameters are updated by using a minimum loss function combined with entropy regularization optimization, and the robot control parameters are used as a chatter suppression strategy to regulate the robot end effector;

[0013] The robot control parameters include updated impedance parameters.

[0014] Optionally, collecting discrete vibration signals during the robot grinding process and determining a force error signal during the grinding process include:

[0015] The discrete vibration signal collected during the robot grinding process is defined as , , and define the force error signal during robot grinding as ;

[0016] in, Indicates time, It is the original discrete vibration signal, which has rich time-frequency information and nonlinear characteristics. The force expected during the robot grinding process, is the actual measured force.

[0017] Optionally, the discrete vibration signal and the force error signal are treated as multi-source signals and subjected to preprocessing and normalized feature extraction fusion to obtain a normalized feature vector with high recognition, including:

[0018] performing multi-layer discrete wavelet decomposition and energy entropy quantification on the discrete vibration signal based on a selected wavelet basis function to obtain vibration signal features characterizing a flutter evolution trend; and retaining original information of the force error signal and statistically analyzing the features of the force error signal through variance and / or mean difference to obtain force error signal features.

[0019] The vibration signal feature and the force error signal feature are concatenated, and scale differences are eliminated through normalization to obtain a normalized feature vector.

[0020] Optionally, performing multi-layer discrete wavelet decomposition and energy entropy quantification on the discrete vibration signal based on a selected wavelet basis function to obtain vibration signal features characterizing a flutter evolution trend, and retaining original information of the force error signal and statistically analyzing the features of the force error signal through variance and / or mean difference to obtain force error signal features, including:

[0021] according to , combined with the preset decomposition level, the selected wavelet basis function is used to decompose the original vibration signal Perform discrete wavelet decomposition layer by layer to obtain the low-frequency approximate coefficients and high-frequency detail coefficients after each layer of decomposition in the wavelet transform;

[0022] Combine the low-frequency approximation coefficients and high-frequency detail coefficients of each layer to obtain the final decomposition ;

[0023] For the final decomposition ,according to ,right Each subband Calculation Energy ;

[0024] Energy calculated based on each sub-band ,according to , calculate the energy proportion of each sub-band, and according to Define wavelet energy entropy ,Wavelet energy entropy is used to quantify the complexity and nonlinear characteristics of vibration signals, as well as an important feature to characterize the evolution trend of chatter;

[0025] Determine the vibration signal characteristics based on the wavelet energy entropy and the energy calculated for each sub-band;

[0026] Retention force error signal Original information, and based on Calculate statistical characteristics to obtain force error signal characteristics;

[0027] in, is the decomposition layer, The low-frequency coefficients obtained by layer decomposition are , the high frequency coefficient is , As the final decomposition in wavelet decomposition, it represents the In the sub-band wavelet coefficients, is the total number of subbands, For the sampling point.

[0028] Optionally, the vibration signal feature and the force error signal feature are concatenated, and scale differences are eliminated by normalization to obtain a normalized feature vector, including:

[0029] The vibration signal features extracted from discrete vibration signals and , and the force error signal characteristics extracted from the force error signal and Perform feature cascade and according to Perform normalization processing to eliminate scale differences and obtain normalized feature vectors;

[0030] in, is the normalized eigenvector.

[0031] Optionally, a convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation, including:

[0032] The normalized feature vector is input into the convolution layer, and the output feature map is obtained after a series of convolution and pooling operations. ;

[0033] Feature map output by the convolutional layer As a benchmark, a multi-head attention module is used to perform weighted fusion on the normalized feature vector. Feature Map Perform linear transformation to obtain query ,key and value ;

[0034] By query ,key and value For input, according to , the attention output of each head is calculated by scaling the dot product attention;

[0035] according to , concatenate the attention outputs of each head and obtain the final multi-head attention output through linear transformation;

[0036] according to , the multi-head attention output is further reduced in dimension through a fully connected layer or a pooling layer to obtain a low-dimensional feature representation;

[0037] in, are all learnable weight matrices, is the dimension of the key vector, is the number of heads, Represents the attention output of the corresponding head.

[0038] Optionally, temporal encoding is performed on the low-dimensional feature representation to obtain a temporal state, and a temporal enhancement strategy model is constructed through a deep reinforcement learning framework embedded with LSTM, including:

[0039] Embed the LSTM network in the reinforcement learning framework, perform temporal encoding on the preprocessed feature sequence, and , generate a time series state containing historical dynamic information ;

[0040] Construct an Actor-Critic framework based on the deep deterministic policy gradient algorithm to obtain a timing-enhanced policy model.

[0041] Optionally, using the timing state as input to an Actor network to obtain an output impedance parameter, and using the timing state and the impedance parameter as input to a Critic network to obtain an output quality evaluation value, including:

[0042] In the Actor network, the state is sequential For input, according to Time limit status Perform full connection layer and LSTM layer processing to obtain output action , as the impedance parameter for regulating the robot end;

[0043] In time series state and output actions As the input of the Critic network, according to Output corresponding to value , used as a quality evaluation value to evaluate the quality of the current chatter suppression strategy;

[0044] in, is the stiffness in the impedance parameter, is the damping in the impedance parameter, Represents the parameters of the Actor network, are the parameters of the Critic network.

[0045] Optionally, based on the quality evaluation value, the robot control parameters are updated by a minimum loss function combined with entropy regularization optimization, and the robot control parameters are used as a chatter suppression strategy to regulate the robot end effector, including:

[0046] When training the policy model, an entropy regularization term is introduced into the loss function of the Critic network. ;

[0047] Add entropy regularization optimization to the loss function of the Critic network Dynamically adjust the exploration weight to maximize the output of the Critic network value;

[0048] Based on the maximum value, use Update the Actor network and obtain updated robot control parameters through the updated Actor network and Critic network;

[0049] in, is the discount factor, is the target Critic network, The output of the target Actor network is used to process time series data, random exploration efficiency, and training convergence speed in the reinforcement learning process. Perform weighted sampling to enhance the model.

[0050] In summary, the embodiment of the present application starts from the vibration signal and force error signal in the robot grinding process, and constructs a highly recognizable multi-source feature vector through discrete wavelet transform, energy entropy calculation and statistical feature extraction. After normalization, the convolutional neural network and the multi-head attention module are used to perform feature fusion and dimensionality reduction to obtain a low-dimensional feature representation. Then, a timing-enhanced strategy model is constructed through the DDPG framework embedded with LSTM. In the strategy model, the Actor network outputs impedance parameters based on the timing state, and the Critic network achieves stable update of the strategy by minimizing the loss function and combining entropy regularization, thereby regulating the robot end effector. This embodiment realizes real-time suppression and efficient control of complex vibration mechanisms in the process of robot grinding thin-walled parts, significantly improves the adaptability and control accuracy of the system, and solves the problems of insufficient adaptability and insufficient control accuracy in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 A flowchart of a method for suppressing online chatter in robot polishing based on reinforcement learning provided in an embodiment of the present application;

[0054] Figure 2 This is a flowchart of a method for suppressing online chatter during robot polishing based on reinforcement learning, provided in an optional embodiment of the present application;

[0055] Figure 3 This is a flowchart of online chatter suppression for robot polishing based on reinforcement learning, provided as an optional example of this application. DETAILED DESCRIPTION

[0056] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0057] To facilitate understanding of the embodiments of the present application, further explanation will be given below in conjunction with the drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present application.

[0058] Figure 1 The flowchart of a method for suppressing online chattering of robot polishing based on reinforcement learning is provided in an embodiment of the present application. Figure 1 As shown, the reinforcement learning-based online chatter suppression method for robot polishing provided in the embodiment of the present application may specifically include the following steps:

[0059] Step 110 : collecting discrete vibration signals during the robot grinding process, and determining a force error signal during the grinding process.

[0060] The force error signal is used to characterize the force control deviation during the grinding process.

[0061] This embodiment mainly collects discrete vibration signals and force error signals during the robot grinding process, where the discrete vibration signal is also called the original discrete vibration signal, which has rich time-frequency information and nonlinear characteristics; the force error signal is mainly determined based on the force collected during the robot grinding process. The force error signal directly reflects the force control deviation during the grinding process and retains real-time working condition information.

[0062] In step 120 , the discrete vibration signal and the force error signal are treated as multi-source signals and subjected to preprocessing and normalized feature extraction fusion to obtain a normalized feature vector with high recognition.

[0063] In related technologies, since discrete vibration signals and force error signals are multi-source signals and have heterogeneity problems, the high-dimensional features extracted from the two signals will also have feature redundancy problems.

[0064] In order to solve the heterogeneity problem of multi-source signals and the high-dimensional feature redundancy problem, this embodiment extracts and processes features of discrete vibration signals and force error signals respectively, and then performs feature fusion. Specifically, this embodiment preprocesses the discrete vibration signal, including discrete wavelet transform and energy entropy calculation, extracts the features of the discrete vibration signal to quantify the complexity and nonlinear characteristics of the discrete vibration signal, and performs statistical characterization on the force error signal. The features extracted from the discrete vibration signal and the statistical features of the force error signal are then cascaded and fused to obtain a normalized feature vector, which provides a unified and highly recognizable input for the subsequent feature compression and enhancement model.

[0065] In step 130, a convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation.

[0066] In this embodiment, in order to further reduce data redundancy and improve the ability to capture key features, this embodiment uses a convolutional neural network (CNN) and a multi-head attention module to process the normalized feature vector, including analyzing the feature map of the normalized feature vector and performing weighted fusion on the feature map. Finally, the normalized feature vector is dimensionality reduced and compressed to obtain a low-dimensional feature representation. The low-dimensional feature serves as an efficient input for the subsequent reinforcement learning model while retaining the features / characteristics of the two signals.

[0067] Step 140: Temporal encoding is performed on the low-dimensional feature representation to obtain a temporal state, and a temporal enhanced strategy model is constructed through a deep reinforcement learning framework embedded with LSTM.

[0068] The policy model includes an Actor network and a Critic network of a deep deterministic policy gradient algorithm.

[0069] Step 150: Using the timing state as input to the Actor network to obtain an output impedance parameter, and using the timing state and the impedance parameter as input to the Critic network to obtain an output quality evaluation value.

[0070] Step 160 : Based on the quality evaluation value, the robot control parameters are updated by using a minimum loss function combined with entropy regularization optimization, and the robot control parameters are used as a vibration suppression strategy to regulate the robot end effector.

[0071] The robot control parameters include updated impedance parameters.

[0072] A unified description of steps 140 to 160 is provided:

[0073] In specific implementations, to address the strong temporal dependencies of chatter systems, this embodiment embeds an LSTM (Long Short-Term Memory) network within a deep reinforcement learning framework (such as Deep Deterministic Policy Gradient (DDPG)). This architecture constructs a temporally enhanced policy model, which incorporates an actor-critic framework based on the DDPG algorithm (consisting of an actor network and a critic network). The model inputs temporal states obtained by temporally encoding low-dimensional feature representations. After processing the temporal states by the actor network, actions (represented as impedance parameters) are output. These actions are then fed into the critic network to evaluate the quality of the current policy. A designed loss function is then used for optimization exploration, updating the actor and critic networks. The policy model then outputs updated robot control parameters, which serve as the chatter suppression strategy and primarily consist of updated impedance parameters. The updated impedance parameters are then used to regulate the robot's end effector.

[0074] As can be seen, the embodiment of the present application starts with the vibration signal and force error signal during the robot grinding process, and constructs a highly recognizable multi-source feature vector through discrete wavelet transform, energy entropy calculation, and statistical feature extraction. After normalization, a convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction to obtain a low-dimensional feature representation. Then, a time-enhanced strategy model is constructed through the DDPG framework embedded with LSTM. In the strategy model, the Actor network outputs impedance parameters based on the time-series state, and the Critic network achieves stable strategy updates by minimizing the loss function and combining entropy regularization, thereby regulating the robot's end effector. As can be seen, this embodiment achieves real-time suppression and efficient control of the complex chatter mechanism during the robot grinding of thin-walled parts, significantly improving the system's adaptability and control accuracy. While solving the problems of insufficient adaptability and insufficient control accuracy in the existing technology, it can also solve key problems such as the heterogeneity of multi-source data, time-series correlation, and insufficient adaptability of traditional methods.

[0075] Reference Figure 2 , shows a schematic flow chart of a method for suppressing online chatter during robot polishing based on reinforcement learning, provided in an optional embodiment of the present application. The method may specifically include the following steps:

[0076] Step 210 : collecting discrete vibration signals during the robot grinding process, and determining a force error signal during the grinding process, wherein the force error signal is used to characterize a force control deviation during the grinding process.

[0077] In the specific implementation, refer to Figure 3 As shown, to quantitatively capture the discrete vibration signal and force error vibration signal during the robotic grinding process, this embodiment defines the input signal using a formula. Specifically, the discrete vibration signal is primarily determined based on time and the original vibration signal, while the force error signal is primarily determined based on the desired force and the actual measured force during the robotic grinding process.

[0078] In an optional embodiment, collecting discrete vibration signals during the robot grinding process and determining the force error signal during the grinding process may specifically include: defining the discrete vibration signals collected during the robot grinding process as , , and define the force error signal during robot grinding as ;in, Indicates time, It is the original discrete vibration signal, which has rich time-frequency information and nonlinear characteristics. The force expected during the robot grinding process, is the actual measured force.

[0079] This embodiment defines discrete vibration signals and force error signals to achieve the acquisition of two signals, so that the discrete vibration signal can have rich time-frequency information and nonlinear characteristics; the force error signal can also retain real-time working condition information. Through the force error signal, this embodiment intuitively determines the force control deviation in the grinding process and optimizes the subsequent grinding control of the robot.

[0080] In step 220 , the discrete vibration signal is subjected to multi-layer discrete wavelet decomposition and energy entropy quantification based on the selected wavelet basis function to obtain vibration signal features that characterize the flutter evolution trend. Furthermore, the original information of the force error signal is retained, and the features of the force error signal are statistically analyzed by variance and / or mean difference to obtain force error signal features.

[0081] In the specific implementation, the vibration signal features are features extracted from the discrete vibration signal, mainly including but not limited to: sub-band energy corresponding to each wavelet coefficient, wavelet energy entropy and other time-frequency statistics; the force error signal features are features extracted from the force error signal, mainly including but not limited to: variance features and mean difference features.

[0082] Reference Figure 3 As shown, for feature extraction of discrete vibration signals, this embodiment uses Daubechies wavelet basis functions to perform multi-layer discrete wavelet decomposition on the discrete vibration signal, representing the signal as two coefficients: low-frequency approximation coefficients and high-frequency detail coefficients. These two coefficients are then represented in subbands, and the energy of each subband is calculated to determine the wavelet energy entropy. This entropy is then used to quantify the important features of the vibration signal that characterize the flutter evolution trend, namely, the discrete vibration signal features.

[0083] Reference Figure 3 As shown, for feature extraction of the force error signal, this embodiment utilizes the original information of the force error signal to calculate statistical features, such as the mean and / or variance, to extract the signal features of the force error signal. The features corresponding to the two signals are then concatenated and normalized to eliminate scale differences between the different signal features, resulting in a normalized vector feature.

[0084] Optionally, the above-mentioned multi-layer discrete wavelet decomposition and energy entropy quantification of the discrete vibration signal based on the selected wavelet basis function to obtain the vibration signal characteristics that characterize the flutter evolution trend, and retaining the original information of the force error signal and obtaining the force error signal characteristics by statistically analyzing the characteristics of the force error signal through variance and / or mean difference, may include: according to , combined with the preset decomposition level, the selected wavelet basis function is used to decompose the original vibration signal Perform discrete wavelet decomposition layer by layer to obtain the low-frequency approximate coefficients and high-frequency detail coefficients after decomposition of each layer in the wavelet transform; combine the low-frequency approximate coefficients and high-frequency detail coefficients of each layer to obtain the final decomposition ; For the final decomposition ,according to ,right Each subband Calculation Energy ; Based on the energy calculated for each subband ,according to , calculate the energy proportion of each sub-band, and according to Define wavelet energy entropy Wavelet energy entropy is used to quantify the complexity and nonlinear characteristics of the vibration signal, as well as important features to characterize the flutter evolution trend; based on the wavelet energy entropy and the energy calculated for each sub-band, the vibration signal characteristics are determined; the force error signal is retained. Original information, and based on Calculate the statistical characteristics and obtain the force error signal characteristics; where, is the decomposition layer, The low-frequency coefficients obtained by layer decomposition are , the high frequency coefficient is , As the final decomposition in wavelet decomposition, it represents the In the sub-band wavelet coefficients, is the total number of subbands, For the sampling point.

[0085] In the specific implementation, in order to ensure that the features extracted from the discrete vibration signal can quantify the complexity and nonlinear characteristics of the vibration signal, this embodiment performs multi-layer wavelet decomposition on the wavelet transform of the discrete vibration signal to obtain the low-frequency and high-frequency coefficients of each layer as the intermediate quantity of the wavelet transform, and finally obtains the final decomposition through multi-layer decomposition. Then, starting from the calculation of energy and energy entropy, the above formula is used to extract important features.

[0086] In addition, to ensure that the features extracted from the force error signal can retain the original information, this embodiment mainly uses the formula Calculate statistical features and implement feature extraction by combining mean and variance.

[0087] Step 230 : Concatenate the vibration signal feature and the force error signal feature, and eliminate scale differences by normalization to obtain a normalized feature vector.

[0088] In the specific implementation, refer to Figure 3As shown, this embodiment concatenates the features extracted from the discrete vibration signal (such as the energy corresponding to each sub-band, wavelet energy entropy, and other time-frequency statistics) with the statistical features of the force error signal (such as variance, mean difference), and performs normalization processing to eliminate scale differences to obtain a normalized feature vector. The normalized feature vector provides a unified and highly recognizable input for subsequent feature compression and reinforcement learning models.

[0089] Optionally, the above-mentioned cascading of the vibration signal feature and the force error signal feature and eliminating the scale difference by normalization to obtain a normalized feature vector may include: extracting the vibration signal feature from the discrete vibration signal and , and the force error signal characteristics extracted from the force error signal and Perform feature cascade and according to Perform normalization processing to eliminate scale differences and obtain normalized feature vectors; among them, is the normalized eigenvector.

[0090] Specifically, this embodiment uses the formula The cascade of different features is realized, the differences in scales of different features are eliminated, and the heterogeneity of multi-source signals and the redundancy of high-dimensional features are solved.

[0091] In step 240, a convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation.

[0092] In the specific implementation, this embodiment adopts the constructed convolutional neural network, combined with the multi-head attention module, with the normalized feature vector as input, and performs feature fusion and dimensionality reduction on it, so as to further reduce data redundancy and improve the ability to capture key features, and obtain a low-dimensional feature representation.

[0093] In an optional embodiment, a convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation, which may specifically include: inputting the normalized feature vector into the convolution layer, and obtaining an output feature map after a series of convolution and pooling operations. ; Feature map output by the convolutional layer As a benchmark, a multi-head attention module is used to perform weighted fusion on the normalized feature vector. Feature Map Perform linear transformation to obtain query ,key and value ; To query ,key and value For input, according to , the attention output of each head is calculated by scaling the dot product attention; according to , concatenate the attention outputs of each head, and obtain the final multi-head attention output through linear transformation; according to , the multi-head attention output is further reduced in dimension through the fully connected layer or pooling layer to obtain a low-dimensional feature representation; are all learnable weight matrices, is the dimension of the key vector, is the number of heads, Represents the attention output of the corresponding head.

[0094] For example, refer to Figure 3 , taking the normalized feature vector as input, convolution and pooling operations are performed in the convolutional layer of the convolutional neural network. In order to improve the ability to capture key features, the formula is mainly used. Realize the extraction of feature maps.

[0095] The feature map is then used as the input of the multi-head attention module, and weighted fusion is performed within the multi-head attention module. The weighted fusion process includes but is not limited to: linear transformation. When performing linear transformation, the formula is used in sequence: 、 as well as Finally, use the formula By realizing feature dimensionality reduction and using this formula for feature dimensionality reduction, the time-frequency energy distribution characteristics of the vibration signal and the statistical information of the force signal can be retained, providing efficient input for the subsequent reinforcement learning model.

[0096] Step 250: Temporal encoding is performed on the low-dimensional feature representation to obtain a temporal state, and a temporal enhanced strategy model is constructed through a deep reinforcement learning framework embedded with LSTM.

[0097] The policy model includes an Actor network and a Critic network of a deep deterministic policy gradient algorithm.

[0098] In the specific implementation, considering the strong temporal dependencies of the chatter system, this embodiment embeds an LSTM network within a reinforcement learning framework to perform temporal encoding on the preprocessed feature sequence, generating a state representation containing historical dynamic information. Based on this, an actor-critic framework based on the Deep Deterministic Policy Gradient (DDPG) algorithm is then constructed.

[0099] Optionally, the temporal encoding of the low-dimensional feature representation is performed to obtain a temporal state, and a temporal enhancement strategy model is constructed by embedding a deep reinforcement learning framework of LSTM, which may specifically include: embedding an LSTM network in the reinforcement learning framework, temporal encoding of the preprocessed feature sequence, and according to , generate a time series state containing historical dynamic information ; Build an Actor-Critic framework based on the deep deterministic policy gradient algorithm to obtain a time-enhanced policy model.

[0100] Therefore, this embodiment introduces the LSTM layer into the DDPG reinforcement learning framework, performs time series encoding on the fused low-dimensional feature sequence, generates a state containing historical dynamic information, and improves the time series perception ability of the policy model, so that the policy decision can not only reflect the current state but also integrate historical information, thereby overcoming the short-sightedness problem of traditional methods.

[0101] In the specific implementation, the timing state obtained according to the formula It not only reflects the current flutter state but also integrates historical information, thereby improving the model's timing perception ability.

[0102] Step 260: Use the timing state as input to the Actor network to obtain an output impedance parameter, and use the timing state and the impedance parameter as input to the Critic network to obtain an output quality evaluation value.

[0103] In an optional embodiment, the timing state is used as the input of the Actor network to obtain the output impedance parameter, and the timing state and the impedance parameter are used as the input of the Critic network to obtain the output quality evaluation value, which may include: using the timing state in the Actor network to obtain the output quality evaluation value. For input, according to Time limit status Perform full connection layer and LSTM layer processing to obtain output action , as the impedance parameter for regulating the robot end; and output actions As the input of the Critic network, according to Output corresponding to value , is used as the quality evaluation value to evaluate the quality of the current chatter suppression strategy; where, is the stiffness in the impedance parameter, is the damping in the impedance parameter, Represents the parameters of the Actor network, are the parameters of the Critic network.

[0104] In this embodiment, the impedance parameters of the robot end are mainly controlled by stiffness. and damping , timing status As the input of the Actor network, it outputs actions after passing through several fully connected layers and LSTM layers Using this as the input of the Critic network, we use the formula Assess the quality of your current strategy.

[0105] Step 270 : Based on the quality evaluation value, the robot control parameters are updated by using a minimum loss function combined with entropy regularization optimization, and the robot control parameters are used as a vibration suppression strategy to regulate the robot end effector.

[0106] The robot control parameters include updated impedance parameters.

[0107] In an optional embodiment, based on the quality evaluation value, the robot control parameters are updated by minimizing the loss function and combining entropy regularization optimization, and the robot control parameters are used as a vibration suppression strategy to regulate the robot end effector. Specifically, the method may include: when training the strategy model, introducing an entropy regularization term in the loss function of the critic network. ;

[0108] Add entropy regularization optimization to the loss function of the Critic network , dynamically adjust the exploration weight to maximize the output of the Critic network value; based on the maximum value obtained value, use Update the Actor network and obtain the updated robot control parameters through the updated Actor network and Critic network; is the discount factor, is the target Critic network, The output of the target Actor network is used to process time series data, random exploration efficiency, and training convergence speed in the reinforcement learning process. Perform weighted sampling to enhance the model.

[0109] In this embodiment, the entropy regularization term As the benchmark parameter, entropy regularization optimization is introduced into the loss function of the Critic network. , thereby maximizing the output of the Critic network value.

[0110] Existing robotic grinding control methods for thin-walled parts rely on passive mechanical structures to absorb the energy surge generated by chatter vibrations. These mechanisms also have limited vibration suppression bandwidth and poor applicability. Furthermore, existing robotic grinding control methods often rely on static threshold settings, adjusting machining parameters when the chatter state reaches a fixed threshold. This lacks the ability to dynamically respond to chatter vibrations.

[0111] To address these technical issues, this embodiment innovatively introduces a dynamic control strategy based on real-time vibration and force signal analysis, primarily proposing a dynamic impedance parameter optimization framework based on a deep deterministic policy gradient algorithm. Specifically, this framework achieves real-time chatter suppression and efficient, stable control of the polishing process through multi-source signal fusion, time-enhanced strategy modeling, a stabilization training mechanism, and redundant information filtering.

[0112] In a specific implementation, a chatter suppression system can be constructed based on the reinforcement learning-based online chatter suppression method for robotic polishing provided in this embodiment. To enhance the system's adaptability, this embodiment utilizes historical data for training, automatically adjusting and optimizing its strategy. This enables the system to adapt to more diverse and complex working conditions, improving the robot's adaptability and control accuracy during complex polishing processes.

[0113] Specifically, refer to Figure 3 During the training process, in order to reduce the possibility of the algorithm falling into the local optimum, this embodiment introduces an entropy regularization term into the loss function of the Critic network. Dynamically adjust the exploration weight, increase the exploration probability when the policy entropy is lower than the threshold, and avoid local optimality. And add the entropy regularization term to the loss function of the Critic network so that the output of the Critic network The value can be maximized and the Actor network can be maximized The value is updated.

[0114] Further, refer to Figure 3 This embodiment also introduces an experience replay mechanism based on timing error priority to address the challenges of overfitting, low random exploration efficiency, and slow training convergence in reinforcement learning when processing time series data. Specifically, this mechanism measures the timing error of the experience samples, assigns priority weights to the experience data, and performs weighted sampling in the experience replay pool according to this weight. For example, using the formula This approach implements weighted sampling to optimize data utilization efficiency, enhance the model's focus on key experiences, and improve training stability and convergence speed. Furthermore, this approach can alleviate the overfitting problem to a certain extent by preventing the model from overly relying on recent experience and instead encouraging it to learn from a wider range of historical experiences, thereby improving the generalization ability of the strategy.

[0115] In summary, the embodiment of the present application uses the vibration signal and force error signal Starting from discrete wavelet transform, energy entropy calculation and statistical feature extraction, a highly recognizable multi-source feature vector is constructed. After normalization, convolutional neural network and multi-head attention module are used to perform feature fusion and dimensionality reduction to obtain low-dimensional feature representation. , and then build a time-series enhanced strategy model by embedding the DDPG framework with LSTM. The Actor network is based on the time-series state Output impedance parameters , while the Critic network achieves stable updates of the strategy by minimizing the loss function and combining it with entropy regularization. The technical solution of this embodiment does not rely on a passive energy-absorbing structure, but constructs a reinforcement learning strategy model that can accurately capture the dynamic characteristics of the flutter phenomenon and provide the robot with effective vibration suppression measures in real time, thereby realizing a set of robotic polishing systems with integrated measurement and control functions. On the one hand, this embodiment realizes the real-time suppression and efficient control of the complex flutter mechanism in the process of robot polishing thin-walled parts, significantly improving the system's adaptability and control accuracy; on the other hand, this embodiment fully solves key problems such as the heterogeneity of multi-source data, time series correlation, and the lack of adaptability of traditional methods.

[0116] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously.

[0117] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0118] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A method for suppressing online chatter in robot polishing based on reinforcement learning, characterized in that: include: Collecting discrete vibration signals during the robot grinding process, and determining a force error signal during the grinding process, wherein the force error signal is used to characterize a force control deviation during the grinding process; The discrete vibration signal and the force error signal are treated as multi-source signals for preprocessing and normalized feature extraction and fusion to obtain a normalized feature vector with high recognition; A convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation. Temporal encoding is performed on the low-dimensional feature representation to obtain a temporal state, and a temporal enhanced policy model is constructed through a deep reinforcement learning framework embedded with LSTM, wherein the policy model includes an Actor network and a Critic network of a deep deterministic policy gradient algorithm; Using the timing state as input to the Actor network to obtain an output impedance parameter, and using the timing state and the impedance parameter as input to the Critic network to obtain an output quality evaluation value; Based on the quality evaluation value, the robot control parameters are updated by using a minimum loss function combined with entropy regularization optimization, and the robot control parameters are used as a chatter suppression strategy to regulate the robot end effector; Wherein, the robot control parameters include updated impedance parameters; The discrete vibration signal and the force error signal are treated as multi-source signals and subjected to preprocessing and normalized feature extraction and fusion to obtain a normalized feature vector with high recognition, including: performing multi-layer discrete wavelet decomposition and energy entropy quantification on the discrete vibration signal based on a selected wavelet basis function to obtain vibration signal features that characterize the flutter evolution trend; and retaining the original information of the force error signal and statistically analyzing the features of the force error signal through variance and / or mean difference to obtain force error signal features; performing feature concatenation on the vibration signal features and the force error signal features, and eliminating scale differences through normalization to obtain a normalized feature vector; Based on the selected wavelet basis function, the discrete vibration signal is subjected to multi-layer discrete wavelet decomposition and energy entropy quantification to obtain vibration signal features that characterize the flutter evolution trend, and the original information of the force error signal is retained, and the features of the force error signal are statistically analyzed by variance and / or mean difference to obtain the force error signal features, including: according to , combined with the preset decomposition level, the selected wavelet basis function is used to decompose the original vibration signal Perform discrete wavelet decomposition layer by layer to obtain the low-frequency approximate coefficients and high-frequency detail coefficients after decomposition of each layer in the wavelet transform; combine the low-frequency approximate coefficients and high-frequency detail coefficients of each layer to obtain the final decomposition ; For the final decomposition ,according to ,right Each subband Calculation Energy ; Based on the energy calculated for each subband ,according to , calculate the energy proportion of each sub-band, and according to Define wavelet energy entropy Wavelet energy entropy is used to quantify the complexity and nonlinear characteristics of the vibration signal, as well as important features to characterize the flutter evolution trend; based on the wavelet energy entropy and the energy calculated for each sub-band, the vibration signal characteristics are determined; the force error signal is retained. Original information, and based on Calculate the statistical characteristics and obtain the force error signal characteristics; where, is the decomposition layer, The low-frequency coefficients obtained by layer decomposition are , the high frequency coefficient is , As the final decomposition in wavelet decomposition, it represents the In the sub-band wavelet coefficients, is the total number of subbands, For the sampling point.

2. The method according to claim 1, characterized in that Collecting discrete vibration signals during the robotic grinding process and determining force error signals during the grinding process include: The discrete vibration signal collected during the robot grinding process is defined as , and define the force error signal during robot grinding as ; in, Indicates time, It is the original discrete vibration signal, which has rich time-frequency information and nonlinear characteristics. The force expected during the robot grinding process, is the actual measured force.

3. The method according to claim 1, characterized in that The vibration signal feature and the force error signal feature are concatenated, and scale differences are eliminated by normalization to obtain a normalized feature vector, including: The vibration signal features extracted from discrete vibration signals and , and the force error signal characteristics extracted from the force error signal and Perform feature cascade and according to Perform normalization processing to eliminate scale differences and obtain normalized feature vectors; in, is the normalized eigenvector.

4. The method according to claim 3, characterized in that A convolutional neural network and a multi-head attention module are used to perform feature fusion and dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation, including: The normalized feature vector is input into the convolution layer, and the output feature map is obtained after a series of convolution and pooling operations. ; Feature map output by the convolutional layer As a benchmark, a multi-head attention module is used to perform weighted fusion on the normalized feature vector. Feature Map Perform linear transformation to obtain query ,key and value ; By query ,key and value For input, according to , the attention output of each head is calculated by scaling the dot product attention; according to , concatenate the attention outputs of each head and obtain the final multi-head attention output through linear transformation; according to , the multi-head attention output is further reduced in dimension through a fully connected layer or a pooling layer to obtain a low-dimensional feature representation; in, 、 、 are all learnable weight matrices, is the dimension of the key vector, is the number of heads, Represents the attention output of the corresponding head.

5. The method according to claim 1, wherein The low-dimensional feature representation is temporally encoded to obtain a temporal state, and a temporal enhancement strategy model is constructed through a deep reinforcement learning framework embedded with LSTM, including: Embed the LSTM network in the reinforcement learning framework, perform temporal encoding on the preprocessed feature sequence, and , generate a time series state containing historical dynamic information ; Construct an Actor-Critic framework based on the deep deterministic policy gradient algorithm to obtain a timing-enhanced policy model.

6. The method according to claim 5, characterized in that Using the time series state as input to the Actor network to obtain an output impedance parameter, and using the time series state and the impedance parameter as input to the Critic network to obtain an output quality evaluation value, including: In the Actor network, the state is sequential For input, according to Time limit status Perform full connection layer and LSTM layer processing to obtain output action , as the impedance parameter for regulating the robot end; In time series state and output actions As the input of the Critic network, according to Output corresponding to value , used as a quality evaluation value to evaluate the quality of the current chatter suppression strategy; in, is the stiffness in the impedance parameter, is the damping in the impedance parameter, Represents the parameters of the Actor network, are the parameters of the Critic network.

7. The method according to claim 6, characterized in that Based on the quality evaluation value, the robot control parameters are updated by using a minimum loss function combined with entropy regularization optimization, and the robot control parameters are used as a chatter suppression strategy to regulate the robot end effector, including: When training the policy model, an entropy regularization term is introduced into the loss function of the Critic network. ; Add entropy regularization optimization to the loss function of the Critic network Dynamically adjust the exploration weight to maximize the output of the Critic network value; Based on the maximum value, use Update the Actor network and obtain updated robot control parameters through the updated Actor network and Critic network; in, is the discount factor, is the target Critic network, The output of the target Actor network is used to process time series data, random exploration efficiency, and training convergence speed in the reinforcement learning process. Perform weighted sampling to enhance the model.

Images