A servo system parameter offline identification method, system, electronic device and computer storage medium

Abstract

This invention discloses an offline parameter identification method, system, electronic device, and computer storage medium for servo systems. The method includes: acquiring input control signals and output position feedback data of the servo system and preprocessing the data; designing the DDPG algorithm structure: determining the system model structure, defining parameter state vectors, defining the action space, constructing an Actor network, constructing a Critic network, designing a reward function, and configuring an experience replay mechanism; and executing the parameter identification training process: initialization, action output, parameter update and next parameter state generation, model prediction and reward calculation, experience storage, network update, backoff mechanism, and termination flag determination, until the training termination condition is met. The identification result of this invention is a discrete state-space model, which is simple in form, has low computational cost, and can be directly ported to embedded platforms such as MCUs, meeting the engineering application needs of real-time control, fault detection, etc., and has broad industrial application prospects.

Description

Technical Field

[0001] This invention relates to a method, system, electronic device, and computer storage medium for offline identification of servo system parameters. Background Technology

[0002] For some controlled objects, due to their complexity, it is difficult to accurately derive mathematical models through theoretical analysis. Taking position servo systems as an example, due to the influence of transmission friction, backlash, and machining accuracy, the models derived through theory are either too complex to be ported to MCUs, or too inaccurate to meet the usage requirements.

[0003] To address this issue, experimental data is typically used to confirm the system model and estimate parameters. Commonly used methods include: (1) parameter identification based on least squares and its variants; (2) parameter identification based on neural networks, including BP, RBF, Hopfield, etc.; and (3) intelligent optimization algorithms, including genetic algorithms, particle swarm optimization, and difference algorithms. Among these, the least squares method has a single evaluation function, relying solely on the sum of squared errors for evaluation. Parameter identification based on neural networks yields a neural network model, but cannot obtain specific system parameters. Under conditions where multiple parameters need to be identified, intelligent optimization algorithms are prone to getting trapped in local optima and cannot impose safety constraints on parameters. They have low information utilization, cannot summarize experience from historical and failure information, rely on random search strategies, lack intelligent guidance, and cannot adapt to changes in the needs of different stages of the optimization process. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, electronic device and computer storage medium for offline identification of servo system parameters.

[0005] The objective of this invention is achieved through the following technical solution: A first aspect of the present invention provides a method for offline identification of servo system parameters, comprising the following steps: S1: Collect the input control signals and output position feedback data of the servo system, and preprocess the data; S2: Design the DDPG algorithm structure, including: S201, Determine the system model structure, which is achieved by determining the discrete state-space model structure of the servo system: ; in This represents the action state vector at the t-th sampling time in the servo system. For location, For speed, For acceleration; This represents the action state vector at the (t+1)th sampling time in the servo system. The system input at the t-th sampling time is A, which is the state matrix, and B, which is the control matrix. All unknown elements of the state matrix A and the control matrix B are the set of parameters to be identified. S202, Define the parameter state vector: The current values ​​of all parameters in the set of parameters to be identified are formed into a vector, which is defined as the parameter state vector s, and serves as the state space of the DDPG algorithm. S203, Define the action space: Define the action dimension according to the number of parameters to be identified, and adjust one parameter for each action a; each action a includes two parts: the index of the selected parameter i and the adjustment magnitude Δ; set the boundary of action a and add exploration noise that decays with the training rounds; S204, Construct the Actor network: The input of the Actor network is the parameter state vector at the current time, i.e., the current state s, and the output of the Actor network is the action a=(i,Δ); the action a means to select the i-th parameter from the parameter state vector s and apply an adjustment amount of Δ. S205, Construct the Critic network: The input to the Critic network is the parameter state vector at the current time, i.e., the current state s, and the action a output by the Actor network; the output of the Critic network is the Q value, which is used to evaluate the expected cumulative reward that can be obtained by performing action a under the current parameter state s. S206, Design the reward function: The reward function r is calculated based on the goodness of fit R2 between the model prediction value and the actual observation value corresponding to the current state s, where the model prediction value and the actual observation value correspond to the first component of the dynamic state vector, namely the position parameter; S207, Configure the experience replay mechanism: Configure the experience replay pool to store the transition samples during the training process; each transition sample is a quintuple (s, a, r, s', done), where s' is the new parameter state vector obtained by updating the parameters after executing the action, i.e. the next parameter state, and done is the termination flag; S208: Set up a soft update mechanism for the target network: Set up target networks for both the Actor network and the Critic network, and slowly update the target network parameters using a soft update method; S3: Execute the parameter identification training process, including: S301, Initialization: Initialize the parameter state vector s to the preset parameter values; S302, Action Output: In each training step, the Actor network outputs an action a=(i,Δ) based on the current system state s; S303, Parameter update and next parameter state generation: Execute action a, parse the selected parameter index i and adjustment range Δ, update the value of the corresponding parameter according to the preset action boundary, and obtain a new parameter state vector as the next parameter state s′; S304, Model Prediction and Reward Calculation: Substitute the parameter values ​​in the updated parameter state vector s′ into the state matrix A and control matrix B in the discrete state-space model in S201, iteratively calculate the values ​​of the servo system's motion state vector x at each sampling time t, and extract the position components. The model prediction output sequence is used as the model prediction output sequence. The model prediction output sequence is compared with the actual observation sequence to calculate the goodness of fit R2, and the reward value r is calculated according to the reward function. S305, Experience Storage: Store the current transfer sample (s, a, r, s', done) into the experience replay pool; S306, Network Update: Sample batch samples from the experience replay pool, use the Critic network to calculate the current Q value and the target Q value, update the Critic network parameters; update the Actor network parameters using the policy gradient; S307, rollback mechanism and termination flag determination: If the preset conditions are triggered, the termination flag done is enabled, and the system state representation is rolled back to the historical optimal state to continue exploration; S308: Repeat steps S302 to S307 until the training termination condition is met.

[0006] Furthermore, in step S202, based on the characteristics of the state vector x, the system model is a third-order state-space model. , z represents the parameter to be identified, and dt is the discrete time. The parameters to be identified include a set of four parameters in state matrix A and three parameters in control matrix B.

[0007] Furthermore, the reward function r in step S206: r = r1 + r2; Where r1 is the basic reward, specifically the goodness-of-fit R2 of the servo system position output y. y The goodness of fit R² of its derivative y' y’ Weighted sum: ; In the formula, 'a' represents the weighting ratio; r2 represents an increase in rewards; this occurs when the base reward r1 is higher than the historical best base reward. hour: ; In the formula, k is the specific gravity; where, when When r2 is less than 0, r2 is 0. If the value is greater than 0, then the current position is the new optimal base.

[0008] Furthermore, the formula for calculating the goodness-of-fit R² is: ; In the formula, Let j be the actual position observation value at time j. Let be the model position prediction value at time j; This represents the average of all observed locations for the samples. , where n represents the total number of samples.

[0009] Furthermore, in step S203, the amplitude of the exploration noise is set to a relatively large value in the early stage of training, and decreases as the training rounds increase according to a preset attenuation coefficient.

[0010] Furthermore, in step S307, the determination condition for the termination flag is: if the reward value obtained after a preset number of consecutive attempts is lower than a preset threshold or is negative, then the rollback operation in step S307 is triggered.

[0011] A second aspect of the present invention provides an offline servo system parameter identification system, comprising: Data acquisition and preprocessing module: used to perform step S1 of the method as described in the first aspect, to acquire the input and output data of the servo system and perform filtering and differentiation processing; Algorithm configuration module: used to execute step S2 of the method as described in the first aspect, and to configure the network structure, parameter state vector dimension, action space, reward function and training hyperparameters; Training and identification module: used to perform step S3 of the method as described in the first aspect, iteratively optimize the parameter state vector, and output the optimal parameter set; The result output module is used to map the parameter values ​​in the optimal parameter state vector s obtained from training to the state matrix A and control matrix B of the discrete state space model of the servo system and output them, thus completing the offline parameter identification.

[0012] A third aspect of the present invention provides an electronic device including a memory and a processor, the memory storing computer instructions executable on the processor, the processor performing steps of the method as described in the first aspect when executing the computer instructions.

[0013] A fourth aspect of the present invention provides a computer storage medium having computer instructions stored thereon, wherein the computer instructions, when executed, perform the steps of the method as described in the first aspect.

[0014] The beneficial effects of this invention are: (1) High identification accuracy: By directly embedding the goodness-of-fit R2 into the reward function, high-precision fitting of the model's predicted output with the actual data in both position and velocity dimensions is achieved. It can reach above 0.99.

[0015] (2) Fast convergence speed: The reward structure of "basic reward + enhancement reward" enables the intelligent agent to quickly perceive parameters and adjust direction; the continuous penalty backoff mechanism avoids invalid exploration, and the convergence speed is improved by more than 60% compared with traditional intelligent optimization algorithms.

[0016] (3) Strong global search capability: The combination of motion noise attenuation design and backoff mechanism ensures both wide-area exploration in the early stage and stable convergence in the later stage, effectively avoiding the local optimum trap in the multidimensional parameter space.

[0017] (4) High parameter safety: By constraining the single-step adjustment range of parameters through action boundary constraints, compared with traditional algorithms that can only constrain the range of parameter values, this invention achieves safe constraints on the parameter change process and avoids model divergence.

[0018] (5) The model is highly deployable: the identification result is a discrete state-space model, which is simple in form and has a small amount of computation. It can be directly ported to embedded platforms such as MCU to meet the engineering application requirements such as real-time control and fault detection.

[0019] (6) Good versatility: The method of the present invention does not depend on a specific system structure and can be applied to offline parameter identification of various linear / nonlinear systems, and has broad industrial application prospects. Attached Figure Description

[0020] Figure 1 A flowchart of an offline servo system parameter identification method provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of the data collected and processed according to an exemplary embodiment of the present invention; Figure 3 A schematic diagram of the identification result provided for an exemplary embodiment of the present invention; Figure 4 A schematic diagram of an offline servo system parameter identification system provided as an exemplary embodiment of the present invention; Figure 5 A schematic diagram of an electronic device provided in an exemplary embodiment of the present invention; Figure 6 A schematic diagram of a computer storage medium provided for an exemplary embodiment of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0022] See Figure 1 As shown, Figure 1 The flowchart illustrates an exemplary embodiment of the present invention, which provides a method for offline identification of servo system parameters, including the following steps: S1: Collect the input control signals and output position feedback data of the servo system, and preprocess the data; A physical test platform was built to test the system's output under different inputs, and the inputs and outputs were processed, such as by filtering them and observing other variables through an observer. Taking a position servo system as an example, its input and position feedback data under proportional control were tested. In actual control, the derivative of the position feedback plays a crucial role; therefore, the derivative of the position feedback can be calculated, and the velocity can be estimated from the position data using Savitzky-Golay smoothing filtering.

[0023] In one specific exemplary embodiment, a position servo system experimental platform is constructed, including a permanent magnet synchronous motor, a driver, a linear encoder position sensor, and an embedded controller. A proportional control law is applied to the controller, and input control signals of different amplitudes and frequencies are acquired. and the corresponding position signal The sampling period Ts = 1ms, and a total of 5000 sampling points are collected. The collected position signals... Savitzky-Golay smoothing filters were used to remove measurement noise, and the velocity signal was estimated by numerical differentiation of the filtered position signal. Input signal Position signal Speed ​​signal , acceleration signal This constitutes a dataset for subsequent model fit calculations. The collected and processed data includes... Figure 2 As shown.

[0024] S2: Design the DDPG algorithm structure, including: S201, Determine the system model structure, which is achieved by determining the discrete state-space model structure of the servo system: ; Among them This represents the action state vector at the t-th sampling time in the servo system. For location, For speed, For acceleration; This represents the action state vector at the (t+1)th sampling time in the servo system. The system input at the t-th sampling time is A, which is the state matrix, and B, which is the control matrix. All unknown elements of the state matrix A and the control matrix B are the set of parameters to be identified.

[0025] Specifically, based on theory or experience, it is determined that system constraints or conditions need to be identified. Due to the limited computing power of MCUs, the system is usually considered to be a linear model, i.e., the discrete state-space model mentioned above.

[0026] In a preferred exemplary embodiment, based on the characteristics of the position servo system, the ideal position servo system is a second-order system, but due to factors such as nonlinearity, it is usually treated as a third-order system. Based on the characteristics of state x, the first row of data can be confirmed. Since position is independent of velocity and acceleration, the state equation of A can be deduced as follows: ; z represents the parameter to be identified, and dt is the discrete time (corresponding to the sampling period Ts); all parameters in B cannot be determined, therefore At this point, the discrete state-space model structure can be replaced with: .

[0027] S202, Define the parameter state vector: The current values ​​of all parameters in the set of parameters to be identified are combined into a vector, which is defined as the parameter state vector s, and serves as the state space of the DDPG algorithm.

[0028] In a preferred exemplary embodiment, the parameters to be identified include a set of four parameters in state matrix A and three parameters in control matrix B. That is: This vector serves as the state space for the DDPG algorithm. During initialization, the parameters and state vector are set empirically. .

[0029] S203, Define the action space: Define the action dimension according to the number of parameters to be identified, and adjust one parameter for each action a; each action a includes two parts: the index of the selected parameter i and the adjustment magnitude Δ; set the boundary of action a and add exploration noise that decays with the training rounds.

[0030] Specifically, in this step, the action space is designed as a 7-dimensional continuous action, and each action a=(i,Δ) consists of two parts: the index of the selected parameter i∈{0,1,...,7}, which can be discretely selected using the Gumbel-Softmax technique; and the adjustment magnitude Δ∈[−1,1], which is preferably obtained through the Tanh activation function of the Actor network output layer.

[0031] Based on the system's undetermined parameters, the action dimension is confirmed. To ensure training doesn't crash and efficiently approximates the optimal solution, the action boundary needs to be defined. The action boundary α is the percentage of parameter change. To avoid getting trapped in local optima, noise needs to be added to the action. Initially, a large noise level can be set, gradually decreasing as training time increases. The final output is the superposition of action and noise. The boundary coefficient for action 'a' is set to 0.08. Noise is explored, with an initial amplitude of 0.3 and a decay coefficient of 0.99, decaying once per training epoch. Therefore, the action output in the x-th training epoch is: For the final output, Let A and B be the parameters corresponding to the selected action among all the parameters needed to identify them during x training iterations. Let be the output amplitude of the Actor network during the x-th training iteration, and let i indicate that the action chosen this time is to change the i-th parameter.

[0032] S204, Construct the Actor network: The input of the Actor network is the parameter state vector at the current time, i.e., the current state s, and the output of the Actor network is the action a=(i,Δ); the action a means to select the i-th parameter from the parameter state vector s and apply an adjustment amount of Δ. DDPG is a reinforcement learning algorithm designed for continuous action spaces, solving the problem that traditional Q-learning (such as DQN) can only handle discrete actions. Its core framework consists of a dual-network (Actor-Critic) + experience replay + soft update, which together ensure stable and efficient training. The Actor-Critic network is mainly used to confirm the current state and action, and to score based on the current state and action; experience replay is used to store historical "state-action-reward-next state" data, which is then shuffled and used for batch training to break sample correlation and avoid training oscillations; soft update is used to set a "target network" for each Actor / Critic, slowly updating the target network parameters to fix the learning objective and improve stability.

[0033] In a specific exemplary embodiment, the Actor network structure is as follows: Input layer (7-dimensional parameter state vector s) → Fully connected layer (128 neurons, ReLU) → Fully connected layer (256 neurons, ReLU) → Fully connected layer (128 neurons, ReLU) → Output layer (8 dimensions, where the first 7 dimensions are action indices logits, and the last dimension is the adjustment magnitude). The action index is obtained by Softmax sampling to obtain i, and the adjustment magnitude is obtained by Tanh activation to obtain Δ∈[−1,1].

[0034] S205, Construct the Critic network: The input to the Critic network is the parameter state vector at the current time, i.e., the current state s, and the action a output by the Actor network; the output of the Critic network is the Q value, which is used to evaluate the expected cumulative reward that can be obtained by performing action a under the current parameter state s. In one specific exemplary embodiment, the Critic network structure is as follows: Input layer (7-dimensional parameter state vector + 7-dimensional action vector, totaling 18 dimensions) → Fully connected layer (256 neurons, ReLU) → Fully connected layer (256 neurons, ReLU) → Fully connected layer (128 neurons, ReLU) → Output layer (1-dimensional Q-value). The network uses the Adam optimizer with a learning rate of 3×10⁻⁶. −4 .

[0035] S206, Design the reward function: The reward function r is calculated based on the goodness of fit R2 between the model prediction value and the actual observation value corresponding to the current state s, where the model prediction value and the actual observation value correspond to the first component of the dynamic state vector, namely the position parameter.

[0036] In a specific exemplary embodiment, the reward function r in step S206 is: r = r1 + r2; Where r1 is the basic reward, specifically the goodness-of-fit R2 of the servo system position output y. y The goodness of fit R² of its derivative y' y’ Weighted sum: ; In the formula, 'a' represents the weighting ratio; The formula for calculating the goodness-of-fit R² is: ; In the formula, Let j be the actual position observation value at time j. Let be the model position prediction value at time j; This represents the average of all observed locations for the samples. , where n represents the total number of samples. It should be noted that the actual position observation at time j corresponds to the data collected in S1.

[0037] r2 represents an increase in rewards; this occurs when the base reward r1 is higher than the historical best base reward. hour: ; In the formula, k is the specific gravity; where, when When r2 is less than 0, r2 is 0. If the value is greater than 0, then the current position is the new optimal base.

[0038] In this exemplary embodiment, the Q value is strongly bound to the curve fitting degree R2, so that the optimization objective of the Critic network is directly aimed at improving the fitting degree of the state equation of the servo system (R2 of y and y'), thus achieving complete alignment between the reinforcement learning objective and the business objective.

[0039] S207, Configure the experience replay mechanism: Configure the experience replay pool to store the transition samples during the training process; each transition sample is a quintuple (s, a, r, s', done), where s' is the new parameter state vector obtained by updating the parameters after executing the action, i.e. the next parameter state, and done is the termination flag.

[0040] It should be noted that the system has a limited amount of identification data. Traditional large-capacity replay pools are prone to high invalid experience ratios and low learning efficiency in small sample scenarios. This exemplary embodiment uses a small-capacity replay pool for storage, which can ensure experience diversity and avoid redundancy. It is a precise optimization for the data characteristics of this project. The replay pool stores (s, a, r, done) quadruple data, where s is the previous state, a is the action, r is the reward, and done is the termination flag. Each time the network is updated, 128 samples are randomly sampled from the replay pool for batch training.

[0041] S208: Set up a soft update mechanism for the target network: Set up target networks for both the Actor network and the Critic network, and slowly update the target network parameters using a soft update method; The target network soft update is used to slowly update the target network parameters. It sets up target networks with identical structures for both the Actor and Critic networks, with initial values ​​and parameters identical to the online network. During the update, the target network parameter is calculated as: Target network parameter = τ * Online network parameter + (1-τ) * Target network parameter, where τ is a very small number (e.g., 0.0001), called the soft update coefficient. The target network update ultimately serves to improve R2 performance, rather than providing a general action value assessment. The core purpose of the soft update is to make the Critic's Q-value assessment more stably reflect the improvement effect of parameter adjustments on R2, thus deeply aligning the algorithm mechanism with business objectives.

[0042] S3: Execute the parameter identification training process, including: S301, Initialization: Initialize the parameter state vector s to the preset parameter values; simultaneously set the historical best reward. The continuous penalty counter is 0.

[0043] S302, Action Output: In each training step, the current parameter state vector s is input into the Actor network. The Actor network outputs action a=(i,Δ) based on the current system state s. After superimposing the exploration noise corresponding to the current training round, the actual action is obtained.

[0044] S303, Parameter Update and Next Parameter State Generation: Execute action a, parse the selected parameter index i and adjustment range Δ, update the corresponding parameter values ​​according to the preset action boundary, and obtain a new parameter state vector as the next parameter state s′; s′=s+ α Δ; S304, Model Prediction and Reward Calculation: Substitute the parameter values ​​in the updated parameter state vector s′ into the state matrix A and control matrix B in the discrete state-space model in S201, and iteratively calculate the values ​​of the servo system's action state vector x at each sampling time t (using this set of parameters, based on the actually acquired input signal). As an incentive, the discrete state-space model is iteratively calculated to extract the position components. As the model's predicted output sequence, simultaneous extraction speed is preferred. ; The model's predicted output sequence is then compared with the actual observed sequence, and the goodness-of-fit R² is calculated. ), and calculate the reward value r based on the reward function; S305, Experience Storage: Store the current transfer sample (s, a, r, s', done) into the experience replay pool; S306, Network Update: Sample a batch of samples (e.g., 128) from the experience replay pool, use the Critic network to calculate the current Q value and the target Q value, and update the Critic network parameters; use the policy gradient to update the Actor network parameters (with the goal of maximizing the expected Q value). S307, rollback mechanism and termination flag determination: If the preset conditions are triggered, the termination flag done is enabled, and the system state representation is rolled back to the historical optimal state to continue exploration; In a specific exemplary embodiment, the determination condition of the termination flag is: if the reward value obtained for a consecutive preset number of times is lower than a preset threshold (20) or is negative, then the rollback operation in step S307 is triggered.

[0045] S308: Repeat steps S302 to S307 until the training termination condition is met. Set the maximum number of training rounds to 500, or when... And If you can maintain this for 50 consecutive rounds, stop training early.

[0046] Specifically, in actual working conditions, microcomputers such as MCUs do not have powerful computing capabilities, cannot meet the hardware requirements for online identification, and do not have the ability to deploy complex models. However, algorithms such as control, control compensation, and fault detection all require models. Therefore, in this example embodiment, an offline identification method is used to identify a simple model.

[0047] Physical data is collected experimentally; the latest action is output using DDPG, tested using a theoretical model, and the reward for the current action is determined based on the evaluation function. The current action score is compared with the historical best score; a higher score results in a reward, while a lower score incurs a penalty. If penalties are incurred multiple times consecutively, the system returns to the optimal score position and continues running until the parameters meet the target.

[0048] Figure 3 The figure shows the identification result obtained by an exemplary embodiment of the present invention. At the end of training, the model prediction effect corresponding to the optimal parameter state vector is: position output goodness of fit. Speed ​​output goodness of fit The identified state matrix A and control matrix B are as follows: ; .

[0049] The parameters were deployed to the MCU for online verification testing. The maximum relative error between the model's predicted output and the actual system output was 3.2%, which meets the requirements for engineering applications.

[0050] In summary, this exemplary embodiment has the following advantages: (1) High identification accuracy: By directly embedding the goodness-of-fit R2 into the reward function, high-precision fitting of the model's predicted output with the actual data in both position and velocity dimensions is achieved. It can reach above 0.99.

[0051] (2) Fast convergence speed: The reward structure of "basic reward + enhancement reward" enables the intelligent agent to quickly perceive parameters and adjust direction; the continuous penalty backoff mechanism avoids invalid exploration, and the convergence speed is improved by more than 60% compared with traditional intelligent optimization algorithms.

[0052] (3) Strong global search capability: The combination of motion noise attenuation design and backoff mechanism ensures both wide-area exploration in the early stage and stable convergence in the later stage, effectively avoiding the local optimum trap in the multidimensional parameter space.

[0053] (4) High parameter safety: By constraining the single-step adjustment range of parameters through action boundary constraints, compared with traditional algorithms that can only constrain the range of parameter values, this invention achieves safe constraints on the parameter change process and avoids model divergence.

[0054] (5) The model is highly deployable: the identification result is a discrete state-space model, which is simple in form and has a small amount of computation. It can be directly ported to embedded platforms such as MCU to meet the engineering application requirements such as real-time control and fault detection.

[0055] (6) Good versatility: The method of the present invention does not depend on a specific system structure and can be applied to offline parameter identification of various linear / nonlinear systems, and has broad industrial application prospects.

[0056] See Figure 4 , Figure 4 This invention illustrates an offline servo system parameter identification system provided in another exemplary embodiment, comprising: Data acquisition and preprocessing module: used to perform tasks such as... Figure 1 Step S1 of the method involves collecting the input and output data of the servo system and performing filtering and differentiation processing. Algorithm configuration module: used to execute algorithms such as... Figure 1 Step S2 of the method involves configuring the network structure, parameter state vector dimension, action space, reward function, and training hyperparameters. Training and recognition module: used to perform tasks such as... Figure 1 Step S3 of the method involves iteratively optimizing the parameter state vector and outputting the optimal parameter set. The result output module is used to map the parameter values ​​in the optimal parameter state vector s obtained from training to the state matrix A and control matrix B of the discrete state space model of the servo system and output them, thus completing the offline parameter identification.

[0057] See Figure 5 , Figure 5 An electronic device provided by yet another exemplary embodiment of the present invention is shown, including a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor performs the steps of the method as described in the first aspect when executing the computer instructions.

[0058] Electronic devices are manifested in the form of general-purpose computing devices. Components of electronic devices may include, but are not limited to: at least one processing unit, at least one storage unit, and a bus connecting different system components (including storage units and processing units).

[0059] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section above, based on various exemplary embodiments of the present invention. For example, the processing unit can perform actions such as... Figure 1 The method shown in the figure.

[0060] The storage unit may include readable media in the form of volatile storage units, such as random access memory (RAM) and / or cache storage units, and may further include read-only memory (ROM).

[0061] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0062] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0063] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0064] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to this exemplary embodiment.

[0065] See Figure 6 , Figure 6 This illustration shows a computer storage medium provided by another exemplary embodiment of the present invention, on which computer instructions are stored, which, when executed, perform the steps of the method as described in the first aspect.

[0066] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product (program product). The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0067] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0068] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0069] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0070] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0071] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for offline identification of servo system parameters, characterized in that, Includes the following steps: S1: Collect the input control signals and output position feedback data of the servo system, and preprocess the data; S2: Design the DDPG algorithm structure, including: S201, Determine the system model structure, which is achieved by determining the discrete state-space model structure of the servo system: ； in This represents the action state vector at the t-th sampling time in the servo system. For location, For speed, For acceleration; This represents the action state vector at the (t+1)th sampling time in the servo system. The system input at the t-th sampling time is A, which is the state matrix, and B, which is the control matrix. All unknown elements of the state matrix A and the control matrix B are the set of parameters to be identified. S202, Define the parameter state vector: The current values ​​of all parameters in the set of parameters to be identified are formed into a vector, which is defined as the parameter state vector s, and serves as the state space of the DDPG algorithm. S203, Define the action space: Define the action dimension according to the number of parameters to be identified, and adjust one parameter for each action a; each action a includes two parts: the index of the selected parameter i and the adjustment magnitude Δ; set the boundary of action a and add exploration noise that decays with the training rounds; S204, Construct the Actor network: The input of the Actor network is the parameter state vector at the current time, i.e., the current state s, and the output of the Actor network is the action a=(i,Δ); the action a means to select the i-th parameter from the parameter state vector s and apply an adjustment amount of Δ. S205, Construct the Critic network: The input to the Critic network is the parameter state vector at the current time, i.e., the current state s, and the action a output by the Actor network; the output of the Critic network is the Q value, which is used to evaluate the expected cumulative reward that can be obtained by performing action a under the current parameter state s. S206, Design the reward function: The reward function r is calculated based on the goodness of fit R2 between the model prediction value and the actual observation value corresponding to the current state s, where the model prediction value and the actual observation value correspond to the first component of the dynamic state vector, namely the position parameter; S207, Configure the experience replay mechanism: Configure the experience replay pool to store the transition samples during the training process; each transition sample is a quintuple (s, a, r, s', done), where s' is the new parameter state vector obtained by updating the parameters after executing the action, i.e. the next parameter state, and done is the termination flag; S208: Set up a soft update mechanism for the target network: Set up target networks for both the Actor network and the Critic network, and slowly update the target network parameters using a soft update method; S3: Execute the parameter identification training process, including: S301, Initialization: Initialize the parameter state vector s to the preset parameter values; S302, Action Output: In each training step, the Actor network outputs an action a=(i,Δ) based on the current system state s; S303, Parameter update and next parameter state generation: Execute action a, parse the selected parameter index i and adjustment range Δ, update the value of the corresponding parameter according to the preset action boundary, and obtain a new parameter state vector as the next parameter state s′; S304, Model Prediction and Reward Calculation: Substitute the parameter values ​​in the updated parameter state vector s′ into the state matrix A and control matrix B in the discrete state-space model in S201, iteratively calculate the values ​​of the servo system's motion state vector x at each sampling time t, and extract the position components. The model prediction output sequence is used as the model prediction output sequence. The model prediction output sequence is compared with the actual observation sequence to calculate the goodness of fit R2, and the reward value r is calculated according to the reward function. S305, Experience Storage: Store the current transfer sample (s, a, r, s', done) into the experience replay pool; S306, Network Update: Sample batch samples from the experience replay pool, use the Critic network to calculate the current Q value and the target Q value, update the Critic network parameters; update the Actor network parameters using the policy gradient; S307, rollback mechanism and termination flag determination: If the preset conditions are triggered, the termination flag done is enabled, and the system state representation is rolled back to the historical optimal state to continue exploration; S308: Repeat steps S302 to S307 until the training termination condition is met.

2. The offline parameter identification method for a servo system according to claim 1, characterized in that: In step S202, based on the characteristics of the state vector x, the system model is a third-order state-space model. , z represents the parameter to be identified, and dt is the discrete time. The parameters to be identified include a set of four parameters in state matrix A and three parameters in control matrix B.

3. The offline parameter identification method for a servo system according to claim 1, characterized in that: The reward function r in step S206: r=r1+r2； Where r1 is the basic reward, specifically the goodness-of-fit R2 of the servo system position output y. y The goodness of fit R² of its derivative y' y’ Weighted sum: ； In the formula, 'a' represents the weighting ratio; r2 represents an increase in rewards; this occurs when the base reward r1 is higher than the historical best base reward. hour: ； In the formula, k is the specific gravity; where, when When r2 is less than 0, r2 is 0. If the value is greater than 0, then the current position is the new optimal base.

4. The offline parameter identification method for a servo system according to claim 3, characterized in that: The formula for calculating the goodness-of-fit R² is: ； In the formula, Let j be the actual position observation value at time j. Let be the model position prediction value at time j; This represents the average of all observed locations for the samples. , where n represents the total number of samples.

5. The offline parameter identification method for a servo system according to claim 1, characterized in that: The amplitude of the exploration noise mentioned in step S203 is set to a large value in the early stage of training, and decreases as the training rounds increase according to a preset attenuation coefficient.

6. The offline parameter identification method for a servo system according to claim 1, characterized in that: In step S307, the determination condition for the termination flag is: if the reward value obtained after a preset number of consecutive attempts is lower than a preset threshold or is negative, then the rollback operation in step S307 is triggered.

7. A servo system parameter offline identification system, characterized in that: include: Data acquisition and preprocessing module: used to execute step S1 of the method as described in any one of claims 1 to 6, to acquire the input and output data of the servo system and perform filtering and differentiation processing; Algorithm configuration module: used to execute step S2 of the method as described in any one of claims 1 to 6, and configure the network structure, parameter state vector dimension, action space, reward function and training hyperparameters; Training and identification module: used to execute step S3 of the method as described in any one of claims 1 to 6, iteratively optimize the parameter state vector, and output the optimal parameter set; The result output module is used to map the parameter values ​​in the optimal parameter state vector s obtained from training to the state matrix A and control matrix B of the discrete state space model of the servo system and output them, thus completing the offline parameter identification.

8. An electronic device comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, characterized in that, When the processor executes the computer instructions, it performs the steps of the method as described in any one of claims 1-6.

9. A computer storage medium storing computer instructions thereon, characterized in that, The computer instructions, when executed, perform the steps of the method as described in any one of claims 1-6.

Images