A motor adaptive control method and system based on deep reinforcement learning

The motor adaptive control method based on deep reinforcement learning, utilizing an innovative deep Q-learning algorithm and online fine-tuning strategy, solves the problems of high precision, fast response, and strong robustness in motor control, achieving efficient and stable motor control results.

CN118508817BActive Publication Date: 2024-11-29MINXI VOCATIONAL & TECHN COLLEGE
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410961704.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-29
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

Existing motor control methods struggle to achieve high-precision control when dealing with nonlinear and time-varying systems. Furthermore, deep reinforcement learning algorithms suffer from problems such as low sample efficiency, estimation bias, poor exploration efficiency, insufficient distribution estimation, and catastrophic forgetting, which limits their application in scenarios requiring high precision, fast response, and strong robustness.

Method used

An adaptive motor control method based on deep reinforcement learning is adopted. By acquiring motor state information, an innovative deep Q-learning algorithm is used to update the deep neural network. Combined with information entropy-first experience replay, adaptive soft update mechanism, state-related noise injection exploration strategy and adaptive quantile distributed Q-learning, online fine-tuning and dynamic optimization are achieved.

Benefits of technology

It achieves high-precision control, fast dynamic response, strong anti-disturbance capability, high energy efficiency and excellent adaptability, significantly improving the performance and robustness of the motor system and meeting the real-time requirements of modern motor control applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118508817B_ABST
    Figure CN118508817B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of motor adaptive control methods, and more specifically, to a motor adaptive control method and system based on deep reinforcement learning, the method comprising the following steps: obtaining state information of a motor motion system; approximating a Q function through a deep neural network based on the state information; updating the deep neural network using an innovative deep Q learning algorithm; determining a motor control action based on the updated deep neural network; applying the motor control action to a motor motion system; collecting new state information and reward information online; and fine-tuning the deep neural network online based on the new state information and reward information. Through the innovative deep Q learning algorithm, a higher control accuracy than traditional methods is achieved, and the steady-state error is reduced to 0.05%. The use of an adaptive noise injection strategy and a distributed Q learning mechanism enables the system to quickly adapt to load changes, and the adjustment time is shortened to 15ms.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of motor adaptive control methods, and more specifically, to a motor adaptive control method and system based on deep reinforcement learning. Background Art

[0002] Motor control technology plays a vital role in industrial automation, electric vehicles, robotics and other fields. With the increasing complexity of application scenarios and the continuous improvement of performance requirements, traditional motor control methods face many challenges. The main existing control methods and their problems are as follows:

[0003] 1. Traditional PID control: PID control is widely used because of its simplicity and reliability. However, when facing nonlinear and time-varying systems, PID control is difficult to achieve high-precision control. Its parameter setting usually relies on experience and lacks adaptive ability, making it difficult to cope with complex situations such as sudden load changes and parameter drift. Under high dynamic performance requirements, the response speed and anti-disturbance ability of PID control often cannot meet the requirements.

[0004] 2. Model Predictive Control (MPC): MPC implements control through predictive models and rolling optimization strategies. It can handle multi-variable, constrained problems, but relies heavily on accurate system models. In practical applications, accurate modeling is often difficult due to factors such as motor parameter drift and load changes. In addition, MPC has a high real-time computational complexity and may face real-time issues in high-speed control scenarios.

[0005] 3. Intelligent control methods: In recent years, intelligent control methods such as fuzzy control and neural network control have been widely studied. These methods have certain self-learning and self-adaptive capabilities, but they have the following problems:

[0006] Fuzzy control: The design of the rule base relies on expert experience and it is difficult to cover all possible working conditions.

[0007] Neural network control: The training process is complex, the real-time learning capability is limited, and there is a problem of local optimality.

[0008] 4. Deep reinforcement learning control: As an emerging control method, deep reinforcement learning (DRL) shows great potential in the field of motor control. However, existing DRL control algorithms (such as conventional DQN) still have some key problems:

[0009] a) Low sample efficiency: The traditional experience replay mechanism randomly samples and ignores the importance differences of samples, resulting in low learning efficiency.

[0010] b) Valuation bias: Conventional DQN is prone to Q-value overestimation, which affects the stability of the control strategy and the optimization effect.

[0011] c) Exploration efficiency: Fixed exploration strategies (such as ε-greedy) have difficulty in achieving a good balance between exploration and exploitation, especially in continuous control problems such as motor systems.

[0012] d) Distribution estimation: Traditional DQN only estimates the expectation of Q value, ignoring uncertainty information and having difficulty dealing with complex probability distribution situations.

[0013] e) Catastrophic forgetting: During online learning, new data may cause the network to forget previously learned knowledge, affecting the continuity of control performance.

[0014] f) Hyperparameter sensitivity: The performance of the DRL algorithm is highly sensitive to hyperparameters (such as learning rate, discount factor, etc.), but lacks an effective automatic tuning mechanism.

[0015] These problems seriously restrict the practical application of deep reinforcement learning in the field of motor control, especially in scenarios requiring high precision, fast response and strong robustness.

[0016] In view of the above-mentioned deficiencies in the prior art, a new motor adaptive control method that can overcome these problems is urgently needed. The motor adaptive control method based on deep reinforcement learning proposed in the present invention is designed to address these challenges. Summary of the invention

[0017] In response to the above technical problems, the present invention proposes a motor adaptive control method and system based on deep reinforcement learning.

[0018] The present invention provides a motor adaptive control method based on deep reinforcement learning, comprising the following steps:

[0019] Get the status information of the motor motion system;

[0020] Based on the state information, approximate the Q function through a deep neural network;

[0021] Updating the deep neural network using an innovative deep Q-learning algorithm;

[0022] Determine motor control actions based on the updated deep neural network;

[0023] applying the motor control action to a motor motion system;

[0024] Collect new status and reward information online;

[0025] Based on the new state information and reward information, the deep neural network is fine-tuned online.

[0026] Specifically, the state information includes the motor speed , motor torque , three-phase current , , , three-phase voltage , , , rotor position angle , Winding temperature , stator temperature and the magnetic linkage estimate The state information is preprocessed by the following steps: a) using the wavelet threshold denoising method, selecting the db4 wavelet, and using the soft threshold function to perform 5-layer decomposition and reconstruction; b) Apply Z-score normalization: ,in is the original data, and are the mean and standard deviation of the dimension respectively. is the standardized data.

[0027] Specifically, the deep neural network is a multi-layer perceptron structure, including:

[0028] Input layer: has the same number of nodes as the dimension of state information;

[0029] Hidden layer 1: has 256 nodes, and the activation function is the Mish function;

[0030] Hidden layer 2: has 128 nodes, and the activation function is the Mish function;

[0031] Output layer: has the same number of nodes as the dimension of the action space, and no activation function;

[0032] Among them, the Mish function is defined as: The network parameters are initialized using the He initialization method: ,in For the The weight matrix of the layer, is the number of input neurons in this layer, Indicates the mean , standard deviation is The normal distribution of .

[0033] Specifically, the innovative deep Q-learning algorithm includes a priority experience replay mechanism based on information entropy, and its implementation steps are as follows:

[0034] a) Calculation status Information entropy: in Status Next select action The probability of

[0035] b) Calculate sample priority: ,in is the sample priority, is the TD error, is a small positive number, is the TD error index, Status The information entropy of is a small positive number, is the entropy index;

[0036] c) Based on sample priority Constructing a priority tree data structure , for efficient sampling;

[0037] d) From Medium Sampling For training, the sampling probability and Directly proportional.

[0038] Specifically, the innovative deep Q learning algorithm also includes a dual Q network structure with an adaptive soft update mechanism, and its implementation steps are as follows:

[0039] a) Initialize the main Q network parameters and target Q network parameters

[0040] b) Calculate the parameter difference:

[0041] in represents the L2 norm;

[0042] c) Calculate the adaptive soft update coefficient:

[0043] in and are the lower and upper limits of the soft update coefficient, respectively. is the slope parameter (e.g. 10), is the difference threshold, is the sigmoid function;

[0044] d) Update the target Q network parameters: .

[0045] Specifically, the innovative deep Q-learning algorithm also includes a state-dependent adaptive noise injection exploration strategy, which is implemented as follows:

[0046] a) Define the state feature representation function

[0047] b) Calculate the noise mean: ,in is the learnable weight matrix, is the learnable bias vector;

[0048] c) Calculate the noise standard deviation: ,in is the learnable weight matrix, is the learnable bias vector,

[0049] d) Generate noise:

[0050] e) Calculate the Q value with noise: ,in Output value for the Q network.

[0051] Specifically, the innovative deep Q-learning algorithm also includes a distributed Q-learning mechanism with adaptive quantiles, and its implementation steps are as follows:

[0052] a) Initialization Quantile , uniformly distributed in the interval [0,1];

[0053] b) For each state-action pair ,calculate Percentile values:

[0054] c) Calculate the distributed TD error: ,in For reward, is the discount factor, , is the next state, that is, the new state that the system transfers from the current state s after executing action a;

[0055] d) Calculate the quantized regression loss: ,in , is the Huber loss function;

[0056] e) Update quantiles: ;

[0057] in is the learning rate, For the general Truncate to Function of the interval.

[0058] Specifically, the step of determining the motor control action includes:

[0059] a) Define the continuous action space ,in , is the dq axis voltage, is the switching frequency, is the magnetic linkage dummy control variable;

[0060] b) Calculate the grid density function: ,in for Function to Action The gradient of is the smoothing parameter;

[0061] c) Calculate the number of adaptive grid points: in Status The information entropy of To adjust the parameters, and are the minimum and maximum number of grid points, respectively;

[0062] d) According to and Dynamically discretize the continuous action space;

[0063] e) Select in the discretized action space The action with the largest value is taken as the control action.

[0064] Specifically, the online fine-tuning step includes:

[0065] a) Use a sliding window strategy to keep the most recent samples of time steps;

[0066] b) Dynamically update the experience replay pool using the reservoirsampling algorithm

[0067] c) Every An incremental training is performed once per control cycle;

[0068] d) Use elastic weight integration algorithm to prevent catastrophic forgetting:

[0069] in is the DQN loss function, is the regularization coefficient, is the diagonal element of the Fisher information matrix and are the current model and the old model respectively. parameters;

[0070] e) Use a meta-controller based on Bayesian optimization to dynamically adjust the hyperparameters of the deep reinforcement learning controller including the learning rate , Exploration Rate , Discount Factor and target network update frequency freq;

[0071] f) The objective function of the meta-controller is: in , , They are control performance, energy efficiency and stability indicators respectively. , , is the corresponding weight coefficient;

[0072] g) Use a Gaussian process regression model to establish a mapping from hyperparameters to the objective function and select the next set of hyperparameters by maximizing the expected improvement.

[0073] The motor adaptive control system based on deep reinforcement learning includes: a state information acquisition module for acquiring state information of the motor motion system;

[0074] A deep neural network module for approximating a Q function based on the state information;

[0075] A deep Q-learning module, used to update the deep neural network using an innovative deep Q-learning algorithm;

[0076] An action determination module, for determining a motor control action based on the updated deep neural network;

[0077] A control execution module, used for applying the motor control action to the motor motion system;

[0078] Online data collection module, used to collect new status information and reward information online;

[0079] An online fine-tuning module is used to perform online fine-tuning on the deep neural network based on the new state information and reward information.

[0080] The present invention has the following beneficial effects:

[0081] 1. High-precision control: Through the innovative deep Q learning algorithm, higher control accuracy is achieved than traditional methods, and the steady-state error is reduced to 0.05%.

[0082] 2. Fast dynamic response: The adaptive noise injection strategy and distributed Q learning mechanism enable the system to quickly adapt to load changes, and the adjustment time is shortened to 15ms.

[0083] 3. Strong anti-disturbance capability: The priority experience replay mechanism based on information entropy significantly improves the system's anti-disturbance capability, and the maximum speed fluctuation is reduced to 2.5%.

[0084] 4. High energy efficiency: The multi-objective optimization strategy improves the average efficiency to 93.5% while ensuring control performance.

[0085] 5. Excellent adaptability: The adaptive soft update mechanism and online fine-tuning strategy enable the system to quickly adapt to parameter changes, and the parameter drift adaptation time is only 0.5s.

[0086] 6. Real-time implementation: Although the computational complexity is slightly higher than that of simple PID, the control cycle calculation time is controlled within 50μs, meeting the real-time requirements of most motor control applications.

[0087] 7. Continuous learning capability: Through online fine-tuning and meta-controller design, the system can continuously optimize and adapt to environmental changes, avoiding catastrophic forgetting problems.

[0088] 8. Robustness: Distributed Q-learning mechanism improves the system's ability to handle uncertainty and enhances the robustness of control.

[0089] In general, the present invention provides a high-performance, high-efficiency, and highly adaptive motor control solution, which effectively overcomes the shortcomings of the prior art and provides a new direction for the development of motor control technology. This method is particularly suitable for modern motor control application scenarios that require high precision, fast response, and strong robustness, such as high-precision industrial automation, electric vehicle power systems, aerospace, and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 This is a flow chart of the motor adaptive control method based on deep reinforcement learning of the present invention.

[0091] Figure 2 This is a schematic diagram of the framework of the motor adaptive control system based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0092] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation method, structure, features and effects of the photovoltaic microgrid low-voltage ride-through fault detection method based on wavelet transform proposed by the present invention are described in detail as follows in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0093] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0094] The present invention provides a motor adaptive control method and system based on deep reinforcement learning, which is used to solve the problem that the existing motor control system is difficult to adapt to load changes and external disturbances under complex working conditions. The method uses a deep reinforcement learning algorithm to adaptively optimize the motor control strategy through continuous interaction with the environment and self-learning, thereby achieving efficient and energy-saving operation of the motor.

[0095] like Figure 1 As shown, the motor adaptive control method based on deep reinforcement learning of the present invention includes the following steps:

[0096] 1. Status information acquisition step, in which the system acquires the status information of the motor motion system. Specifically, the status information includes the motor speed , motor torque , three-phase current , , , three-phase voltage , , , rotor position angle , Winding temperature , stator temperature and the magnetic linkage estimate These status information are collected in real time through high-precision sensors, such as the Omron E6B2-CWZ6C speed sensor (resolution: 2000P / R), KISTLER4503A torque sensor (range: 0-100N·m, accuracy: wait.

[0097] The acquired state information is preprocessed as follows: a) Wavelet threshold denoising method is used, db4 wavelet is selected, and 5-layer decomposition and reconstruction is performed using soft threshold function. This method can effectively remove high-frequency noise in the signal while retaining the important features of the signal. b) For each dimension Apply Z-score normalization: .in is the original data, and are the mean and standard deviation of the dimension respectively. is the standardized data. Standardization can make the state quantities of different dimensions comparable, which is beneficial to the training of neural networks.

[0098] 2. A deep neural network approximates the Q function step. In this step, the system approximates the Q function through a deep neural network based on the preprocessed state information. The present invention adopts a deep neural network with a multi-layer perceptron (MLP) structure, which specifically includes:

[0099] Input layer: has 12 nodes, corresponding to 12 dimensions of state information;

[0100] Hidden layer 1: has 256 nodes, and the activation function is the Mish function;

[0101] Hidden layer 2: has 128 nodes, and the activation function is the Mish function;

[0102] Output layer: has the same number of nodes as the dimension of the action space, and no activation function;

[0103] Among them, the Mish function is defined as:

[0104] Compared with the traditional ReLU function, the Mish function is smooth and non-monotonic. It can avoid the gradient vanishing problem while maintaining nonlinearity, which is conducive to the training of deep networks.

[0105] The network parameters are initialized using the He initialization method:

[0106] in For the The weight matrix of the layer, is the number of input neurons in this layer, Indicates the mean , standard deviation is He initialization can maintain the appropriate gradient amplitude in the deep network and help the network convergence.

[0107] 3. Innovative Deep Q Learning Algorithm Update Step This step uses the innovative deep Q learning algorithm to update the deep neural network. The algorithm contains multiple innovative points, which are specifically implemented as follows:

[0108] 3.1 Innovative deep Q-learning algorithm update steps, priority experience replay mechanism based on information entropy,

[0109] a) Calculation status Information entropy: in Status Next select action Information entropy reflects the uncertainty of the state, and high entropy states usually contain more valuable information.

[0110] This step uses an innovative deep Q learning algorithm to update the deep neural network. The algorithm contains multiple innovations, which are specifically implemented as follows: b) Calculate sample priority: in is the sample priority, is the TD error, is a small positive number (such as 0.01), is the TD error index (e.g. 0.6), Status The information entropy of is a small positive number (such as 0.1), is the entropy index (such as 0.4). This priority calculation method takes into account the TD error and state information entropy, and can more effectively select samples with learning value. In motor control, this helps to focus on samples that contain important dynamic information or reflect system mutations, thereby improving learning efficiency. c) Based on sample priority Constructing a priority tree data structure , for efficient sampling. d) From Medium Sampling For training, the sampling probability and Directly proportional.

[0111] 3.2 Dual-Q network structure with adaptive soft update mechanism

[0112] a) Initialize the main Q network parameters and target Q network parameters .

[0113] b) Calculate the parameter difference:

[0114] in represents the L2 norm. The parameter difference reflects the difference between the main network and the target network.

[0115] c) Calculate the adaptive soft update coefficient:

[0116] in and are the lower and upper limits of the soft update coefficient (such as 0.001 and 0.1), is the slope parameter (e.g. 10), is the difference threshold (such as 0.05), is a sigmoid function. This adaptive soft update mechanism can dynamically adjust the update rate according to the degree of difference in network parameters. In motor control, when the system is in a stable state, a smaller update rate can be used to maintain stability; when the system state changes significantly, a larger update rate can be used to quickly adapt to new working conditions.

[0117] d) Update the target Q network parameters: .

[0118] 3.3 State-dependent adaptive noise injection exploration strategy

[0119] a) Define the state feature representation function , which can extract the key features of the state.

[0120] b) Calculate the noise mean:

[0121] in is the learnable weight matrix, is the learnable bias vector.

[0122] c) Calculate the noise standard deviation:

[0123] in is the learnable weight matrix, is the learnable bias vector, .

[0124] d) Generate noise:

[0125] e) Calculate the Q value with noise:

[0126] in Output value for the Q network.

[0127] This adaptive noise injection strategy can dynamically adjust the degree of exploration based on the current state. In motor control, the degree of exploration can be increased for states with high uncertainty (such as load mutations, external disturbances, etc.), while for stable conditions, the exploration can be reduced to maintain control stability.

[0128] 3.4 Distributed Q-Learning Mechanism with Adaptive Quantiles

[0129] a) Initialization Quantile , evenly distributed in the interval [0,1]. In this embodiment, Take 51.

[0130] b) For each state-action pair ,calculate Percentile values:

[0131] c) Calculate the distributed TD error:

[0132] in For reward, is the discount factor (such as 0.99), , is the next state, that is, the new state that the system transfers from the current state s after executing action a;

[0133] d) Calculate the quantized regression loss:

[0134] in is the Huber loss function.

[0135] e) Update quantiles: in is the learning rate (such as 0.01), clip For the general Truncate to The distributed Q-learning mechanism is able to capture the full distribution of Q values, not just the expected value. In motor control, this helps to better assess the risk and uncertainty of different control actions, leading to more robust decisions.

[0136] 4. Steps to determine motor control action

[0137] In this step, the system determines the motor control action based on the updated deep neural network. The specific implementation is as follows:

[0138] a) Define the continuous action space in , is the dq axis voltage (range: -400V to 400V), is the switching frequency (range: 1kHz to 20kHz), is the magnetic linkage dummy control variable (range: 0 to 1).

[0139] b) Calculate the grid density function: ,in for Function to Action The gradient of is a smoothing parameter (such as 0.1). The grid density function reflects the sensitivity of the Q value to the action.

[0140] c) Calculate the number of adaptive grid points:

[0141] in Status The information entropy of is a tuning parameter (such as 0.5), and are the minimum and maximum number of grid points respectively (e.g. 10 and 100).

[0142] d) According to and Dynamically discretize the continuous action space, using finer meshes in high-density areas and coarser meshes in low-density areas.

[0143] e) Select the action with the largest Q value in the discretized action space as the control action. This dynamic discretization method can reduce the computational complexity while ensuring control accuracy. For motor control, this means that more refined control can be achieved near the critical operating point, while a relatively coarse control strategy can be used in less important areas.

[0144] 5. Control action application step: applying the determined motor control action to the motor motion system. Specifically, the dq axis voltage and The inverse Park transformation is converted into a three-phase voltage command, and the PWM modulator generates a switching signal to control the on and off of the power device. At the same time, according to the determined switching frequency Adjust the carrier frequency of the PWM modulator. Flux linkage virtual control variable Used to adjust the parameters of the flux estimator to adapt to different working conditions.

[0145] 6. Online data collection step: During the operation of the motor, the system continuously collects new state information and reward information. The reward information is calculated using a multi-objective combined reward function:

[0146] in:

[0147]

[0148] is the reference speed, is the maximum speed, is the efficiency at the current moment, is the power loss, is the rated power, is the maximum acceleration, Speed ​​fluctuation caused by load disturbance.

[0149] Weight coefficient , , , Adaptive adjustment is made through: in is the learning rate (such as 0.01), is the regularization coefficient (such as 0.001).

[0150] 7. Online fine-tuning step: Based on the collected new state information and reward information, the deep neural network is fine-tuned online. The specific implementation is as follows:

[0151] a) Use a sliding window strategy to keep the most recent time step samples (such as .

[0152] b) Dynamically update the experience replay pool using the reservoirsampling algorithm , in order to maintain the diversity and representativeness of the sample.

[0153] c) Every Control cycle (such as ) to perform an incremental training.

[0154] d) Use the Elastic Weight Consolidation algorithm to prevent catastrophic forgetting: in is the DQN loss function, is the regularization coefficient (such as 0.1), is the diagonal element of the Fisher information matrix and are the current model and the old model respectively. parameters.

[0155] e) Use a meta-controller based on Bayesian optimization to dynamically adjust the hyperparameters of the deep reinforcement learning controller, including the learning rate , exploration rate e, discount factor and the target network update frequency freq.

[0156] f) The objective function of the meta-controller is:

[0157] in , , They are control performance, energy efficiency and stability indicators respectively. , , is the corresponding weight coefficient.

[0158] g) Use the Gaussian process regression model to establish a mapping from hyperparameters to the objective function and select the next set of hyperparameters by maximizing the expected improvement.

[0159] Please refer to Figure 2 The motor adaptive control system based on deep reinforcement learning of the present invention includes the following modules for executing the above method:

[0160] State information acquisition module 1, which is composed of high-precision sensors and data acquisition systems, including: Omron E6B2-CWZ6C speed sensor; KISTLER 4503A torque sensor; LEMHTFS 400-P current sensor; LEMCV 3-500 voltage sensor; Renishaw RESOLUTE™ position sensor; PT100 temperature sensor; Texas Instruments TMS320F28379D DSP as a data acquisition and preprocessing unit; this module is responsible for real-time acquisition of the state information of the motor motion system and performs preliminary data preprocessing, including wavelet denoising and standardization processing.

[0161] Deep neural network module 2, which consists of a high-performance GPU-accelerated deep learning processor, such as NVIDIA Jetson AGX Xavier. This processor has a 512-core Volta GPU and an 8-core ARM v8.2 64-bit CPU, which can efficiently perform forward propagation and backpropagation calculations of deep neural networks.

[0162] Deep Q learning module 3, which also runs on NVIDIA Jetson AGX Xavier, is responsible for executing innovative deep Q learning algorithms, including core functions such as prioritized experience replay, dual Q network structure, adaptive noise injection, and distributed Q learning. The module also contains a large-capacity solid-state drive (such as a 1TB NVMe SSD) for storing the experience replay pool.

[0163] Action determination module 4, which runs on a real-time operating system (such as FreeRTOS) and is responsible for quickly determining the optimal control action based on the output of the deep neural network. This module uses the parallel computing capability of DSP to achieve dynamic discretization and optimal action selection.

[0164] Control execution module 5, which includes: Texas Instruments TMS320F28379D DSP as the main controller; Infineon IGB module FF300R12ME4 as the power drive unit, and ADI ADSP-CM408F as the auxiliary controller, responsible for PWM generation and over-current protection; this module receives control instructions from the action determination module, performs coordinate transformation and PWM modulation through DSP, and finally drives the IGBT module to control the motor.

[0165] Online data collection module 6, which uses the multi-channel ADC and high-speed DMA of TMS320F28379D DSP to collect the state information and control effect of the motor in real time during operation. After preliminary processing, the collected data is transmitted to the deep learning processor through a high-speed interface (such as PCIe).

[0166] Online fine-tuning module 7, which runs on the NVIDIA Jetson AGX Xavier and is responsible for online fine-tuning of the deep neural network based on newly acquired data. This module also contains an ARM Cortex-A57-based coprocessor specifically designed to execute the Bayesian optimization algorithm and dynamically adjust the hyperparameters of deep reinforcement learning.

[0167] All modules are interconnected through high-speed buses (such as PCIe or RapidIO) to ensure the real-time and reliability of data transmission. A distributed architecture is used to separate real-time control tasks and deep learning tasks to ensure real-time control and learning efficiency. Redundant design and fault detection mechanisms are used to improve the reliability and robustness of the system. A modular design is used to facilitate system upgrades and maintenance.

[0168] The system workflow is as follows: The state information acquisition module continuously collects motor state information. The deep neural network module estimates the Q value based on the current state. The deep Q learning module updates the neural network based on the latest data. The action determination module quickly selects the optimal control action. The control execution module applies the control action to the motor. The online data collection module records new state and reward information. The online fine-tuning module regularly updates the deep neural network and adjusts hyperparameters. Through the collaborative work of the above-mentioned hardware modules, the motor adaptive control method based on deep reinforcement learning of the present invention can be effectively implemented. The system can adapt to changes in motor operating conditions in real time and continuously optimize the control strategy, thereby significantly improving the performance, efficiency and reliability of the motor system.

[0169] In order to verify the effectiveness of the present invention, the applicant provides a comparison between the embodiment of the present invention and three comparative examples, and gives detailed test results and analysis.

[0170] Embodiment 1 (the present invention): Motor adaptive control method based on deep reinforcement learning

[0171] Comparative Example 1: Traditional PID Control Method

[0172] Comparative Example 2: Method based on model predictive control (MPC)

[0173] Comparative Example 3: Conventional Deep Q Learning (DQN) Control Method

[0174] Test indicators and test methods:

[0175] 1. Control accuracy:

[0176] Indicator: Steady-state error (%)

[0177] Detection method: Under rated load, measure the relative error between the actual speed and the given speed when the motor is running stably.

[0178] 2. Dynamic response:

[0179] Metric: Adjustment time (ms)

[0180] Detection method: At rated speed, increase 50% of rated load and measure the time required for the speed to return to within ±2% of the given value.

[0181] 3. Anti-disturbance capability:

[0182] Indicator: Maximum speed fluctuation (%)

[0183] Detection method: At rated speed, apply a pulse disturbance of 20% rated torque and measure the ratio of the maximum speed deviation to the rated speed.

[0184] 4. Energy efficiency:

[0185] Index: Average efficiency (%)

[0186] Detection method: Under different load conditions (25%, 50%, 75%, 100% rated load), measure the average value of the ratio of output mechanical power to input electrical power.

[0187] 5. Adaptability:

[0188] Indicator: Parameter drift adaptation time (s)

[0189] Detection method: Increase the motor stator resistance by 20% and measure the time required for the control system to regain a stable state.

[0190] 6. Computational efficiency:

[0191] Index: Average calculation time of a single control cycle (μs)

[0192] Detection method: Measure the average calculation time of 1000 control cycles.

[0193] The test results are shown in the following table:

[0194] Control Algorithm Steady state error (%) Adjustment time (ms) Maximum speed fluctuation (%) Average efficiency (%) Parameter drift adaptation time (s) Control cycle calculation time (μs) Example 1 0.05 15 2.5 93.5 0.5 50 Comparative Example 1 0.3 50 8 89 5 100 Comparative Example 2 0.1 25 4 91.5 2 10 Comparative Example 3 0.15 30 5.5 92 1.5 45

[0195] The analysis and explanation of the above test results are as follows:

[0196] 1. Control accuracy: The steady-state error of the method of the present invention is the smallest (0.05%), which is 83.3% higher than the traditional PID control, 50% higher than the MPC, and 66.7% higher than the conventional DQN. This shows that the innovative deep Q learning algorithm of the present invention can more accurately approximate the Q function, thereby achieving higher-precision control.

[0197] 2. Dynamic response: The adjustment time of the method of the present invention is the shortest (15ms), which is 70% faster than PID, 40% faster than MPC, and 50% faster than conventional DQN. This is due to the adaptive noise injection exploration strategy and distributed Q learning mechanism in the present invention, which enables the system to adapt to load changes more quickly.

[0198] 3. Anti-disturbance capability: The maximum speed fluctuation of the method of the present invention is the smallest (2.5%), which is 68.8% better than PID, 37.5% better than MPC, and 54.5% better than conventional DQN. This shows that the information entropy-based priority experience playback mechanism of the present invention can effectively learn and cope with various disturbance situations.

[0199] 4. Energy efficiency: The average efficiency of the method of the present invention is the highest (93.5%), which is 4.5 percentage points higher than PID, 2 percentage points higher than MPC, and 1.5 percentage points higher than conventional DQN. This shows that the multi-objective optimization strategy of the present invention can effectively improve energy utilization while ensuring control performance.

[0200] 5. Adaptability: The parameter drift adaptation time of the method of the present invention is the shortest (0.5s), which is 90% faster than PID, 75% faster than MPC, and 66.7% faster than conventional DQN. This highlights the superiority of the adaptive soft update mechanism and online fine-tuning strategy in the present invention, which enables the system to quickly adapt to parameter changes.

[0201] 6. Computational efficiency: Although the control cycle calculation time (50μs) of the method of the present invention is longer than PID, it is much shorter than MPC and slightly better than conventional DQN. Considering the significant advantages of the method of the present invention in other indicators, this calculation time is acceptable and still meets the real-time requirements of most motor control applications.

[0202] The motor adaptive control method based on deep reinforcement learning of the present invention is significantly superior to traditional PID control, model-based predictive control and conventional deep Q learning methods in key indicators such as control accuracy, dynamic response, anti-disturbance capability, energy efficiency and adaptability. These advantages are mainly derived from several innovative points of the present invention:

[0203] 1. The priority experience replay mechanism based on information entropy improves learning efficiency and anti-disturbance ability.

[0204] 2. The dual-Q network structure with adaptive soft update mechanism enhances the stability and adaptability of the algorithm.

[0205] 3. The state-dependent adaptive noise injection exploration strategy improves dynamic response and control accuracy.

[0206] 4. The distributed Q-learning mechanism with adaptive quantiles improves the robustness and accuracy of control.

[0207] 5. The design of online fine-tuning and meta-controller enables the system to continuously optimize and adapt to environmental changes.

[0208] These innovations enable the method of the present invention to perform well in complex and ever-changing motor control scenarios, especially when dealing with nonlinear, time-varying and uncertain problems. Although the computational complexity is slightly higher than that of simple PID control, the significant improvement in performance is worth it. Moreover, with the development of hardware technology, there is room for further improvement in computational efficiency.

[0209] The method of the present invention provides a high-performance, high-efficiency, and highly adaptive solution for motor adaptive control, and is particularly suitable for modern motor control application scenarios that require high precision, fast response, and strong robustness.

[0210] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A motor adaptive control method based on deep reinforcement learning, characterized in that: The following steps are involved: Get the status information of the motor motion system; Based on the state information, approximate the Q function through a deep neural network; Updating the deep neural network using an innovative deep Q-learning algorithm; Determine motor control actions based on the updated deep neural network; applying the motor control action to a motor motion system; Collect new status and reward information online; Based on the new state information and reward information, online fine-tuning the deep neural network; The state information includes the motor speed , motor torque , three-phase current , , , three-phase voltage , , , rotor position angle , Winding temperature , stator temperature and the magnetic linkage estimate The state information is preprocessed by the following steps: a) using wavelet threshold denoising method, selecting db4 wavelet, and using soft threshold function to perform 5-layer decomposition and reconstruction; b) for each dimension Apply Z-score normalization: ,in is the original data, and are the mean and standard deviation of the dimension respectively. is the standardized data; The deep neural network is a multi-layer perceptron structure, including: Input layer: has the same number of nodes as the dimension of state information; Hidden layer 1: has 256 nodes, and the activation function is the Mish function; Hidden layer 2: has 128 nodes, and the activation function is the Mish function; Output layer: has the same number of nodes as the dimension of the action space, and no activation function; Among them, the Mish function is defined as: ; The network parameters are initialized using the He initialization method: ,in For the The weight matrix of the layer, is the number of input neurons in this layer, It means the mean , standard deviation is Normal distribution of The innovative deep Q-learning algorithm includes a priority experience replay mechanism based on information entropy, and its implementation steps are as follows: a) Calculation status Information entropy: ,in Status Next select action probability; b) Calculate sample priority: ,in is the sample priority, is the TD error, is a small positive number, is the TD error index, Status The information entropy of is a small positive number, is the entropy index; c) Based on sample priority Constructing a priority tree data structure , for efficient sampling; d) From Mini-batch sampling For training, the sampling probability and is proportional to; The step of determining the motor control action includes: a) Define the continuous action space ,in , is the dq axis voltage, is the switching frequency, is the magnetic linkage dummy control variable; b) Calculate the grid density function: ,in for Function to Action The gradient of is the smoothing parameter; c) Calculate the number of adaptive grid points: ,in Status The information entropy of To adjust the parameters, and are the minimum and maximum number of grid points, respectively; d) According to and Dynamically discretize the continuous action space; e) Select in the discretized action space The action with the largest value is taken as the control action; The online fine-tuning step includes: a) Use a sliding window strategy to keep the most recent samples of time steps; b) Dynamically update the experience replay pool using the reservoirsampling algorithm ; c) Every An incremental training is performed once per control cycle; d) Use elastic weight integration algorithm to prevent catastrophic forgetting: , in is the DQN loss function, is the regularization coefficient, is the diagonal element of the Fisher information matrix and are the current model and the old model respectively. parameters; e) Use a meta-controller based on Bayesian optimization to dynamically adjust the hyperparameters of the deep reinforcement learning controller including the learning rate , Exploration Rate , Discount Factor and target network update frequency freq; f) The objective function of the meta-controller is: ,in , , They are control performance, energy efficiency and stability indicators respectively. , , is the corresponding weight coefficient; g) Use a Gaussian process regression model to establish a mapping from hyperparameters to the objective function and select the next set of hyperparameters by maximizing the expected improvement.

2. The method according to claim 1, characterized in that: The innovative deep Q learning algorithm also includes a dual Q network structure with an adaptive soft update mechanism, which is implemented as follows: a) Initialize the main Q network parameters and target Q network parameters ; b) Calculate the parameter difference: ; in represents the L2 norm; c) Calculate the adaptive soft update coefficient: , in and are the lower and upper limits of the soft update coefficient, respectively. is the slope parameter, is the difference threshold, is the sigmoid function; d) Update the target Q network parameters: .

3. The method according to claim 1, characterized in that The innovative deep Q-learning algorithm also includes a state-dependent adaptive noise injection exploration strategy, which is implemented as follows: a) Define the state feature representation function ; b) Calculate the noise mean: ,in is the learnable weight matrix, is the learnable bias vector; c) Calculate the noise standard deviation: ,in is the learnable weight matrix, is the learnable bias vector, ; d) Generate noise: ; e) Calculate the Q value with noise: ,in Output value for the Q network.

4. The method according to claim 1, characterized in that The innovative deep Q-learning algorithm also includes a distributed Q-learning mechanism with adaptive quantiles, which is implemented as follows: a) Initialization Quantile , uniformly distributed in the interval [0,1]; b) For each state-action pair ,calculate Percentile values: ; c) Calculate the distributed TD error: ,in For reward, is the discount factor, , For the next state; d) Calculate the quantized regression loss: ,in, , is the Huber loss function; e) Update quantiles: ; in is the learning rate, For the general Truncate to Function of the interval.

5. A motor adaptive control system based on deep reinforcement learning that executes the method described in any one of claims 1 to 4, characterized in that: include: A state information acquisition module is used to acquire state information of the motor motion system; A deep neural network module for approximating a Q function based on the state information; A deep Q-learning module, used to update the deep neural network using an innovative deep Q-learning algorithm; An action determination module, for determining a motor control action based on the updated deep neural network; A control execution module, used for applying the motor control action to the motor motion system; Online data collection module, used to collect new status information and reward information online; An online fine-tuning module, used for performing online fine-tuning on the deep neural network based on the new state information and reward information; The state information includes the motor speed , motor torque , three-phase current , , , three-phase voltage , , , rotor position angle , Winding temperature , stator temperature and the magnetic linkage estimate The state information is preprocessed by the following steps: a) using the wavelet threshold denoising method, selecting the db4 wavelet, and using the soft threshold function to perform 5-layer decomposition and reconstruction; b) Apply Z-score normalization: ,in is the original data, and are the mean and standard deviation of the dimension respectively. is the standardized data; The deep neural network is a multi-layer perceptron structure, including: Input layer: has the same number of nodes as the dimension of state information; Hidden layer 1: has 256 nodes, and the activation function is the Mish function; Hidden layer 2: has 128 nodes, and the activation function is the Mish function; Output layer: has the same number of nodes as the dimension of the action space, and no activation function; Among them, the Mish function is defined as: ; The network parameters are initialized using the He initialization method: ,in For the The weight matrix of the layer, is the number of input neurons in this layer, It means the mean , standard deviation is The normal distribution of .

Citation Information

Patent Citations

  • Immune deep reinforcement learning-based mobile robot autonomous navigation method

    CN113406957A

  • Permanent magnet synchronous motor efficiency optimization control method and system

    CN115149859A