Motor driving methods, apparatus, computer equipment and storage media

Abstract

This application relates to a motor driving method, apparatus, computer device, and storage medium. The method includes: acquiring motor parameters of a target motor and constructing a digital twin model of the motor; initializing an initial reinforcement learning model and a backup reinforcement learning model; constructing a training dataset based on the motor digital twin model and the initial reinforcement learning model, and based on preset motor commands; adjusting the initial reinforcement learning model according to the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model; adjusting the backup reinforcement learning model according to the adjusted initial reinforcement learning model; until the initial reinforcement learning model meets preset conditions, obtaining a target reinforcement learning model; and deploying the target reinforcement learning model to the target motor to achieve high-performance driving of the motor. This method provides an efficient and low-cost solution for driving motors.

Description

Technical Field

[0001] This application relates to the field of motor control technology, and in particular to a motor drive method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Permanent magnet synchronous motors (PMSMs) are widely used in industrial automation, electric vehicles, and robotics due to their high efficiency, high power density, and excellent dynamic response performance. PMSMs typically employ field-oriented control (FOC) strategies. In FOC control, the performance of the d-axis and q-axis current loops is crucial, and the current loops generally use proportional-integral (PI) controllers. The performance of the PI controller directly depends on the tuning of its proportional parameter Kp and integral parameter Ki.

[0003] In related technologies, the proportional parameter Kp and integral parameter Ki are usually adjusted manually, which is inefficient and costly.

[0004] Furthermore, even if the parameters of the PI controller are tuned under specific operating conditions, traditional PI controllers, due to their linear control nature, still fall short in suppressing motor speed ripples and handling complex operating conditions (such as sudden load changes and parameter variations). In particular, the speed control loop, as the outer loop, directly determines the response quality and operational stability of the entire system, but traditional PI controllers struggle to balance the speed of dynamic processes with the stability of steady-state processes. Therefore, relying solely on optimizing fixed PI parameters is insufficient to meet the extreme control precision requirements of modern high-performance servo systems.

[0005] In summary, there is an urgent need for a solution that can drive motors efficiently and at low cost. Summary of the Invention

[0006] Therefore, it is necessary to provide a motor driving method, apparatus, computer equipment, and storage medium that can drive motors efficiently and at low cost, in order to address the above-mentioned technical problems.

[0007] In a first aspect, this application provides a motor driving method. The method includes: acquiring motor parameters of a target motor and constructing a digital twin model of the motor based on the motor parameters; initializing a pre-set initial reinforcement learning model and a backup reinforcement learning model; constructing a training dataset based on the motor digital twin model and the initial reinforcement learning model, and constructing a training dataset based on preset motor commands; adjusting the initial reinforcement learning model according to the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model; adjusting the backup reinforcement learning model according to the adjusted initial reinforcement learning model; until the initial reinforcement learning model reaches preset conditions, obtaining a target reinforcement learning model; deploying the target reinforcement learning model to the target motor, so that the target reinforcement learning model acquires the state parameters of the target motor to generate actual compensation values; and superimposing the actual compensation values ​​with the output of the speed loop proportional-integral controller of the target motor to generate the final q-axis current command.

[0008] In one embodiment, the step of constructing a training dataset based on preset motor commands according to the motor digital twin model and the initial reinforcement learning model includes: driving the motor digital twin model with a preset step size according to the preset motor commands; obtaining the initial state parameters of the motor digital twin model for a target preset step size; the initial state parameters are the state parameters at the start of the target preset step size; the initial state parameters include: initial speed, initial q-axis current, initial d-axis current, initial rotor electrical angle, and initial q-axis output; the target preset step size is any preset step size among multiple preset step sizes; and determining the initial state parameters and the initial reinforcement learning model based on the initial reinforcement learning model. A motor compensation value is determined; the motor compensation value is input into the motor digital twin model to determine the target state parameters and speed sequence; the target state parameters are the state parameters at the end of the target preset step length; the target state parameters include: target speed, target q-axis current, target d-axis current, target rotor electrical angle, and target q-axis output; the speed sequence is multiple motor speeds collected within the target preset step length; a reward value is determined based on the speed sequence and the reward function; the initial state parameters, motor compensation value, target state parameters, and reward value are used as training data corresponding to the target preset step length; a training dataset is constructed based on the training data corresponding to each preset step length.

[0009] In one embodiment, the initial reinforcement learning model includes an initial value network and an initial policy network; the backup reinforcement learning model includes a backup value network and a backup policy network; the step of adjusting the initial reinforcement learning model based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, and adjusting the backup reinforcement learning model based on the adjusted initial reinforcement learning model, until the initial reinforcement learning model meets preset conditions to obtain the target reinforcement learning model, includes: determining a target dataset based on the training dataset; adjusting the initial value network based on the target dataset, according to the backup value network, the backup policy network, and the initial value network; adjusting the initial policy network based on the target dataset, according to the initial policy network and the adjusted initial value network; adjusting the backup value network and the backup policy network based on the adjusted initial value network and the adjusted initial policy network; iteratively adjusting the initial value network and the initial policy network until the initial value network and the initial policy network meet preset conditions to obtain the target reinforcement learning model.

[0010] In one embodiment, the target dataset includes multiple training data sets, which include: initial state parameters, motor compensation values, target state parameters, and reward values. The step of adjusting the initial value network based on the target dataset, according to the backup value network, backup policy network, and initial value network, includes: determining an initial value based on the initial state parameters and motor compensation values ​​corresponding to the target training data, and according to the initial value network; the target training data being any training data set in the target dataset; determining a predicted compensation value based on the target state parameters corresponding to the target training data, and according to the backup policy network; determining a predicted value based on the target state parameters and predicted compensation values ​​corresponding to the target training data, and according to the backup value network; determining a target value based on the reward value corresponding to the target training data and the predicted value; and adjusting the initial value network based on the initial value and the target value using a backpropagation algorithm.

[0011] In one embodiment, adjusting the initial policy network based on the target dataset, according to the initial policy network and the adjusted initial value network, includes: calculating the value gradient corresponding to each training data point in the target dataset based on the target dataset and the adjusted initial value network; calculating the compensation gradient corresponding to each training data point in the target dataset based on the target dataset and the initial policy network; determining the policy gradient based on the value gradient and the compensation gradient corresponding to each training data point; and adjusting the initial policy network based on the policy gradient.

[0012] In one embodiment, adjusting the backup value network and backup policy network based on the adjusted initial value network and the adjusted initial policy network includes: obtaining preset update coefficients; and adjusting the backup value network and backup policy network based on the update coefficients, the adjusted initial value network, and the adjusted initial policy network.

[0013] In one embodiment, the iterative adjustment of the initial value network and the initial policy network until the initial value network and the initial policy network meet preset conditions to obtain the target reinforcement learning model includes: iteratively adjusting the initial value network and the initial policy network until the initial value network and the initial policy network converge, or iterating the initial value network and the initial policy network a preset number of times to obtain the target reinforcement learning model.

[0014] Secondly, this application also provides a motor drive device. The device includes:

[0015] The first construction module is used to obtain the motor parameters of the target motor and construct a digital twin model of the motor based on the motor parameters;

[0016] The initialization module is also used to initialize the pre-set initial reinforcement learning model and to back up the reinforcement learning model;

[0017] The construction module is also used to construct a training dataset based on preset motor instructions, according to the motor digital twin model and the initial reinforcement learning model.

[0018] The second adjustment module is used to adjust the initial reinforcement learning model based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, and to adjust the backup reinforcement learning model based on the adjusted initial reinforcement learning model, until the initial reinforcement learning model meets the preset conditions to obtain the target reinforcement learning model.

[0019] The deployment module is used to deploy the target reinforcement learning model to the target motor, so that the target reinforcement learning model can obtain the state parameters of the target motor to generate actual compensation values, and superimpose the actual compensation values ​​with the output of the speed loop proportional-integral controller of the target motor to generate the final q-axis current command.

[0020] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the methods in the first aspect above.

[0021] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the first aspect described above.

[0022] The aforementioned motor driving method, apparatus, computer equipment, and storage medium acquire motor parameters of the target motor and construct a digital twin model of the motor based on these parameters. They then initialize a pre-set initial reinforcement learning model and a backup reinforcement learning model. Next, based on the motor digital twin model and the initial reinforcement learning model, a training dataset is constructed using preset motor commands. The initial reinforcement learning model is then adjusted based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model. The backup reinforcement learning model is then adjusted based on the adjusted initial reinforcement learning model until the initial reinforcement learning model meets preset conditions, resulting in a target reinforcement learning model. Finally, the target reinforcement learning model is deployed to the target motor, enabling it to acquire the target motor's state parameters, generate actual compensation values, and superimpose these actual compensation values ​​with the output of the target motor's speed loop proportional-integral controller to generate the final q-axis current command. This achieves efficient and low-cost motor driving. Attached Figure Description

[0023] Figure 1 This is an application environment diagram of the motor drive method in one embodiment;

[0024] Figure 2 This is a flowchart illustrating a motor driving method in one embodiment;

[0025] Figure 3 This is a structural block diagram of the motor drive device in one embodiment;

[0026] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] Permanent magnet synchronous motors (PMSMs) are widely used in industrial automation, electric vehicles, and robotics due to their high efficiency, high power density, and excellent dynamic response performance. PMSMs typically employ field-oriented control (FOC) strategies, the core of which lies in precisely controlling the stator current to separately control the motor's flux linkage and torque, achieving decoupled control of an AC motor similar to a DC motor. In FOC control strategies, the performance of the d-axis and q-axis current loops is crucial, directly affecting the motor's torque response speed and control accuracy. The current loops generally employ proportional-integral (PI) controllers, whose performance directly depends on the tuning of their proportional parameter Kp and integral parameter Ki.

[0029] In related technologies, the proportional parameter Kp and integral parameter Ki are typically adjusted manually or through theoretical calculations, which is inefficient and costly. These technologies heavily rely on manual intervention and have long iteration cycles, significantly increasing product development and production costs. Furthermore, manual tuning or traditional theoretical calculations struggle to find the globally optimal parameter combination. This is particularly true in suppressing speed ripples and torque pulsations during steady-state motor operation, often failing to achieve ideal results. Fluctuations in motor rotational speed reduce control precision, increase wear on mechanical structures, and generate annoying operating noise. Moreover, changes in motor workload, supply voltage, or operating environment can render preset fixed PI parameters inapplicable, leading to a sharp decline in control performance and insufficient robustness of the motor system.

[0030] The motor driving method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Server 104 is used to execute the motor driving method. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0031] To address the aforementioned problems, in one embodiment of this application, such as Figure 2 As shown, a motor driving method is provided, including the following steps:

[0032] Step 201: Obtain the motor parameters of the target motor and construct a digital twin model of the motor based on the motor parameters.

[0033] The motor parameters of the target motor include various structural performance parameters of the target motor, the connection relationship of the circuit components driving the motor, and the parameters of the circuit components. After obtaining the motor parameters of the target motor, the target motor is modeled to construct a digital twin model of the motor. The digital twin model of the motor is a digitized simulation model of the target motor, which can simulate the physical behavior of the target motor.

[0034] In this application, the digital twin model of the motor includes: a three-phase inverter module, a permanent magnet synchronous motor (PMSM) body module, a rotor position-based coordinate transformation module, and a field-oriented control module based on FOC logic. The field-oriented control module contains a d-axis PI controller and a q-axis PI controller. The d-axis PI controller is used to implement the d-axis current loop, and the q-axis PI controller is used to implement the q-axis current loop.

[0035] In the embodiments of this application, the field-oriented control module adopts an industry-standard inner and outer loop decoupled control structure. The speed loop is the outer loop, and the d-axis and q-axis current loops are the inner loops. The speed loop is responsible for calculating the torque command through its PI controller based on the error between the set target speed and the actual speed fed back by the motor. This command is specifically expressed as the command value of the q-axis current. Subsequently, the d-axis and q-axis current loops, as the inner loops, are responsible for quickly and accurately tracking their respective current command values. The innovation of this invention lies in that it does not directly interfere with the PI controller of the inner loop (current loop), but rather performs intelligent dynamic compensation on the q-axis current command output by the speed loop between the speed loop and the current loop. This achieves efficient optimization of the motor's dynamic performance and steady-state stability without changing the underlying controller structure.

[0036] The d-axis current loop is used to control the excitation magnetic field of the target motor, enhancing or weakening the rotor magnetic field, while the q-axis current loop is used to control the torque and speed of the target motor.

[0037] Furthermore, the motor digital twin model incorporates standardized Action Input, State Output, and simulation control interfaces, enabling precise invocation and control by external programming environments (such as Python) in discrete time steps. This fulfills the interactive environment requirements of reinforcement learning algorithms. The Action Input interface receives compensation values ​​from the reinforcement learning model, while the State Output interface outputs the current state parameters of the motor digital twin model. The motor digital twin model simulates at preset step sizes. Once the simulation duration reaches the preset step size, the simulation pauses until the simulation control interface receives a continue command from the external control program, at which point the simulation resumes.

[0038] It should be noted that the digital twin model of the motor, when built in simulation software such as MATLAB and Simulink, not only simulates the physical behavior of the motor, but also serves as an "environment" for real-time interaction with reinforcement learning algorithms.

[0039] Step 202: Initialize the pre-set initial reinforcement learning model and the backup reinforcement learning model.

[0040] The initial reinforcement learning model is the first reinforcement learning model. The backup reinforcement learning model is a reinforcement learning model with the same structure as the initial model but whose network parameters are updated relatively slowly; this is the target network, used to provide stable target values ​​for subsequent adjustments to the initial reinforcement learning model based on the training dataset, the initial reinforcement learning model, and the backup model, thereby improving the overall efficiency of the training process. A reinforcement learning model is a model that learns to maximize long-term cumulative rewards through interaction with the environment by an agent, and includes a policy network and a value network.

[0041] Step 203: Based on the motor digital twin model and the initial reinforcement learning model, construct a training dataset based on preset motor commands.

[0042] The preset motor commands are used to drive the digital twin model of the motor based on a preset step size, enabling the digital twin model to reach and maintain the target speed. During this process, the initial reinforcement learning model acquires the state parameters of the digital twin model and generates corresponding motor compensation values ​​based on these parameters. The digital twin model then acquires these compensation values ​​and performs simulations to obtain multiple training data sets, which are then used to construct the training dataset. The target speed is the preset target motor speed. The multiple training data sets include the state parameters of the digital twin model and the motor compensation values.

[0043] Step 204: Based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, adjust the initial reinforcement learning model. Based on the adjusted initial reinforcement learning model, adjust the backup reinforcement learning model. Continue until the initial reinforcement learning model meets the preset conditions to obtain the target reinforcement learning model.

[0044] Based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, the neural network in the initial reinforcement learning model is adjusted. Then, based on the adjusted initial reinforcement learning model, the neural network in the backup reinforcement learning model is adjusted. This process continues until the initial reinforcement learning model meets the preset conditions, thus obtaining the target reinforcement learning model.

[0045] The objective reinforcement learning model is the final reinforcement learning model that has been trained.

[0046] Step 205: Deploy the target reinforcement learning model to the target motor so that the target reinforcement learning model can obtain the state parameters of the target motor to generate actual compensation values, and superimpose the actual compensation values ​​with the output of the speed loop proportional-integral controller of the target motor to generate the final q-axis current command.

[0047] It is important to emphasize that after deploying the target reinforcement learning model to the target motor in step 205, its operation is a supplementary compensation control based on the speed loop output. Specifically, in the motor's field-oriented control (FOC) closed-loop system, the proportional-integral (PI) controller of the speed loop first calculates a basic q-axis current command based on the error between the target speed and the actual speed. Simultaneously, the target reinforcement learning model acquires the motor's current state parameters (including speed, current, etc.) in real time and outputs a dynamic actual compensation value. This compensation value is summed (superimposed) with the basic q-axis current command output by the speed loop PI controller to form a corrected, final q-axis current command. This final command is then sent to the q-axis current loop as its control target, thereby achieving precise control of the motor's torque and speed. This approach allows the reinforcement learning model to effectively compensate for the shortcomings of traditional speed loop PI controllers in terms of dynamic response and disturbance suppression.

[0048] The status parameters include: the current speed of the target motor, q-axis current, d-axis current, rotor electrical angle, and q-axis output. Speed ​​is the rotational speed of the target motor; q-axis current is the input current of the q-axis current loop of the target motor; d-axis current is the input current of the d-axis current loop of the target motor; rotor electrical angle is the angle of the permanent magnet magnetic field of the target motor's rotor in space; and q-axis output is the output value of the speed loop PI controller for the target motor.

[0049] In the aforementioned motor driving method, the motor parameters of the target motor are obtained, and a digital twin model of the motor is constructed based on these parameters. A pre-set initial reinforcement learning model and a backup reinforcement learning model are initialized. Then, a training dataset is constructed based on the motor digital twin model and the initial reinforcement learning model, using preset motor commands. Next, the initial reinforcement learning model is adjusted based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model. The backup reinforcement learning model is then adjusted based on the adjusted initial reinforcement learning model. This process continues until the initial reinforcement learning model meets preset conditions, resulting in the target reinforcement learning model. Finally, the target reinforcement learning model is deployed to the target motor, enabling it to acquire the target motor's state parameters, generate actual compensation values, and drive the target motor's proportional-integral controller based on these actual compensation values. This achieves efficient and low-cost motor driving.

[0050] It should be noted that the above-described motor drive method achieves end-to-end automatic tuning of control parameters, requiring no manual intervention from model building to parameter output. This frees engineers from tedious parameter tuning work, greatly improving R&D efficiency. Furthermore, through a specially designed reward function, it achieves superior control performance compared to traditional methods, significantly reducing speed fluctuations during steady-state motor operation and improving system stability and control accuracy. In addition, by introducing noise during training and allowing the agent to explore a broad state space, the trained controller, i.e., the target reinforcement learning model, exhibits excellent generalization ability and stronger adaptability and suppression of external disturbances such as load changes. Finally, the above-described motor drive method can be applied to motors beyond a specific model. By adjusting the motor's physical parameters, such as resistance, inductance, and flux linkage, in the simulation modeling unit, this scheme can be easily applied to permanent magnet synchronous motors of different specifications.

[0051] In other embodiments of this application, the training dataset is constructed based on preset motor commands, according to the motor digital twin model and the initial reinforcement learning model, including:

[0052] Step 1: Drive the digital twin model of the motor according to the preset motor command with a preset step size.

[0053] Based on a preset step size, the motor digital twin model is driven according to preset motor commands, so that the speed of the motor digital twin model reaches the target speed and maintains the target speed.

[0054] It should be noted that the other steps in this application are all performed during the stage when the digital twin model of the motor maintains the target speed.

[0055] Step 2: Preset the step size for the target and obtain the initial state parameters of the motor digital twin model.

[0056] The initial state parameters are those at the start of the target preset step size. These parameters include: initial speed, initial q-axis current, initial d-axis current, initial rotor electrical angle, and initial q-axis output. The preset step size is a preset time step. During simulation, the motor digital twin model divides the entire process into multiple preset steps. After completing one preset step, the simulation pauses. It only resumes after the simulation control interface receives a "continue" command from the external control program. The target preset step size can be any of the multiple preset step sizes.

[0057] It should be noted that although in this embodiment, the relevant steps are only performed for the target preset step size, in actual applications, the relevant steps will be performed for each preset step size.

[0058] Step 3: Determine the motor compensation value based on the initial state parameters and the initial reinforcement learning model.

[0059] The initial state parameters are input into the initial reinforcement learning model to determine the motor compensation value.

[0060] It should be noted that, in this embodiment, the motor compensation value is a compensation value used to correct the speed loop output for the q-axis current command.

[0061] Step 4: Input the motor compensation value into the motor digital twin model to determine the target state parameters and speed sequence.

[0062] The target state parameters are the state parameters at the end of the target preset step length; the target state parameters include: target speed, target q-axis current, target d-axis current, target rotor electrical angle, and target q-axis output; the speed sequence is the multiple motor speeds collected within the target preset step length.

[0063] The motor compensation value is input into the motor digital twin model. The speed of the motor digital twin model is sampled during the target preset step period to obtain the speed sequence. After the target preset step period ends, the state parameters of the motor digital twin model are used as the target state parameters.

[0064] Step 5: Determine the reward value based on the rotational speed sequence and the reward function.

[0065] The reward value is calculated and determined based on the reward function and the rotational speed sequence.

[0066] The reward value is the reward value for the current preset step size.

[0067] For example, the reward function is Where N is the number of multiple rotational speeds in the rotational speed sequence. This represents the average value of multiple speeds in the speed sequence.

[0068] Step 6: Use the initial state parameters, motor compensation value, target state parameters, and reward value as training data corresponding to the target preset step size.

[0069] Step 7: Construct a training dataset based on the training data corresponding to each preset step size.

[0070] The training data corresponding to each preset step size is used as the training dataset.

[0071] It should be noted that in this embodiment, the motor digital twin model is driven by a preset motor command with a preset step size. Then, for the target preset step size, the initial state parameters of the motor digital twin model are obtained. Based on the initial state parameters and the initial reinforcement learning model, the motor compensation value is determined. The motor compensation value is then input into the motor digital twin model to determine the target state parameters and speed sequence. Based on the speed sequence and the reward function, the reward value is determined. The initial state parameters, motor compensation value, target state parameters, and reward value are then used as training data corresponding to the target preset step size. Finally, a training dataset is constructed based on the training data corresponding to each preset step size, laying the foundation for subsequent training of the initial reinforcement learning model based on the training dataset.

[0072] In other embodiments of this application, the initial reinforcement learning model includes an initial value network and an initial policy network; the backup reinforcement learning model includes a backup value network and a backup policy network.

[0073] It should be noted that the initial reinforcement learning model employs the Deep Deterministic Policy Gradient (DDPG) algorithm. This algorithm is an Actor-Critic method suitable for continuous action space problems, and is particularly well-suited for handling continuously changing physical quantities in motor control, such as current and voltage.

[0074] The initial policy network is a deep neural network, which directly outputs a specific motor compensation value based on the initial state parameters. Its structure can be a hidden layer with two 256 neurons using the ReLU activation function and an output layer activated by tanh, to ensure that the output motor compensation value is normalized to the range of [-1, 1].

[0075] The initial value network is a deep neural network whose function is to evaluate the value Q of performing a certain motor compensation value under the initial state parameters, i.e., the expected long-term return. Its inputs are the initial state parameters and the motor compensation value, and its output is a scalar value Q.

[0076] The backup value network is a copy of the initial value network, and the backup policy network is a copy of the initial policy network. It should be noted that the network parameter updates of the backup value network and the backup policy network are slower than those of the initial value network and the initial policy network. The backup value network and the backup policy network are used to improve the efficiency of training the initial reinforcement learning model until convergence.

[0077] In other embodiments of this application, the data interaction between the motor digital twin model and the initial reinforcement learning model is implemented programmatically. The program launches a background MATLAB engine instance, and then loads and starts the motor digital twin model with the configured interface through this engine instance. At each preset step, the program writes the motor compensation value output by the initial reinforcement learning model into the Action Input interface of the motor digital twin model through the engine API. The program commands the motor digital twin model to run for a preset step length dt. After the motor digital twin model pauses, the program reads new state data from the State Output interface through the engine API and calculates the reward value.

[0078] Based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, adjust the initial reinforcement learning model; then, based on the adjusted initial reinforcement learning model, adjust the backup reinforcement learning model; until the initial reinforcement learning model meets the preset conditions, the target reinforcement learning model is obtained; including:

[0079] Step 1: Determine the target dataset based on the training dataset.

[0080] The target dataset is a subset of the training dataset, and the number of training data in the target dataset is a preset number.

[0081] Based on a preset number, a preset number of training data points are randomly selected from the training dataset to serve as the target dataset.

[0082] Step 2: Based on the target dataset, adjust the initial value network according to the backup value network, the backup strategy network, and the initial value network.

[0083] Based on the target dataset, an initial value is determined according to the initial value network, and a target value is determined according to the backup value network and the backup strategy network. Finally, based on the initial value and the target value, the initial value network is adjusted according to the backpropagation algorithm.

[0084] Step 3: Based on the target dataset, adjust the initial policy network according to the initial policy network and the adjusted initial value network.

[0085] Based on the target dataset, the value gradient is calculated according to the adjusted initial value network, the compensation gradient is calculated according to the initial policy network, the policy gradient is determined according to the value gradient and the compensation gradient, and the initial policy network is adjusted according to the policy gradient.

[0086] Step 4: Adjust the backup value network and backup policy network based on the adjusted initial value network and the adjusted initial policy network.

[0087] Based on preset update coefficients, the backup value network and backup strategy network are adjusted according to the adjusted initial value network and the adjusted initial strategy network.

[0088] Step 5: Iteratively adjust the initial value network and the initial policy network until they meet the preset conditions to obtain the target reinforcement learning model.

[0089] Based on the preset conditions, repeat steps 1 to 4 above, iteratively adjusting the initial value network and the initial policy network until the initial value network and the initial policy network meet the preset conditions, thus obtaining the target reinforcement learning model.

[0090] By repeating steps 1 to 4 above, the initial value network and the initial policy network are iteratively adjusted until they meet the preset conditions, thereby obtaining the target reinforcement learning model and improving the performance of the final target reinforcement learning model.

[0091] In other embodiments of this application, the target dataset includes multiple training data, which include: initial state parameters, motor compensation values, target state parameters, and reward values.

[0092] The initial state parameters are stored in a 5-dimensional continuous state space. For example, ,in, This refers to the motor speed. and These are the q-axis current and the d-axis current, respectively. For rotor electrical angle, Output for the q-axis. These are the initial state parameters.

[0093] The motor compensation value is the motor compensation value corresponding to the initial state parameters, and it is stored in a one-dimensional continuous compensation space. For example, ,in, This is the motor compensation value.

[0094] Understandably, the target state parameters are stored in the same way as the initial state parameters.

[0095] Based on the target dataset, the initial value network is adjusted according to the backup value network, backup policy network, and initial value network, including:

[0096] Step 1: Based on the initial state parameters and motor compensation values ​​corresponding to the target training data, determine the initial value according to the initial value network.

[0097] The initial state parameters and motor compensation values ​​corresponding to the target training data are input into the initial value network to determine the initial value.

[0098] The target training data is any training data in the target dataset.

[0099] It should be noted that although in this embodiment, the relevant steps are performed only on the target training data, in actual applications, the relevant steps will be performed on each training data in the target dataset.

[0100] Step 2: Based on the target state parameters corresponding to the target training data, determine the prediction compensation value according to the backup policy network.

[0101] The target state parameters corresponding to the target training data are input into the backup policy network to determine the prediction compensation value.

[0102] Step 3: Based on the target state parameters and predicted compensation values ​​corresponding to the target training data, determine the predicted value according to the backup value network.

[0103] The target state parameters and predicted compensation values ​​corresponding to the target training data are input into the backup value network to determine the predicted value.

[0104] Step 4: Determine the target value based on the reward value and prediction value corresponding to the target training data.

[0105] In fact, ,in, For target value, The reward value corresponding to the target training data. The preset discount factor, For predictive value, The target state parameters are the training data corresponding to the target. To predict the compensation value.

[0106] The target value is obtained by adding the reward value corresponding to the target training data to the predicted value multiplied by a preset discount factor.

[0107] Step 5: Based on the initial value and the target value, adjust the initial value network using the backpropagation algorithm.

[0108] The loss between the initial value and the target value is calculated based on the initial value and the target value, and the network parameters of the initial value network are adjusted based on the backpropagation algorithm.

[0109] It should be noted that by determining the initial value and target value based on the backup value network, backup strategy network, and initial value network, and then adjusting the initial value network based on the backpropagation algorithm according to the initial value and target value, the performance of the initial value network is improved.

[0110] In other embodiments of this application, adjusting the initial policy network based on the target dataset, according to the initial policy network and the adjusted initial value network, includes:

[0111] Step 1: Based on the target dataset, calculate the value gradient corresponding to each training data in the target dataset according to the adjusted initial value network.

[0112] It should be noted that the value gradient is the gradient of the value Q of the initial value network output with respect to the motor compensation value of the initial policy network output. Based on the target dataset, the value gradient corresponding to each training data in the target dataset is calculated according to the adjusted initial value network. This is specifically implemented using the built-in functions within the reinforcement learning framework.

[0113] The value gradient corresponds one-to-one with each training data point in the target dataset.

[0114] Step 2: Based on the target dataset, calculate the compensation gradient corresponding to each training data in the target dataset according to the initial policy network.

[0115] It should be noted that the compensation gradient refers to the gradient of the motor compensation value output by the initial policy network with respect to the initial policy network itself. Based on the target dataset, the compensation gradient corresponding to each training data point in the target dataset is calculated according to the initial policy network. This is specifically implemented using a built-in function within the reinforcement learning framework.

[0116] The compensation gradient corresponds one-to-one with each training data point in the target dataset.

[0117] Step 3: Determine the policy gradient based on the value gradient and compensation gradient corresponding to each training data point. The "value gradient" represents the sensitivity of the value network to the compensation value output by the policy network, while the "compensation gradient" transmits this sensitivity to the internal parameters of the policy network through a chain rule, thereby optimizing the policy.

[0118] For example, ,in, The value gradient corresponding to each training data point in the target dataset. The compensation gradient is the gradient corresponding to each training data in the target dataset. Let M be the policy gradient, and M be the number of training datasets in the target dataset.

[0119] Step 4: Adjust the initial policy network according to the policy gradient.

[0120] Adjust the network parameters of the initial policy network based on the policy gradient.

[0121] It should be noted that the adjustment of the initial policy network based on the policy gradient is specifically implemented through the Optimizer built into the DDPG algorithm framework.

[0122] By calculating the value gradient and compensation gradient corresponding to each training data point in the target dataset, the policy gradient is determined, and the initial policy network is adjusted based on the policy gradient, thereby improving the performance of the initial policy network.

[0123] In other embodiments of this application, adjusting the backup value network and the backup policy network based on the adjusted initial value network and the adjusted initial policy network includes:

[0124] Step 1: Obtain the preset update coefficients.

[0125] Step 2: Adjust the backup value network and backup policy network based on the update coefficients, the adjusted initial value network, and the adjusted initial policy network.

[0126] Based on the updated coefficients and the adjusted initial value network, adjust the network parameters of the backup value network; based on the updated coefficients and the adjusted initial policy network, adjust the network parameters of the backup policy network.

[0127] For example, , The network parameters for the backup value network and the backup strategy network represent the set of all weights and biases in the backup value network or the backup strategy network. To update the coefficients, The network parameters of the initial value network, representing the initial value network or the set of all weights and biases in the initial value network.

[0128] The arrow points to For the updated network parameters, the formula in These are the network parameters before the update.

[0129] In other embodiments of this application, iteratively adjusting the initial value network and the initial policy network until the initial value network and the initial policy network meet preset conditions to obtain the target reinforcement learning model includes:

[0130] The initial value network and initial policy network are iteratively adjusted until they converge, or until the initial value network and initial policy network are iterated a predetermined number of times, to obtain the target reinforcement learning model.

[0131] The initial value network and initial policy network are continuously iterated and adjusted. When the initial value network and initial policy network converge, or when the number of iterations of the initial value network and initial policy network reaches the preset number, the target reinforcement learning model is finally obtained.

[0132] In other embodiments of this application, a target reinforcement learning model is deployed to a target motor so that the model acquires the motor's state parameters to generate actual compensation values. The proportional-integral controller (PI controller) of the target motor is then driven based on these actual compensation values. Specifically, this involves model conversion and optimization. Since actual motor controllers are typically resource-constrained embedded systems, such as microcontrollers (MCUs), and cannot directly run Python or PyTorch frameworks, the initial policy network in the trained target reinforcement learning model must be converted and optimized. First, the initial policy network is converted into a general, framework-independent intermediate format. Then, for the target embedded hardware platform, a specialized toolchain is used to optimize the intermediate format initial policy network and generate executable code. The executable code is integrated into the target motor's controller, and real-time state parameters of the motor are acquired through sensors. By inputting these real-time state parameters into the target reinforcement learning model, corresponding actual compensation values ​​are generated. These values ​​are superimposed on the output of the speed loop PI controller, and the result serves as the target instruction for the q-axis current loop, thereby achieving high-performance driving of the target motor.

[0133] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0134] Based on the same inventive concept, this application also provides a motor drive device for implementing the motor drive method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more motor drive device embodiments provided below can be found in the limitations of the motor drive method described above, and will not be repeated here.

[0135] In one embodiment of this application, such as Figure 3 As shown, a motor drive device is provided, comprising:

[0136] The first construction module 100 is used to obtain the motor parameters of the target motor and construct a digital twin model of the motor based on the motor parameters.

[0137] The initialization module 200 is used to initialize the pre-set initial reinforcement learning model and the backup reinforcement learning model.

[0138] The second construction module 300 is used to construct a training dataset based on preset motor instructions, according to the motor digital twin model and the initial reinforcement learning model.

[0139] The adjustment module 400 is used to adjust the initial reinforcement learning model based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, and to adjust the backup reinforcement learning model based on the adjusted initial reinforcement learning model, until the initial reinforcement learning model meets the preset conditions to obtain the target reinforcement learning model.

[0140] The deployment module 500 is used to deploy the target reinforcement learning model to the target motor, so that the target reinforcement learning model can obtain the state parameters of the target motor to generate actual compensation values, and superimpose the actual compensation values ​​with the output of the speed loop proportional-integral controller of the target motor to generate the final q-axis current command.

[0141] The second construction module 300 is further configured to drive the motor digital twin model according to the preset motor command with a preset step size; obtain the initial state parameters of the motor digital twin model for a target preset step size; the initial state parameters are the state parameters at the start of the target preset step size; the initial state parameters include: initial speed, initial q-axis current, initial d-axis current, initial rotor electrical angle, and initial q-axis output; the target preset step size is any preset step size among multiple preset step sizes; determine the motor compensation value according to the initial state parameters and the initial reinforcement learning model; and input the motor compensation value into the motor. A digital twin model is used to determine target state parameters and a speed sequence. The target state parameters are the state parameters at the end of the target preset step length. The target state parameters include: target speed, target q-axis current, target d-axis current, target rotor electrical angle, and target q-axis output. The speed sequence consists of multiple motor speeds collected within the target preset step length. A reward value is determined based on the speed sequence and a reward function. The initial state parameters, motor compensation value, target state parameters, and reward value are used as training data corresponding to the target preset step length. A training dataset is constructed based on the training data corresponding to each preset step length.

[0142] The adjustment module 400 is further configured to: determine a target dataset based on the training dataset; adjust the initial value network based on the target dataset, according to the backup value network, the backup policy network, and the initial value network; adjust the initial policy network based on the target dataset, according to the initial policy network and the adjusted initial value network; adjust the backup value network and the backup policy network based on the adjusted initial value network and the adjusted initial policy network; iteratively adjust the initial value network and the initial policy network until the initial value network and the initial policy network meet preset conditions, thereby obtaining a target reinforcement learning model.

[0143] The adjustment module 400 is further configured to determine an initial value based on the initial state parameters and motor compensation value corresponding to the target training data, according to the initial value network; the target training data is any training data in the target dataset; determine a predicted compensation value based on the target state parameters corresponding to the target training data, according to the backup policy network; determine a predicted value based on the target state parameters and predicted compensation value corresponding to the target training data, according to the backup value network; determine a target value based on the reward value corresponding to the target training data and the predicted value; and adjust the initial value network based on the backpropagation algorithm according to the initial value and the target value.

[0144] The adjustment module 400 is further configured to: calculate the value gradient corresponding to each training data in the target dataset based on the target dataset and according to the adjusted initial value network; calculate the compensation gradient corresponding to each training data in the target dataset based on the target dataset and according to the initial policy network; determine the policy gradient based on the value gradient and the compensation gradient corresponding to each training data; and adjust the initial policy network based on the policy gradient.

[0145] The adjustment module 400 is also used to obtain preset update coefficients; and adjust the backup value network and backup strategy network according to the update coefficients, the adjusted initial value network, and the adjusted initial strategy network.

[0146] The adjustment module 400 is also used to iteratively adjust the initial value network and the initial policy network until the initial value network and the initial policy network converge, or the initial value network and the initial policy network are iterated a preset number of times to obtain the target reinforcement learning model.

[0147] Each module in the aforementioned motor drive device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0148] In one embodiment of this application, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores all relevant data for executing the motor drive method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a motor drive method.

[0149] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0150] In one embodiment of this application, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the motor driving method in the above embodiment.

[0151] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program being executed by a processor to implement the steps of the motor drive method in the above-described method embodiments.

[0152] In one embodiment of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the motor drive method in the above-described method embodiments.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A motor driving method, characterized in that, The method includes: Obtain the motor parameters of the target motor, and construct a digital twin model of the motor based on the motor parameters; Initialize the pre-set initial reinforcement learning model and backup reinforcement learning model; the initial reinforcement learning model adopts the Deep Deterministic Policy Gradient (DDPG) algorithm; Based on the aforementioned digital twin model of the motor and the initial reinforcement learning model, a training dataset is constructed based on preset motor commands; Based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, the initial reinforcement learning model is adjusted, and the backup reinforcement learning model is adjusted based on the adjusted initial reinforcement learning model; until the initial reinforcement learning model meets the preset conditions, the target reinforcement learning model is obtained. The target reinforcement learning model is deployed to the target motor so that the target reinforcement learning model can obtain the state parameters of the target motor to generate actual compensation values, and the actual compensation values ​​are superimposed with the output of the speed loop proportional-integral controller of the target motor to generate the final q-axis current command. The step of constructing a training dataset based on the motor digital twin model and the initial reinforcement learning model, and based on preset motor commands, includes: Drive the motor digital twin model according to the preset motor command with a preset step size; For a target preset step size, the initial state parameters of the motor digital twin model are obtained; the initial state parameters are the state parameters at the start of the target preset step size; the target preset step size is any preset step size among multiple preset step sizes; The motor compensation value is determined based on the initial state parameters and the initial reinforcement learning model. The motor compensation value is input into the motor digital twin model to determine the target state parameters and speed sequence; the target state parameters are the state parameters at the end of the target preset step length; the speed sequence is the multiple motor speeds collected within the target preset step length. The reward value is determined based on the rotational speed sequence and the reward function; The initial state parameters, motor compensation value, target state parameters, and reward value are used as training data corresponding to the target preset step size. A training dataset is constructed based on the training data corresponding to each preset step size.

2. The motor driving method according to claim 1, characterized in that, The initial reinforcement learning model includes an initial value network and an initial policy network; the backup reinforcement learning model includes a backup value network and a backup policy network; the process of adjusting the initial reinforcement learning model based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, and adjusting the backup reinforcement learning model based on the adjusted initial reinforcement learning model, until the initial reinforcement learning model meets preset conditions, to obtain the target reinforcement learning model, includes: Based on the training dataset, determine the target dataset; Based on the target dataset, the initial value network is adjusted according to the backup value network, the backup strategy network, and the initial value network; Based on the target dataset, the initial policy network is adjusted according to the initial policy network and the adjusted initial value network; The backup value network and backup policy network are adjusted based on the adjusted initial value network and the adjusted initial policy network. The initial value network and the initial policy network are iteratively adjusted until they meet preset conditions, thus obtaining the target reinforcement learning model.

3. The motor driving method according to claim 2, characterized in that, The target dataset includes multiple training data sets, which include: initial state parameters, motor compensation values, target state parameters, and reward values. The adjustment of the initial value network based on the target dataset, according to the backup value network, backup policy network, and initial value network, includes: Based on the initial state parameters and motor compensation values ​​corresponding to the target training data, the initial value is determined according to the initial value network; the target training data is any training data in the target dataset. Based on the target state parameters corresponding to the target training data, the predicted compensation value is determined according to the backup strategy network. Based on the target state parameters and predicted compensation values ​​corresponding to the target training data, the predicted value is determined according to the backup value network. The target value is determined based on the reward value corresponding to the target training data and the predicted value. Based on the initial value and the target value, the initial value network is adjusted using the backpropagation algorithm.

4. The motor driving method according to claim 3, characterized in that, The step of adjusting the initial policy network based on the target dataset, according to the initial policy network and the adjusted initial value network, includes: Based on the target dataset, the value gradient corresponding to each training data in the target dataset is calculated according to the adjusted initial value network. Based on the target dataset, the compensation gradient corresponding to each training data in the target dataset is calculated according to the initial policy network; The policy gradient is determined based on the value gradient and the compensation gradient corresponding to each training data. The initial policy network is adjusted according to the policy gradient.

5. The motor driving method according to claim 4, characterized in that, The step of adjusting the backup value network and backup policy network based on the adjusted initial value network and adjusted initial policy network includes: Obtain the preset update coefficients; The backup value network and backup policy network are adjusted based on the update coefficients, the adjusted initial value network, and the adjusted initial policy network.

6. The motor driving method according to claim 5, characterized in that, The iterative adjustment of the initial value network and the initial policy network until the initial value network and the initial policy network meet preset conditions, resulting in the target reinforcement learning model, includes: The initial value network and the initial policy network are iteratively adjusted until they converge, or until the initial value network and the initial policy network are iterated a preset number of times to obtain the target reinforcement learning model.

7. A motor drive device, characterized in that, The device includes: The first construction module is used to obtain the motor parameters of the target motor and construct a digital twin model of the motor based on the motor parameters; The initialization module is also used to initialize a pre-set initial reinforcement learning model and a backup reinforcement learning model; the initial reinforcement learning model adopts the Deep Deterministic Policy Gradient (DDPG) algorithm. The second construction module is also used to construct a training dataset based on the motor digital twin model and the initial reinforcement learning model, and on the basis of preset motor instructions. The adjustment module is used to adjust the initial reinforcement learning model based on the training dataset, the initial reinforcement learning model, and the backup reinforcement learning model, and to adjust the backup reinforcement learning model based on the adjusted initial reinforcement learning model, until the initial reinforcement learning model meets the preset conditions to obtain the target reinforcement learning model. The deployment module is used to deploy the target reinforcement learning model to the target motor, so that the target reinforcement learning model can obtain the state parameters of the target motor to generate actual compensation values, and superimpose the actual compensation values ​​with the output of the speed loop proportional-integral controller of the target motor to generate the final q-axis current command. The second construction module is further configured to drive the motor digital twin model according to the preset motor command with a preset step size; obtain the initial state parameters of the motor digital twin model for the target preset step size; the initial state parameters are the state parameters at the start of the target preset step size; the initial state parameters include: initial speed, initial q-axis current, initial d-axis current, initial rotor electrical angle, and initial q-axis output; the target preset step size is any preset step size among multiple preset step sizes; determine the motor compensation value according to the initial state parameters and the initial reinforcement learning model; and input the motor compensation value into the motor digital twin model. A twin model is used to determine the target state parameters and the speed sequence. The target state parameters are the state parameters at the end of the target preset step length. The target state parameters include: target speed, target q-axis current, target d-axis current, target rotor electrical angle, and target q-axis output. The speed sequence consists of multiple motor speeds collected within the target preset step length. A reward value is determined based on the speed sequence and a reward function. The initial state parameters, motor compensation value, target state parameters, and reward value are used as training data corresponding to the target preset step length. A training dataset is constructed based on the training data corresponding to each preset step length.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images