Multi-motor torque balance intelligent control method based on hierarchical reinforcement learning algorithm

Abstract

The invention discloses a multi-motor torque balance intelligent control method based on a hierarchical reinforcement learning algorithm, and aims to solve the problems that a traditional master-slave control structure of a multi-motor control system in a networking scene lacks an effective coupling structure and the torque is not uniform. According to the invention, the problem of wireless communication resource waste caused by independent design of a control system and a communication system is solved, and networking multi-motor control optimization is realized. The sampling period and the control input of the networked multi-motor control system are jointly optimized based on a hierarchical reinforcement learning algorithm, and the sampling period and the control input are adaptively output through hierarchical reinforcement learning agent joint training by taking minimization of total performance consisting of communication energy consumption and control errors as an optimization target. According to the method, the communication energy consumption is reduced, the torque balance problem is solved by optimizing the control input, and the control precision of the multi-motor system in the networking scene is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer control systems, specifically relates to a control algorithm for multiple permanent magnet synchronous motors, and particularly relates to an intelligent control method for torque balance of multiple motors based on a hierarchical reinforcement learning algorithm. Background Art

[0002] With the rapid development of industrial Internet of Things and intelligent manufacturing, networked multi-motor control systems are increasingly widely used in modern industrial automation and have become one of the key technologies to improve production efficiency and flexibility. Permanent magnet synchronous motors are widely used in fields such as automated production lines, robot control, and machine tool manufacturing due to their high efficiency, precision, and low maintenance characteristics. In complex industrial servo scenarios, a single permanent magnet synchronous motor often fails to meet the requirements. Therefore, multi-motor control systems are widely used in scenarios such as rail transit, industrial drive, and high-performance servo control, providing an efficient, robust, and cost-effective solution for servo systems. Modern multi-motor control systems usually rely on wireless communication technologies to enable interconnection of each motor, sensor, and actuator through a wireless network, supporting remote monitoring and collaborative control. However, although wireless communication brings flexibility, it also faces many challenges:

[0003] First, multiple permanent magnet synchronous motors are rigidly connected and operate in coordination, usually for large-scale assembly tasks. Whether in mechanical structure or electrical characteristics, the motors and gears may exhibit slight differences. Even the smallest deviation can cause uneven motor output. When a large disturbance or load change occurs, the distribution of the total load in the system becomes unbalanced, resulting in inconsistent torque outputs of each motor. This may affect performance and stability, and may damage workpieces or cause motor overload. Second, the network system and the control system are often coupled and interact with each other. Separated design of the control system and the communication network leads to low utilization efficiency of wireless resources. The sampling period is a key parameter affecting control performance and communication energy consumption. By dynamically adjusting the sampling period, a balance between communication energy consumption and control error can be achieved.

[0004] Therefore, how to design a multi-motor intelligent control algorithm for networked scenarios, which can jointly optimize the sampling period and control input of the networked multi-motor control system to minimize the total performance composed of communication energy consumption and control error, and solve the torque balance problem, is of great significance and quite challenging. Summary of the Invention

[0005] Object of the Invention: Aiming at the problems of wireless resource waste and the inability of traditional control structures to solve the torque balance problem in the networked scenario of the above-mentioned multi-permanent magnet synchronous motor servo control system, the present invention provides an intelligent control method for torque balance of multiple motors based on a hierarchical reinforcement learning algorithm.

[0006] Technical solution: A multi-motor torque balance intelligent control method based on a hierarchical reinforcement learning algorithm, characterized in that, for a multi-permanent magnet synchronous motor servo control system connected to a network, on the basis of a traditional master-slave control structure, the sampling period and control input of the multi-motor control system connected to the network are jointly optimized through a hierarchical reinforcement learning algorithm, with the goal of minimizing the total performance composed of communication energy consumption and control error. Through the joint training of hierarchical reinforcement learning agents, the sampling period and control input are adaptively output to achieve the non-linear control of the multi-motor system connected to the network. While reducing communication energy consumption, this method solves the torque balance problem by optimizing the control input and improves the control accuracy of the multi-motor system in a networked scenario.

[0007] The method includes the following steps:

[0008] (1) Construct a multi-motor control system for a networked scenario;

[0009] The method considers the behavior of a servo system driven by multi-permanent magnet synchronous motors in a networked scenario in a communication network, including transmission rate, transmission delay and its constraints, and total communication energy consumption; then constructs a test platform for the multi-motor control system, specifically including the following steps:

[0010] (11) Allocate a dedicated resource block (RB) to each motor node This allocation is implemented based on the framework of orthogonal frequency division multiple access (OFDMA) technology. The transmission rate of the state of the terminal motor node to the cloud can be calculated as:

[0011]

[0012] where B w is the bandwidth of each resource block; P i is the transmission power of each motor node ; h i is the channel gain between each motor node and the cloud, is the expected value of h i ; N0 is the unit noise power spectral density.

[0013] (12) The transmission delay between the motor node and the cloud can be calculated as:

[0014]

[0015] where Z(x i ) represents the number of bits required for each motor sensor to transmit the motor state x i to the cloud;

[0016] (13) The energy consumption of each motor node in time frame t can be described as:

[0017]

[0018] Where: S p (t) is the sampling period of the control loop within time frame t.

[0019] The total energy consumption of the entire multi-motor control system over all time frames can be expressed as:

[0020]

[0021] (2) Construct a coupled control scheme for multi-permanent magnet synchronous motors based on a master-slave control structure;

[0022] Furthermore, for a servo system driven by multi-permanent magnet synchronous motors with rigid connections, the method uses a master-slave control structure for basic control of speed synchronization to ensure the stability and robustness of the system operation. The specific steps include:

[0023] (21) The meshing gear transmission model of multi-motors with rigid connections is as shown on the right. This model consists of a load, multiple permanent magnet synchronous motors, gears, and shafts. The high-order non-linear model of a permanent magnet synchronous motor can be expressed through a synchronous d-q coordinate system, where the voltages U Figure 1 d U q and the currents I d I q are the basic parameters that make up the model. According to the Clarke and Park transformations, based on the FOC strategy, the phase variables of the permanent magnet synchronous motor are decomposed into magnetic field components and torque components and independently controlled. To accurately control the motor's magnetic field and torque, the rotor position and speed information of the motor are required for the transformation calculation between the phase variables in the stationary coordinate system and the relative rotational variables in the d-q coordinates. In the d-q coordinate system, the motor equation can be described as:

[0024]

[0025] Where: R s is the stator resistance; U d U q are the d-axis and q-axis voltages respectively; I d I q are the d-axis and q-axis currents respectively; L d L q are the d-axis and q-axis inductances respectively; is the d-axis magnetic flux of the permanent magnet; T e T L are the output torque and load torque respectively; B mis the bearing viscosity coefficient; J is the total moment of inertia of the motor and the load; ω m is the mechanical angular velocity of the rotor, and n p is the number of pairs of permanent magnets;

[0026] (22) Each surface-mounted permanent magnet synchronous motor is connected to a pinion through a drive shaft, and the other end of the pinion is connected to a large gear through an output shaft to drive the load to move. During the operation of the system, the motor transmits the driving torque from the pinion to the large gear, and then drives the load to move. Each gear connected to the permanent magnet synchronous motor is regarded as a whole, and the large gear and the load are also regarded as a whole. When the gear clearance is ignored, the rotor angle of each permanent magnet synchronous motor corresponds to the rotation angle of its respective pinion. According to the torque transmission relationship between the gears, the coupled motion equation of the transmission system is obtained:

[0027]

[0028] where, in the formula, J i and respectively represent the moment of inertia and viscous friction coefficient of each permanent magnet synchronous motor , ω g is the mechanical speed of the load, represents the total torque acting on the large gear, and the torque of each motor depends on the gear ratio r g , R g , stiffness and damping coefficient The rotor angle θ i of each permanent magnet synchronous motor and the rotation angle θ g of the large gear.

[0029] (23) The cloud server includes an administrator and a controller to remotely manage multiple permanent magnet synchronous motors. Each motor is equipped with a wireless sensor to collect its status output and transmit it to the cloud server. The time scale of the system operation is defined as . At each time slot t, the cloud server first determines the sampling period S p (t) and the voltage compensation value from t to t + 1 During the time interval from t to t + 1, the cloud server receives the multi-motor state feedback signals within the sampling period S p (t), calculates the voltage applied to each motor and superimposes the voltage compensation value. The determined voltage is sent back to each terminal motor through a wireless link to generate an appropriate torque. The multi-permanent magnet synchronous motors transmit the driving torque from the pinion to the large gear, and the large gear in turn drives the terminal load to move. The actual control input of each permanent magnet synchronous motor can be expressed as:

[0030]

[0031] (24) In the proposed rigidly connected multi-permanent magnet synchronous motor drive system, both speed tracking performance and torque synchronization are crucial considerations. For motor node i, the control error function within each time slot t is defined as:

[0032]

[0033] where ω * (t) is the reference signal, denotes the q-axis current reference value in the time frame t.

[0034] For all motors, the control of the total control cost function C of the entire multi-motor system over the entire time frame can be described as:

[0035]

[0036] (3) Construct a joint optimization problem for multi-motor control in the networked scenario;

[0037] Step (3) constructs a joint optimization problem for the multi-permanent magnet synchronous motor control system in the networked scenario and decouples it into two sub-problems. The specific steps include:

[0038] (31) For the multi-permanent magnet synchronous motor control system in the networked scenario, the long-term total performance of the system is defined as:

[0039]

[0040] where α is the weight coefficient.

[0041] A joint optimization problem for the multi-motor control system with torque synchronization is proposed, and its mathematical expression is:

[0042]

[0043] The constraint conditions are:

[0044]

[0045]

[0046] (32) Split the joint optimization problem into two sub-problems. The upper-level sub-problem focuses on the communication energy consumption problem, and the lower-level sub-problem focuses on the torque synchronization problem:

[0047]

[0048] The constraint conditions are:

[0049]

[0050] The constraints are as follows:

[0051]

[0052]

[0053] (4) The hierarchical reinforcement learning algorithm is adopted to jointly optimize the sampling period and control input of the networked multi-motor control system, so as to reduce the communication energy consumption. At the same time, the torque balance problem is solved by optimizing the control input, and the control accuracy of the multi-motor system in the networked scenario is improved.

[0054] In step (4) for the joint optimization problem, the sampling period and control input are jointly optimized based on the hierarchical reinforcement learning algorithm. Specifically, the DQN algorithm and TD3 algorithm are used. With the goal of minimizing the total performance composed of communication energy consumption and control error, by jointly training the upper and lower layer agents in the networked multi-motor control system environment, the knowledge of the servo motor control system is learned, and then the sampling period and control input are adaptively optimized to achieve the non-linear control of the networked multi-motor system. While reducing the communication energy consumption, the torque balance problem is solved by optimizing the control input, and the control accuracy of the multi-motor system in the networked scenario is improved. The specific steps of step (4) include:

[0055] (41) Build the upper layer agent network; the upper layer agent learns how to optimize and adjust the system sampling period through the DQN algorithm to minimize the total energy consumption and error.

[0056] (42) Build the lower layer agent network. The lower layer agent learns to optimize the control input for each motor node through the TD3 algorithm to achieve precise torque control and dynamic response.

[0057] (43) Jointly optimize the upper and lower layer agents. Through the co-training method in hierarchical reinforcement learning, the upper and lower layer agents can share information and jointly adjust their decisions to achieve the global optimization of the system.

[0058] On the other hand, the present invention provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a multi-motor torque balance intelligent control method based on the hierarchical reinforcement learning algorithm as described in the present invention.

[0059] A computer storage medium stores a computer program, and when the computer program is executed by a computer, it implements the multi-motor torque balance intelligent control method based on the hierarchical reinforcement learning algorithm as described above.

[0060] Beneficial effects: Aiming at the problems of wireless resource waste and the inability of traditional control structures to solve the torque balance problem in the scenario of a networked multi-permanent magnet synchronous motor servo control system, the method described in the present invention proposes an intelligent multi-motor torque balance control method based on a hierarchical reinforcement learning algorithm. For the networked multi-permanent magnet synchronous motor servo control system, based on the traditional master-slave control structure, the hierarchical reinforcement learning algorithm is used to jointly optimize the sampling period and control input of the networked multi-motor control system, with the goal of minimizing the total performance composed of communication energy consumption and control error. Through the joint training of hierarchical reinforcement learning agents, the sampling period and control input are adaptively output to achieve the non-linear control of the networked multi-motor system. While reducing communication energy consumption, this method solves the torque balance problem by optimizing the control input and improves the control accuracy of the multi-motor system in the networked scenario. Description of the Drawings

[0061] Figure 1 is the overall process model structure diagram of the multi-permanent magnet synchronous motor system in the present invention;

[0062] Figure 2 is the comparison diagram of the total performance of the system under different method controls in the present invention;

[0063] Figure 3 is the current diagram of each motor based on master-slave control (Figure 3(a)), the current diagram of each motor based on coordinated control (Figure 3(b)), and the current diagram of each motor based on the hierarchical reinforcement learning algorithm control (Figure 3(c)) in the present invention. Detailed Implementation Modes

[0064] In order to elaborate in detail the technical solutions disclosed in the present invention, the present invention will be further described below in conjunction with the drawings and specific implementation modes.

[0065] Firstly, the key problems to be solved by the method described in the present invention are how to achieve the non-linear control of the multi-motor system based on the hierarchical reinforcement learning algorithm in the networked scenario of the multi-permanent magnet synchronous motor servo control system, which has the problems of wireless resource waste and the inability of traditional control structures to solve the torque balance problem, while reducing communication energy consumption, solving the torque balance problem by optimizing the control input, and improving the control accuracy of the multi-motor system in the networked scenario.

[0066] The present invention realizes the optimization of multi-motor control based on the hierarchical reinforcement learning algorithm. For the networked multi-permanent magnet synchronous motor servo control system, based on the traditional master-slave control structure, the hierarchical reinforcement learning algorithm is used to jointly optimize the sampling period and control input of the networked multi-motor control system, with the goal of minimizing the total performance composed of communication energy consumption and control error. Through the joint training of hierarchical reinforcement learning agents, the sampling period and control input are adaptively output. The overall control process provided by this method is as Figure 1 shown.

[0067] A multi-motor torque balancing intelligent control method based on a hierarchical reinforcement learning algorithm, comprising the following steps:

[0068] Step 1: Construct a test platform for a multi-motor control system for a networking scenario.

[0069] In the test platform for the multi-motor control system for a networking scenario proposed in the present invention, a dedicated resource block (RB) is allocated to each motor node This allocation is implemented based on the framework of orthogonal frequency division multiple access (OFDMA) technology. This method ensures that each motor node has its own unique communication frequency band, thereby minimizing interference and optimizing bandwidth utilization. The transmission rate of the status of the terminal motor node to the cloud can be calculated as:

[0070]

[0071] where B w is the bandwidth of each resource block; P i is the transmission power of each motor node ; h i is the channel gain between each motor node and the cloud, is the expected value of h i ; and N0 is the unit noise power spectral density.

[0072] (12) The transmission delay between the motor node and the cloud can be calculated as:

[0073]

[0074] where Z(x i ) represents the number of bits required for each motor sensor to transmit the motor state x i to the cloud; in this system, the transmission delay l i can be regarded as part of the system disturbance.

[0075]

[0076] where: S p (t) is the sampling period of the control loop within the time frame t;

[0077] The total energy consumption of the entire multi-motor control system within all time frames can be expressed as:

[0078]

[0079] Step 2: Construct a multi-permanent magnet synchronous motor coupling control scheme based on a master-slave control structure;

[0080] In the present invention, the high-order nonlinear model of the permanent magnet synchronous motor can be expressed in the synchronous d-q coordinate system, where the voltage U d , U q and the current I d , I q are the basic parameters that make up the model. According to the Clarke and Park transformations, based on the FOC strategy, the phase variables of the permanent magnet synchronous motor are decomposed into magnetic field components and torque components and independently controlled. To accurately control the motor's magnetic field and torque, the rotor position and speed information of the motor are required for the transformation calculation between the phase variables in the stationary coordinate system and the relative rotation variables of the d-q coordinates. In the d-q coordinate system, the motor equation can be described as:

[0081]

[0082] where: R s is the stator resistance; U d , U q are the d-axis and q-axis voltages respectively; I d , I q are the d-axis and q-axis currents respectively; L d , L q are the d-axis and q-axis inductances respectively; is the d-axis magnetic flux of the permanent magnet; T e , T L are the output torque and the load torque respectively; B m is the bearing viscous coefficient; J is the total moment of inertia of the motor and the load; ω m is the rotor mechanical angular velocity, n p is the number of pairs of permanent magnets;

[0083] Each surface-mounted permanent magnet synchronous motor is connected to a pinion through a drive shaft, and the other end of the pinion is connected to a large gear through an output shaft to drive the load to move. During the operation of the system, the motor transmits the driving torque from the pinion to the large gear, and then drives the load to move. Each gear connected to the permanent magnet synchronous motor is regarded as a whole, and the large gear and the load are also regarded as a whole. When the gear clearance is ignored, the rotor angle of each permanent magnet synchronous motor corresponds to the rotation angle of its respective pinion. According to the torque transmission relationship between the gears, the coupled motion equation of the transmission system is obtained:

[0084]

[0085] where, in the formula, J i and respectively represent each permanent magnet synchronous motor The moment of inertia and viscous friction coefficient, ω g is the mechanical rotational speed of the load, represents the total torque acting on the large gear, the torque of each motor depends on the gear ratio r g 、R g , stiffness and damping coefficient The rotor angle θ of each permanent magnet synchronous motor i and the rotation angle θ of the large gear g .

[0086] From the perspective of the motion of the permanent magnet synchronous motor, the type, parameters and material properties of the motor will affect the electromagnetic torque T of the system e synchronization. In terms of mechanical motion, frictional losses of the gears, the stiffness coefficient and damping coefficient of the motor shaft or wear will cause the load torque T of each motor to g vary, resulting in uneven distribution of the load torque T of each motor L , thus leading to uneven electromagnetic torque T e . The electromagnetic torque T e The recovery of synchronization takes a long time, which may lead to potential safety hazards of overload of the permanent magnet synchronous motor.

[0087] According to the above formula, it can be seen that motor torque synchronization is equivalent to q-axis current synchronization. However, the master-slave coordination control structure in the cloud lacks a coupling mechanism, which means that torque deviation will not be reflected to the current regulator in a timely manner. Therefore, system disturbances are likely to cause deterioration of torque synchronization.

[0088] The cloud server includes an administrator and a controller, and remotely manages multiple permanent magnet synchronous motors. Each motor is equipped with a wireless sensor to collect its status output and transmit it to the cloud server. The time scale of the system operation is defined as In each time slot t, the cloud server first determines the sampling period S p (t) and the voltage compensation value from t to t + 1 During the time interval from t to t + 1, the cloud server receives the multi-motor state feedback signals within the sampling period S p (t), calculates the voltage applied to each motor and superimposes the voltage compensation value. The decision of the voltage is sent back to each terminal motor through a wireless link to generate an appropriate torque. The multi-permanent magnet synchronous motors transmit the driving torque from the pinion to the large gear, and the large gear in turn drives the terminal load to move. The actual control input of each permanent magnet synchronous motor can be expressed as:

[0089]

[0090] In the proposed rigidly connected multi-permanent magnet synchronous motor drive system, speed tracking performance and torque synchronization are both crucial considerations. For motor node i, the control error function within each time slot t is defined as:

[0091]

[0092] where ω * (t) is the reference signal, representing the q-axis current reference value at time frame t.

[0093] For all motors, the overall control cost function C of the entire multi-motor system over the entire time frame can be described as:

[0094]

[0095] Step 3: Construct the joint optimization problem of multi-motor control for the networked scenario;

[0096] For the multi-permanent magnet synchronous motor control system in the networked scenario, the long-term overall performance of the system is defined as:

[0097]

[0098] The joint optimization problem of the multi-motor control system with torque synchronization is proposed, and its mathematical expression is:

[0099]

[0100] The constraint conditions are:

[0101]

[0102]

[0103] Furthermore, the joint optimization problem is split into two sub-problems. Considering the influence of the decision variable sampling period S p on communication and control, the upper-layer problem aims to jointly optimize the communication energy consumption E and control error C of the networked multi-motor control system by determining S p from a global perspective, and its mathematical expression is:

[0104]

[0105] The constraint conditions are:

[0106] Considering the torque balance problem of the terminal multi-motor control system, the lower-layer control problem aims to optimize the control error C by determining the q-axis voltage tuning value applied to each motor, and its mathematical expression is:

[0107]

[0108] The constraint conditions are as follows:

[0109]

[0110]

[0111] Step 4: Construct a multi - motor control optimization scheme based on the hierarchical reinforcement learning algorithm

[0112] In view of the joint optimization problem constructed in Step 3, the present invention proposes to jointly optimize the sampling period and control input based on the hierarchical reinforcement learning algorithm. Specifically, the DQN algorithm and TD3 algorithm are used. With the goal of minimizing the total performance composed of communication energy consumption and control error, by jointly training the upper and lower layer agents in the environment of a networked multi - motor control system, knowledge of the servo - motor control system is learned, and then the sampling period and control input are adaptively optimized to achieve the non - linear control of the networked multi - motor system. While reducing communication energy consumption, the torque balance problem is solved by optimizing the control input, and the control accuracy of the multi - motor system in the networked scenario is improved.

[0113] Construct the upper - layer agent network. The upper - layer agent is trained using the DQN algorithm, and the action predicted by the agent is determined as the system sampling period. At the same time, the state input of the agent is determined as the signal fed back by the multi - motor control system, including the reference speed signal ω * , the current speed ω of each motor i , the reference q - axis current I q * and the q - axis current of each motor

[0114] To balance the communication energy consumption and control error, the total communication energy consumption and total control error of the multi - motor are used as the reward value for each time step:

[0115]

[0116] For the state s u Use the target Q - network function to enable the agent to output the tuning value a * .

[0117] The upper - layer agent guides the decision - making of the lower - layer agent through decision - making actions. At the same time, the feedback information and control performance of the lower - layer agent affect the subsequent state and serve as an important input for adjusting the upper - layer strategy. This co - training mechanism ensures the harmonious operation of the two levels in a dynamic environment, enabling the multi - motor control system to maintain the best performance under different conditions.

[0118] Construct the lower-layer agent network. The lower-layer agents are trained using TD3, and the actions predicted by the agents are determined as the tuning values of the q-axis voltage output by the PID current loop. At the same time, the state input of the agents is determined as the actions of the upper-layer agents and the reference speed signal ω * , the current speed ω of each motor i , the reference q-axis current I q * and the q-axis current of each motor

[0119] To reduce the speed error and torque error of each motor, the negative of the sum of the absolute values of their differences is used as the reward value for each time step:

[0120]

[0121] For the state s l , use the optimized policy function so that the agent can output the tuning value For this purpose, two Q networks need to be established to evaluate the value of the output of the policy network, and the evaluation value is used to optimize the gradient of the actor network.

[0122] Then, jointly optimize the upper and lower-layer agents. Specifically, first initialize the networked control multi-motor system environment and generate the reference commands for the motor speeds, and initialize the parameters of the upper-layer DQN network and the parameters of the lower-layer TD3. Subsequently, in each time frame t ∈ T, the upper-layer agent observes the system state s u (t) through sensor feedback and makes an action decision a u (t), and determines the system sampling period S p (t) within the time frame t. The lower-layer agent observes its state s l (t) = s u (t), a u (t) and makes a decision a l (t), which represents the tuning value of the q-axis voltage of each motor. During the time slot t' ∈ [t, t + 1], the cloud controller receives the sensor feedback signal and controls the terminal multi-motor system with the sampling period of a u (t). At the same time, keep the tuning value of the q-axis voltage as a l (t). After the action is executed, the upper and lower-layer agents observe the new state and reward, and then store the resulting state-action-reward tuple in the replay buffer. When the number of samples in the upper or lower buffer reaches the maximum capacity, the samples are drawn to update the network. The above process is repeated until the hierarchical reinforcement learning training is completed. Through the co-training method in hierarchical reinforcement learning, the upper and lower-layer agents can share information and jointly adjust their respective decisions to achieve the global optimization of the system.

[0123] To comprehensively verify the multi-motor torque balancing intelligent control method based on the hierarchical reinforcement learning algorithm proposed in the present invention, the following two metrics are used to evaluate the performance: the overall performance experiment and the torque balancing experiment.

[0124] Combined with Figure 2 As shown, to examine the balancing performance of the multi-motor torque balancing intelligent control method based on the hierarchical reinforcement learning algorithm, in terms of the overall system performance, the proposed intelligent control method based on hierarchical reinforcement learning is superior to the master-slave control with a sampling period of 1e-4 and the master-slave control with a sampling period of 2e-4. This is because the intelligent control method based on the hierarchical reinforcement learning algorithm continuously updates its strategy according to the real-time system feedback, enabling it to make optimal decisions under uncertain and time-varying conditions. By adaptively adjusting the sampling period and voltage in response to the real-time state of the system, the dynamic balance of energy consumption and control error is effectively achieved. This adaptive method effectively minimizes the overall performance index, reduces the energy cost, and maintains precise control at the same time.

[0125] Combined with Figure 3, the intelligent control method based on the hierarchical reinforcement learning algorithm proposed in the present invention is superior to the master-slave control and the cross-coupling control in terms of torque synchronization performance. When the external load disturbance T L occurs at t = 20 s, the load distribution among the four motors becomes unbalanced, manifested as uneven current distribution among the motors. All control algorithms require a certain amount of time to restore torque synchronization. As can be seen from Figure 3(a), i) for the master-slave control algorithm, the maximum current deviation among the four motors is 0.14 after the load disturbance, and it takes about 1 second to achieve current synchronization (equivalent to torque synchronization) at t = 21 s; ii) in Figure 3(b), the performance of the cross-coupling control algorithm is improved, and the maximum current deviation is reduced to 0.08, and synchronization is achieved at t = 21 s with a similar recovery time of 1 s; and iii) in Figure 3(c), the intelligent control method based on the hierarchical reinforcement learning algorithm is superior to the two traditional algorithms, with a maximum current deviation of only 0.06, and synchronization is achieved within 0.8 s at t = 20.8 s. This is because the intelligent control method based on the hierarchical reinforcement learning algorithm has the adaptability to handle complex nonlinear systems. It can dynamically adjust the control strategy according to the real-time state of the system and quickly adapt to dynamic scenarios such as load disturbances. In addition, the intelligent control method based on the hierarchical reinforcement learning algorithm takes into account the nonlinear and strong coupling characteristics of the multi-motor system and achieves precise torque synchronization control.

Claims

1. A multi-motor torque balancing intelligent control method based on hierarchical reinforcement learning algorithm, characterized in that: This method targets the master-slave control structure in the networked multi-permanent magnet synchronous motor servo control system. It uses a hierarchical reinforcement learning algorithm to jointly optimize the sampling period and control input of the networked multi-motor control system, with the optimization goal of minimizing the total performance consisting of communication energy consumption and control error. Through the joint training of hierarchical reinforcement learning agents, the sampling period and control input are adaptively output to achieve nonlinear control of the networked multi-motor system. The method comprises the following steps: (1) Build a multi-motor control system for networking scenarios, allocate resource blocks to each motor node through orthogonal frequency division multiple access technology, and calculate the transmission rate, transmission delay and energy consumption of the terminal motor node state transmission to the cloud in time frame t; (2) Construct a multi-PMSM coupling control scheme based on a master-slave control structure, including constructing a high-order nonlinear model of the PMSM, remotely managing multiple PMSMs through a cloud server, and each motor is equipped with a wireless sensor to collect its state output and transmit it to the cloud server; for a servo system driven by multiple PMSMs with rigid connections, a master-slave control structure is used to perform basic control of speed synchronization to ensure the stability and robustness of the system operation; (3) Construct a joint optimization problem for the multi-PMSM control system in a networked scenario, including: (31) For the multi-permanent magnet synchronous motor control system in the networking scenario, the long-term overall performance of the system is defined as: Where α is the weight coefficient; The joint optimization problem is transformed into a joint optimization problem of a multi-motor control system with torque synchronization, and its mathematical expression is: The constraints are: (32) The joint optimization problem is divided into two sub-problems. The upper sub-problem focuses on the communication energy consumption problem, and the lower sub-problem focuses on the torque balance problem: The constraints are: The constraints are: (4) Solving the joint optimization problem described in step (3), a hierarchical reinforcement learning algorithm is used to jointly optimize the sampling period and control input of the networked multi-motor control system to achieve torque balance and reduce communication energy consumption.

2. The multi-motor torque balancing intelligent control method based on hierarchical reinforcement learning algorithm according to claim 1 is characterized in that: The specific steps of step (1) are: (11) Based on orthogonal frequency division multiple access technology, each motor node Allocating dedicated resource blocks, the transmission rate of the state of terminal motor node i to the cloud is calculated as: Among them, B w is the bandwidth of each resource block; P i Each motor node The transmission power; h i is the channel gain between each motor node i and the cloud server, Yes i The expected value of; N0 is the unit noise power spectral density; (12) Motor node The transmission delay between the cloud server and the server is calculated as: Where Z(x i ) indicates that each motor sensor i needs to transmit the motor state x i The number of bits required to reach the cloud; (13) The energy consumption of each motor node in time frame t is described as: Where: S p (t) is the sampling period of the control loop in time frame t; The total energy consumption of the entire multi-motor control system in all time frames is expressed as:

3. The multi-motor torque balancing intelligent control method based on hierarchical reinforcement learning algorithm according to claim 1 is characterized in that: Step (2) specifically includes: (21) The rigidly connected multi-motor meshing gear transmission model consists of a load, a permanent magnet synchronous motor, a gear, and a shaft. The high-order nonlinear model of the permanent magnet synchronous motor is expressed by a synchronous dq coordinate system, where the voltage U d , U q and current I d ,I q It is the basic parameter of the model. According to the transformation of Clarke and Park, the phase variable of the permanent magnet synchronous motor is decomposed into magnetic field component and torque component based on the FOC strategy, and they are controlled independently. The precise control of the motor magnetic field and torque requires the motor's rotor position and speed information to be used for the transformation calculation between the phase variable in the stationary coordinate system and the relative rotation variable of the dq coordinate. In the dq coordinate system, the motor equation is described as: Where: R s is the stator resistance; U d , U q are d-axis and q-axis voltages respectively; I d , I q are d-axis and q-axis current respectively; L d , L q They are d-axis and q-axis inductance respectively; is the d-axis magnetic flux of the permanent magnet; T e , T L are the output torque and load torque respectively; B m is the bearing viscosity coefficient; J is the total moment of inertia of the motor and load; ω m is the rotor mechanical angular velocity, n p is the number of permanent magnet pairs; (22) Each surface-mounted permanent magnet synchronous motor is connected to a pinion gear through a drive shaft, and the other end of the pinion gear is connected to a large gear through an output shaft, thereby driving the load to move; during the operation of the system, the motor transmits the driving torque from the pinion gear to the large gear, thereby driving the load to move; each gear connected to the permanent magnet synchronous motor is regarded as a whole, and the large gear and the load are also regarded as a whole. When the gear clearance is ignored, the rotor angle of each permanent magnet synchronous motor corresponds to the rotation angle of its respective pinion gear; According to the torque transmission relationship between gears, the coupled motion equation of the transmission system is obtained: Among them, Ji and Respectively represent the moment of inertia and viscous friction coefficient of each permanent magnet synchronous motor i, ω g is the mechanical speed of the load, Represents the total torque acting on the large gear, the torque of each motor Depends on gear ratio rg, Rg, stiffness and the damping coefficient The rotor angle θi of each permanent magnet synchronous motor and the rotation angle θg of the large gear; (23) Each motor is equipped with a wireless sensor to collect its status output and transmit it to a cloud server, which contains an administrator and a controller to remotely manage each permanent magnet synchronous motor; The time scale of system operation is defined as T∈N + , in each time slot t, the cloud server first determines the sampling period S p (t) and the voltage compensation value from t to t+1 In the time interval from t to t+1, the cloud server receives the multi-motor state feedback signal within the sampling period Sp(t), calculates the voltage applied to each motor and superimposes the voltage compensation value. The voltage decision is sent back to each terminal motor through a wireless link to generate appropriate torque; the multi-permanent magnet synchronous motor transmits the driving torque from the small gear to the large gear, and the large gear in turn drives the terminal load to move; The actual control input of each permanent magnet synchronous motor is expressed as: (24) For motor node i, the control error function in each time slot t is defined as: Where ω*(t) is the reference signal, represents the q-axis current reference value at time frame t; For all motors, the total control cost function C of the entire multi-motor system in the entire time frame can be described as:

4. The multi-motor torque balancing intelligent control method based on hierarchical reinforcement learning algorithm according to claim 1 is characterized in that: Step (4) specifically includes: (41) Constructing an upper-layer intelligent agent network; the upper-layer intelligent agent learns to optimize and adjust the system sampling period through the DQN algorithm to minimize the total energy consumption and control error; (42) constructing a lower-level intelligent agent network, wherein the lower-level intelligent agent optimizes the control input of each motor node through TD3 algorithm learning to achieve accurate torque control and dynamic response; (43) Jointly optimize the upper and lower-level intelligent agents. Through the collaborative training method in hierarchical reinforcement learning, the upper and lower-level intelligent agents can share information and coordinately adjust their respective decisions to achieve global optimization of the system.

5. A computing device, characterized in that: It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a multi-motor torque balancing intelligent control method based on a hierarchical reinforcement learning algorithm as described in any one of claims 1 to 4.

6. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, a multi-motor torque balancing intelligent control method based on a hierarchical reinforcement learning algorithm as described in any one of claims 1 to 4 is implemented.

Images