Motor control method and system of intelligent training equipment based on reinforcement learning

Abstract

The invention provides a motor control method and system for intelligent training equipment based on reinforcement learning, and the method comprises the steps: constructing a motion current model based on motor operation data and motor control current; obtaining a motion current model of each training mode based on the training modes in combination with the motion current models; constructing a state space model, an action space model and a reward model; based on the state space model, the action space model and the reward model, optimizing the motion current model of each training mode to obtain an optimal motion current model of each training mode; and obtaining real-time motor operation data, and obtaining real-time optimal motor control current of each training mode in combination with the optimal motion current model of each training mode. According to the method, the special strategy of the training mode is generated for different training modes, online adjustment is performed on the model by utilizing reinforcement learning continuous optimization, and the model can quickly adapt to a new user or a new training mode and perform overspeed protection.

Description

Technical Field

[0001] This invention relates to the field of motor control, and specifically to a motor control method and system for intelligent training equipment based on reinforcement learning. Background Technology

[0002] Traditional training equipment often uses weighted blocks, which is inconvenient to use. Applying electric motors to training equipment to provide resistance is a highly efficient, intelligent, and adjustable solution, especially suitable for modern smart fitness equipment (such as electric resistance bikes, rowing machines, and strength training machines). In this equipment, the motor operates in generator mode, producing a reverse electromagnetic torque, or counter-traction force. The force generated by the exerciser's movements is applied to the motor as input torque, overcoming the reverse electromagnetic torque and thus completing the workout.

[0003] However, existing smart fitness equipment relies on fixed current and load mapping relationships for control, which cannot adapt to the dynamic training needs of users (such as real-time changes in exertion habits and fatigue states). This leads to a mismatch between exercise resistance / assistance and the user's actual ability, affecting training effectiveness and safety. For example, if the motor response is lagging during strength training, it may cause muscle strain. Different training modes (such as endurance and power modes) require pre-configuration of motor parameters, lacking adaptive adjustments based on the user's real-time exercise data. For example, for the same user in the treadmill's incline and flat running modes, the motor current switching only relies on preset thresholds, failing to learn the user's individual exercise characteristics. Moreover, existing technologies are limited to offline modeling, failing to combine the online interactive decision-making capabilities of reinforcement learning with motor control. Offline-trained models struggle to cover the diverse exercise patterns of users, requiring frequent manual parameter tuning. Traditional feedback control only addresses instantaneous adjustments and lacks long-term optimization. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a motor control method and system for intelligent training equipment based on reinforcement learning.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A motor control method for an intelligent training device based on reinforcement learning, wherein the intelligent training device provides multiple training modes, includes the following steps: A motion current model is constructed based on motor operating data and motor control current; wherein, the motor operating data includes at least motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping, and motor moment of inertia; Based on the training mode and combined with the motion current model, the motion current model for each training mode is obtained; When the motor angular velocity is greater than the first preset angular velocity threshold, the motion current model of each training mode is adjusted according to the difference between the motor angular velocity and the first preset angular velocity threshold. A state-space model is constructed based on motor operating data; an action-space model is constructed based on motor control current; and a reward model is constructed based on whether motor torque and motor angular velocity are within a reasonable range. Based on the state space model, action space model, and reward model, the motion current model for each training mode is optimized to obtain the optimal motion current model for each training mode. Real-time motor operation data is acquired. Based on the target training mode and the real-time motor operation data, and combined with the optimal motion current model of the target training mode, the real-time optimal motor control current of the target training mode is obtained.

[0006] As one possible implementation method, the motion current model is represented as follows:

[0007] in, Indicates the motor control current. Indicates motor stiffness. Indicates motor damping, Indicates the moment of inertia of the motor. Indicates the angular position of the motor. Indicates the angular velocity of the motor. Indicates the angular acceleration of the motor. These represent the set angular position, set angular velocity, and set angular acceleration, respectively. This represents the motor torque constant term. This represents the motor torque constant.

[0008] As one possible implementation method, the training mode includes at least a fluid resistance mode, a centripetal isotensile mode, an eccentric isotensile mode, and an elastic mode; In the fluid resistance mode, the motor stiffness and the motor moment of inertia are zero, and the motor damping is negative. In the centripetal isotensile mode, the motor stiffness, the motor damping, and the motor moment of inertia are all zero, and the motor angular velocity is a positive value. In the centrifugal isotensile mode, the motor stiffness, the motor damping, and the motor moment of inertia are all zero, and the motor angular velocity is negative. In the elastic mode, the motor damping and the motor moment of inertia are zero, and the motor stiffness is negative.

[0009] As one possible implementation, the motion current model of each training mode is adjusted based on the difference between the motor angular velocity and a first preset angular velocity threshold. The adjusted motion current model of each training mode is expressed as follows:

[0010] in, This indicates the adjusted motor control current. Indicates proportional gain. Indicates the angular velocity of the motor. This indicates the first preset angular velocity threshold. This represents the motor torque constant.

[0011] As one possible implementation method, the method of constructing a reward model based on whether the motor torque and motor angular velocity are within a reasonable range includes the following steps: Based on the motor operating data, the predicted value of the motor torque is obtained; The bonus value for the force application position is obtained based on whether the predicted value of the motor torque is equal to the ideal value of the motor torque. The overspeed bonus value is obtained based on whether the motor angular velocity exceeds the maximum motor angular velocity. The total reward value is obtained by weighted summation of the power position reward value and the speed reward value.

[0012] As one possible implementation, the optimization of the motion current model for each training mode based on the state space model, action space model, and reward model to obtain the optimal motion current model for each training mode includes the following steps: Construct a policy model, which includes a policy function, a policy gradient model, a cumulative reward model, and a policy update model; Obtain the initial state as the current step state, and based on the current step state, combine it with the policy function to obtain the action to be taken; Based on the current step state and the action taken, and combined with the reward model, the reward value for the current step is obtained; Based on the action taken, obtain the next state and update the step count; Until the round termination condition is met, based on the reward value of each step and combined with the cumulative reward model, the cumulative reward value of each step in the current round is obtained, and thus the cumulative reward value of the current round is obtained; based on the reward value of each step and combined with the policy gradient model, the policy gradient of the current round is obtained; based on the policy gradient of the current round and combined with the policy update model, the policy parameters are updated. The training continues until the training termination condition is met, at which point the optimal motor control current, i.e., the optimal motion current model, is obtained for each state space. The training termination condition includes the most recent average cumulative reward value for rounds being greater than a preset reward threshold or the training reaching the maximum number of rounds.

[0013] A motor control system for intelligent training equipment based on reinforcement learning, capable of implementing the motor control method for intelligent training equipment based on reinforcement learning as described above, includes a basic model module, a training mode model module, an overspeed adjustment module, an inference model construction module, an inference model training module, and a real-time optimization module. The basic model module constructs a motion current model based on motor operating data and motor control current; wherein, the motor operating data includes at least motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping, and motor moment of inertia; The training mode model module, based on the training mode and combined with the motion current model, obtains the motion current model for each training mode; The overspeed adjustment module adjusts the motion current model of each training mode according to the difference between the motor angular velocity and the first preset angular velocity threshold when the motor angular velocity is greater than the first preset angular velocity threshold. The reasoning model construction module constructs a state space model based on motor operating data; constructs an action space model based on motor control current; and constructs a reward model based on whether the motor torque and motor angular velocity are within a reasonable range. The inference model training module optimizes the motion current model for each training mode based on the state space model, action space model, and reward model to obtain the optimal motion current model for each training mode. The real-time optimal module acquires real-time motor operation data, and based on the target training mode and the real-time motor operation data, combines the optimal motion current model of the target training mode to obtain the real-time optimal motor control current of the target training mode.

[0014] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the preceding methods.

[0015] A motor control device for an intelligent training instrument based on reinforcement learning includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the method described in any one of the above descriptions.

[0016] This invention, employing the above technical solutions, achieves significant technical advantages: By training a reinforcement learning model online and combining real-time motor operating data (motor angular position, angular velocity, angular acceleration, stiffness, damping, and moment of inertia) with the training mode, it dynamically generates the optimal control current. Simultaneously, this invention also incorporates overspeed protection for hands-free operation under different training modes. For each training mode, the model generates a mode-specific strategy based on the motion current model and continuously optimizes it using reinforcement learning. Online model adjustments allow for rapid adaptation to new users or training modes and enable overspeed protection, reducing manual calibration costs and improving safety. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the motor control method for intelligent training equipment based on reinforcement learning according to the present invention. Figure 2 This is a schematic diagram of a module of an embodiment of the motor control system for an intelligent training device based on reinforcement learning according to the present invention. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to embodiments. These embodiments are illustrative of the invention and the invention is not limited thereto. Unless otherwise specified, the features in the following embodiments can be combined with each other.

[0020] A motor control method for intelligent training equipment based on reinforcement learning, such as... Figure 1 As shown, the intelligent training equipment can provide multiple training modes, including the following steps: S100: Construct a motion current model based on motor operating data and motor control current; wherein, the motor operating data includes at least motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping, and motor moment of inertia; S200: Based on the training mode and combined with the motion current model, the motion current model of each training mode is obtained; S300: When the motor angular velocity is greater than the first preset angular velocity threshold, the motion current model of each training mode is adjusted according to the difference between the motor angular velocity and the first preset angular velocity threshold. S400: Construct a state-space model based on motor operating data; construct an action-space model based on motor control current; construct a reward model based on whether motor torque and motor angular velocity are within a reasonable range; S500: Based on the state space model, action space model and reward model, the motion current model of each training mode is optimized to obtain the optimal motion current model of each training mode. Specifically, based on the reward value obtained by the reward model, the motor control current in each state space is adjusted to adjust the motion current model until the optimization conditions are met, and the optimal motor control current in each state space is obtained, which is the optimal motion current model. S600: Acquires real-time motor operation data, and based on the target training mode and the real-time motor operation data, combines the optimal motion current model of the target training mode to obtain the real-time optimal motor control current of the target training mode.

[0021] The aforementioned intelligent training equipment offers multiple training modes and is a smart personal training device with a motor as its core component, typically consisting of two motors. The motors provide resistance during training and operate in generator mode. Therefore, during training, over 80% of the kinetic energy can be recovered through a supercapacitor. Furthermore, in fluid resistance mode, the device not only consumes no electricity but can also power external devices. The motors are preferably high-torque axial flux motors, which possess rapid current regulation characteristics and millisecond-level response, ensuring imperceptible delays in mode switching.

[0022] Intelligent training equipment allows users to set specific training methods, such as training biceps, legs, or lower back. Through motor control, it offers multiple training modes for different muscle groups to meet the diverse needs of different individuals. These modes include fluid resistance, concentric isotonic, eccentric isotonic, and elasticity modes, among others.

[0023] The goal of motor control in intelligent training equipment is to provide resistance that meets the training needs according to different training modes. To achieve this, it is necessary to monitor the trainee's motion parameters (displacement, velocity, acceleration) in real time. These parameters are reflected in the motor's angular position, angular velocity, and angular acceleration. When the trainee's speed or direction of movement changes, the system responds quickly by adjusting the motor control current to adjust the motor output torque to provide the resistance required for different training modes.

[0024] In S100, training mode and motor operation data are acquired; based on the motor operation data and motor control current, a motion current model is constructed. The derivation process of the motion current model is as follows: (1) Based on torque control technology, the motor torque is obtained, as follows: (1) Formula (1) expresses the motor torque generation model based on impedance control. This control method simulates the mechanical behavior of the spring-damped-mass system. The formula is essentially an extension of Newton's second law in a rotating system. It describes that the motor torque needs to overcome the elastic, damping and inertial effects of the system. Therefore, the motor output torque = elastic force + damping force + inertial force.

[0025] Therefore, angular position, acceleration, and angular acceleration are all relative to the error value, essentially representing a closed-loop control expression for the motor torque. Among these, Indicates motor torque. This indicates the angular position of the motor, which is obtained through the motor encoder. This indicates the motor's angular velocity, which is obtained directly from a sensor. This represents the angular acceleration of the motor. Angular acceleration can be derived from the rate of change of angular velocity or obtained directly from a sensor. This represents the motor torque constant term. These represent the set angular position, set angular velocity, and set angular acceleration, respectively. The set angular position is set by the system's built-in standard motion trajectory database based on the training plan selected by the trainee (such as squats or rows), and is directly given by controlling the motor's control current. The set angular velocity and set angular acceleration can be specified directly or generated through differentiation. Indicates angular position error, Indicates angular velocity error. Indicates angular acceleration error. It represents the motor stiffness (unit: Nm / rad), similar to the spring constant, and indicates the motor's response strength to diagonal position deviation. This represents the motor damping (unit: Nm·s / rad), indicating the motor's response strength to angular velocity deviation. This represents the motor's own moment of inertia (unit: kg·m²), simulating inertial force to compensate for acceleration deviations. When a trainee pulls the smart training equipment, the motor generates resistance torque based on the deviation between the set position, speed, and acceleration and the actual values. For example, when the trainee suddenly accelerates (actual angular acceleration > set angular acceleration), the inertia term... It will generate a reverse torque to provide resistance.

[0026] Motor stiffness reflects the motor's ability to resist external disturbances. The equivalent stiffness can be dynamically changed by adjusting control parameters in real time (e.g., increasing the proportional gain Kp or increasing the current loop bandwidth). For example, stiffness can be adjusted by modifying controller parameters via host computer commands. Motor damping characterizes the system's ability to suppress oscillations and is related to the speed feedback gain (e.g., the differential gain Kd) or the damping design in the current loop. Damping parameters in the control algorithm (e.g., the differential gain Kd in PID control or virtual damping in the observer) can be adjusted. Motor moment of inertia is an inherent physical property of the motor and load, depending on the mass distribution. The physical moment of inertia cannot be changed in real time, but in control, the effect of inertia changes can be simulated through inertia identification algorithms and feedforward compensation to achieve equivalent inertia transformation. For example, the inertia parameters of the speed loop can be adjusted to make the motor response resemble systems with different inertia (but this does not change the actual physical inertia). It is important to note that adjusting the equivalent moment of inertia is essentially optimizing the control strategy, not changing the physical properties.

[0027] The motor can be an axial flux motor, which features high torque density and low moment of inertia. The motor rotor responds quickly to current changes, outputting adjusted torque within milliseconds. This allows motor stiffness K, damping B, and moment of inertia M to take effect within milliseconds, significantly improving the smoothness of force control and mode switching speed of the fitness equipment. Even in high-damping mode, it can still stably output large torque, avoiding mechanical shock caused by sudden changes in resistance.

[0028] (2) According to impedance control technology, the motor torque is determined by the motor's current torque constant. The motor torque is calculated based on the motor control current, and then assigned to the motor's direct output torque. The motor torque is then expressed as follows: (2) in, This indicates the motor control current, mainly referring to the motor's... shaft current, The current torque constant of the motor, also known as the motor torque constant, represents the torque generated per unit current of the motor. It is an important design parameter of the motor and reflects the electromagnetic conversion efficiency of the motor. Formula (2) expresses the linear relationship between motor torque and current. For permanent magnet motors (such as axial flux motors), the motor torque is proportional to the motor control current, and the proportionality coefficient is the torque constant.

[0029] (3) Combining formulas (1) and (2), the motion current model is obtained, as follows:

[0030] After the control device inside the intelligent fitness equipment calculates the resistance / motor torque required for training according to formula (1), it converts the required resistance into motor control current through formula (2), thereby driving the motor to output the target torque / resistance. This is the bottom layer of motor control, which is usually achieved by the current loop (inner loop).

[0031] The overall workflow of smart fitness equipment is as follows: Setting goals: The intelligent fitness equipment converts the training mode and the trainee's settings (such as the magnitude of constant resistance, speed-related resistance, etc.) into the set angular position, angular velocity, and angular acceleration.

[0032] Sensor feedback: After the trainee starts training, the sensor measures the actual angular position of the motor. The actual angular velocity of the motor can be obtained by differentiating the diagonal position or by the sensor. The angular acceleration of the motor can be obtained by differentiating the diagonal position twice or by the sensor.

[0033] Torque Calculation: According to formula (1), the required output motor torque is calculated, which is the ideal motor torque mentioned below.

[0034] Current control: The motor torque is converted into motor control current according to formula (2), and the motor is driven by the inverter.

[0035] In S200, based on the training mode and combined with the motion current model, the motion current model of each training mode is obtained. The training mode includes at least the fluid resistance mode, the centrifugal isotensile mode, the centripetal isotensile mode, and the elastic mode.

[0036] (1) Fluid resistance mode The fluid resistance mode mimics the characteristics of wind and water resistance rowing machines, with increased resistance as the user moves faster. This mode effectively strengthens muscles in the legs, waist, upper limbs, front of the torso, and back, providing excellent training for the entire body. In fluid resistance mode, motor stiffness and moment of inertia are both zero, and motor damping is minimal. A negative number with adjustable magnitude, indicating that the faster the trainee moves, the greater the resistance. Motor torque in fluid resistance mode. , means as follows:

[0037] The motion current model for the fluid resistance mode is then expressed as follows:

[0038] in, It is a constant compensation term and is set to be non-adjustable.

[0039] (2) Centrifugal isotonic mode In eccentric isotonic mode, resistance is provided when the trainee performs outward relaxation movements, but no resistance is provided when contracting inward. Exercises like weightlifting exemplify this. In this mode, it can effectively enhance the relaxation strength of muscles. In eccentric isotonic mode, the motor stiffness, motor damping, and motor moment of inertia are all zero, and the motor angular velocity... The direction of the normal force is positive, and the motor torque is positive. A positive and adjustable value indicates that the more the trainee pulls inward, the greater the output force, and thus the greater the output torque of the motor. The motion current model for the eccentric isotropic mode is then expressed as follows: .

[0040] (3) Centripetal isotensive mode In concentric isotonic mode, resistance is provided when the trainee performs inward contraction movements, and no resistance is provided when relaxing outward, similar to the arm strokes in swimming. This mode can effectively enhance the contractile strength of muscles. In concentric isotonic mode, the motor stiffness, motor damping, and motor moment of inertia are zero, and the motor angular velocity... The value is negative, the normal force direction is negative, and the motor torque is... , The value is an adjustable positive value, while the motor torque is a negative value, indicating that the more the trainee pulls outward, the greater the output force, and the greater the reverse torque of the motor. The motion current model for the centripetal isotonic mode is then expressed as follows: .

[0041] (4) Elasticity mode In elastic mode, the longer the trainee pulls the traction rope, the greater the resistance provided by the intelligent training equipment. This mode can effectively improve muscle strength, physical activity, and flexibility, becoming an aerobic exercise that strengthens cardiopulmonary function and improves posture. In elastic mode, the motor damping and motor moment of inertia are both zero, and the motor stiffness is negative. In fluid resistance mode, the motor torque... , means as follows:

[0042] The motion current model of the elastic mode is expressed as follows:

[0043] in, It is a constant compensation term and cannot be set or adjusted. A negative number indicates that the greater the distance / displacement the trainee pulls, the greater the force. The size is adjustable in several increments.

[0044] In S300, when the motor angular velocity exceeds a first preset angular velocity threshold, the motion current model for each training mode is adjusted based on the difference between the motor angular velocity and the first preset angular velocity threshold. The training equipment provides adjustable resistance via a motor. When the trainee grips the handle, the motor outputs torque to counteract the force exerted. If the trainee suddenly releases their grip, the rope and load will rebound rapidly due to inertia or elasticity after the motor resistance disappears, potentially causing equipment damage or personal injury. Many factors can cause this release, such as exhaustion, insufficient physical fitness of the trainee, improper training movements, and equipment malfunctions. Therefore, to prevent excessively rapid rebound of the rope and other components, which could injure the intelligent training equipment or surrounding trainees, the rebound speed of the rope needs to be limited, automatically activating the release protection mechanism. The essence of release protection is a speed-limiting mode, with the motor torque output being determined at the motor end. By monitoring the motor angular velocity (i.e., the linear velocity corresponding to the rope rebound) in real time, when the detected angular velocity exceeds the first preset angular velocity threshold (e.g., 10 rad / s), it is determined to be a release state, and then the speed loop proportional control (P control) is activated. The motor outputs reverse torque to suppress the rebound. The adjusted motor torque is expressed as follows:

[0045] in, This indicates the adjusted motor torque. This indicates the first preset angular velocity threshold. It can be based on the maximum angular velocity Settings, for example . Represents the proportional gain, which indicates the increase in torque output corresponding to a unit speed deviation. Its physical meaning is the motor's "resistance strength" to speed overshoot.

[0046] The adjusted motor control current is obtained based on the adjusted motor torque. The adjusted motion current model for each training mode is expressed as follows:

[0047] in, This indicates the adjusted motor control current.

[0048] The essence of drop detection is overspeed protection, which aims to protect trainees from drop-out situations that may occur during normal training.

[0049] In S400, a reinforcement learning model is constructed. The reinforcement learning environment simulates the dynamic behavior of the motor in the intelligent training equipment. The reinforcement learning model includes a state space model, an action space model, and a reward model, specifically including: (1) Construct a state-space model based on motor operating data. The state space S includes the motor's angular position and angular velocity. Angular acceleration of motor Motor stiffness Motor damping and the moment of inertia of the motor The state-space model is then represented as follows:

[0050] (2) Constructing an action space model based on motor control current. Action space A includes the motor's input signals, mainly the motor control current, etc. The action space model is represented as follows:

[0051] in, This represents a state in the state space. This represents an action in the action space. This represents a six-dimensional real vector.

[0052] (3) Based on whether the motor torque and motor angular velocity are within a reasonable range, construct a reward model. The reward model is constructed according to the motor performance (the state of motor torque and motor angular velocity). The reward model needs to be able to reflect the requirements of force position memory and overspeed protection. The reward model is represented as follows:

[0053] in, This represents the total reward value. This indicates the bonus value for the position where the force is applied. This represents the speeding bonus value. Both represent weighting coefficients, and .

[0054] Power Position Bonus Value This is used to reward the motor for producing the correct torque at a specific location, and is represented as follows:

[0055] The power position reward formula represents the amount of motor torque applied. (That is, the motor torque predicted by the reinforcement learning model is the predicted value of the motor torque obtained based on the current state) equals the ideal value of the motor torque. When this happens, the system will receive a positive reward. Otherwise, the system will be subject to a negative penalty. Ideal value of motor torque This is the final motor torque after training mode adjustment and hands-free overspeed detection adjustment.

[0056] The overspeed bonus value represents the negative reward given to the system when the motor angular velocity exceeds the safety limit. , means as follows:

[0057] The overspeed bonus formula represents the percentage of angular velocity that is increased. Exceeded the maximum motor angular velocity The system will be subject to a negative penalty. If the maximum allowed value is not exceeded, there is no additional reward or penalty.

[0058] (4) This application also imposes constraints on the motor's operating data, stipulating that the motor's operating data (motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping and motor moment of inertia) and motor torque must operate within the safe range formed by the minimum and maximum values.

[0059] In the S500, based on the state-space model, action-space model, and reward model, the motion current model for each training mode is optimized to obtain the optimal motion current model for each training mode. Specifically, based on the reward value obtained from the reward model, the motor control current in each state space is adjusted to adjust the motion current model until the optimization conditions are met, thus obtaining the optimal motor control current in each state space, i.e., the optimal motion current model. This includes the following steps: S510: Construct a policy model, which includes a policy function, a policy gradient model, a cumulative reward model, and a policy update model; The policy function represents the state. Select action The probability is determined by the policy parameter θ, and the policy function is expressed as follows:

[0060] in, Indicates the state given Choose action The probability of. Indicates the state given Time action The expected value, or the mean of the action, is usually output by a neural network. These represent policy parameters used to determine the probability of choosing an action, including parameters of mean and variance networks, such as the weights of a neural network. The covariance matrix representing the actions is usually a diagonal matrix (independent action dimension).

[0061] The policy gradient model is represented as follows:

[0062] in, Describe the policy objective function Regarding strategy parameters The gradient. Representing state The steady-state distribution represents the policy. Next state The probability of occurrence. The action-value function represents the state. Take action below And follow the strategy Expected cumulative rewards The policy function represents the policy parameters. The policy gradient model is used to calculate the expected value of the policy gradient, which guides the direction of policy parameter updates, thereby maximizing the cumulative reward. The core idea of ​​the policy gradient model is to adjust... This increases the probability of high-reward actions and decreases the probability of low-reward actions.

[0063] The cumulative reward model is represented as follows:

[0064] in, This represents the cumulative reward value. Indicates from time step Initial cumulative reward value, This indicates the total number of time steps in the current round. This represents the discount factor.

[0065] The policy update model is represented as follows:

[0066] in, This indicates the updated policy parameters. Indicates the strategy parameters, This represents the learning rate (step size), used to control the magnitude of policy parameter updates.

[0067] S520: Train the reinforcement learning model for each training mode separately using the REINFORCE algorithm, including the following steps: (1) Initialize hyperparameters (learning rate, discount factor, maximum number of rounds and maximum number of steps per round), policy parameters and trajectory buffer.

[0068] (2) Obtain the initialization state As the current state, the action to be taken is obtained based on the current state and the policy function. The initial state can be obtained by the reset function, or the initial state can be randomly sampled to increase the diversity of training, or it can start from a fixed initial state. This application does not limit this.

[0069] (3) Based on the current state and the action taken, and combined with the reward model, the current reward value is obtained, and the current state, the action taken, and the current reward value are added to the trajectory buffer.

[0070] (4) Based on the action taken, obtain the next state and update the step count.

[0071] (5) Repeat steps (1) to (4) until the round termination condition is met. The round termination condition includes the environment returning a termination signal (such as the motor running data exceeding the safe range or the task being completed) or reaching the maximum number of steps in the current round.

[0072] (6) Based on the trajectory buffer and combined with the cumulative reward model, the cumulative reward value of each step in the current round is obtained, and then the cumulative reward value of the current round is obtained; based on the trajectory buffer and combined with the policy gradient model, the policy gradient of the current round is obtained, and based on the policy gradient of the current round and combined with the policy update model, the policy parameters are updated.

[0073] (7) Repeat steps (1) to (6) until the training termination condition is met, and obtain the optimal motor control current, i.e. the optimal motion current model, in each state space. The training termination condition includes the most recent average cumulative reward value of the rounds being greater than the preset reward threshold or the training reaching the maximum number of rounds.

[0074] In the S600, real-time motor operation data is acquired. Based on the target training mode and the real-time motor operation data, combined with the optimal motion current model of the target training mode, the real-time optimal motor control current for the target training mode is obtained. The trained reinforcement learning model is also the optimal motion current model for each training mode. Based on the target training mode and the real-time motor operation data, combined with the optimal motion current model corresponding to the target training mode, the probability distribution of the output action (motor control current) is obtained. The probability distribution of the action is usually assumed to be a Gaussian distribution, so the mean of the motor control current is selected as the optimal motor control current.

[0075] This invention simulates the operation of a motor in fitness equipment using a reinforcement learning model. The REINFORCE algorithm is a strictly online algorithm, and its training data must be generated in real time through the interaction between the current policy and the environment. Training the reinforcement learning model using the REINFORCE algorithm does not require the use of historical data (i.e., past interaction experience) for training. After training, the resulting model can intelligently select the optimal action (motor control current) based on the current state (motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping, and motor moment of inertia) to maximize long-term cumulative rewards (capable of force position memory and overspeed protection), and corresponding protective measures. It corrects the problem of uneven torque under low motor speed and counter-driving force, enabling the intelligent training equipment to provide force position memory and overspeed protection in different training modes.

[0076] Example 2: A motor control system for intelligent training equipment based on reinforcement learning, such as Figure 2 As shown, it includes a basic model module 100, a training mode model module 200, an ultra-fast adjustment module 300, an inference model construction module 400, an inference model training module 500, and a real-time optimization module 600. The basic model module 100 constructs a motion current model based on motor operating data and motor control current; wherein, the motor operating data includes at least motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping, and motor moment of inertia; The training mode model module 200, based on the training mode and combined with the motion current model, obtains the motion current model for each training mode. The overspeed adjustment module 300 adjusts the motion current model of each training mode according to the difference between the motor angular velocity and the first preset angular velocity threshold when the motor angular velocity is greater than the first preset angular velocity threshold. The reasoning model construction module 400 constructs a state space model based on motor operating data; constructs an action space model based on motor control current; and constructs a reward model based on whether the motor torque and motor angular velocity are within a reasonable range. The inference model training module 500 optimizes the motion current model for each training mode based on the state space model, action space model, and reward model to obtain the optimal motion current model for each training mode. The real-time optimal module 600 acquires real-time motor operation data, and based on the target training mode and the real-time motor operation data, combines the optimal motion current model of the target training mode to obtain the real-time optimal motor control current of the target training mode.

[0077] Various changes and modifications made without departing from the spirit and scope of this invention, and all equivalent technical solutions, also fall within the scope of this invention.

[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0079] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] It should be noted that: The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0084] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A motor control method for an intelligent training device based on reinforcement learning, wherein the intelligent training device provides multiple training modes, characterized in that, Includes the following steps: A motion current model is constructed based on motor operating data and motor control current; wherein, the motor operating data includes at least motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping, and motor moment of inertia; Based on the training mode and combined with the motion current model, the motion current model for each training mode is obtained; When the motor angular velocity is greater than the first preset angular velocity threshold, the motion current model of each training mode is adjusted according to the difference between the motor angular velocity and the first preset angular velocity threshold. A state-space model is constructed based on motor operating data; an action-space model is constructed based on motor control current; and a reward model is constructed based on whether motor torque and motor angular velocity are within a reasonable range. Based on the state space model, action space model, and reward model, the motion current model for each training mode is optimized to obtain the optimal motion current model for each training mode. Real-time motor operation data is acquired. Based on the target training mode and the real-time motor operation data, and combined with the optimal motion current model of the target training mode, the real-time optimal motor control current of the target training mode is obtained.

2. The motor control method for intelligent training equipment based on reinforcement learning according to claim 1, characterized in that, The motion current model is represented as follows: in, Indicates the motor control current. Indicates motor stiffness. Indicates motor damping, Indicates the moment of inertia of the motor. Indicates the angular position of the motor. Indicates the angular velocity of the motor. Indicates the angular acceleration of the motor. These represent the set angular position, set angular velocity, and set angular acceleration, respectively. This represents the motor torque constant term. This represents the motor torque constant.

3. The motor control method for an intelligent training device based on reinforcement learning according to claim 1, characterized in that, The training modes include at least the fluid resistance mode, the centripetal isotensive mode, the eccentric isotensive mode, and the elastic mode; In the fluid resistance mode, the motor stiffness and the motor moment of inertia are zero, and the motor damping is negative. In the centripetal isotensile mode, the motor stiffness, the motor damping, and the motor moment of inertia are all zero, and the motor angular velocity is a positive value. In the centrifugal isotensile mode, the motor stiffness, the motor damping, and the motor moment of inertia are all zero, and the motor angular velocity is negative. In the elastic mode, the motor damping and the motor moment of inertia are zero, and the motor stiffness is negative.

4. The motor control method for an intelligent training device based on reinforcement learning according to claim 1, characterized in that, The motion current model for each training mode is adjusted based on the difference between the motor angular velocity and a first preset angular velocity threshold. The adjusted motion current model for each training mode is expressed as follows: in, This indicates the adjusted motor control current. Indicates proportional gain. Indicates the angular velocity of the motor. This indicates the first preset angular velocity threshold. This represents the motor torque constant.

5. The motor control method for an intelligent training device based on reinforcement learning according to claim 1, characterized in that, The process of constructing a reward model based on whether the motor torque and motor angular velocity are within a reasonable range includes the following steps: Based on the motor operating data, the predicted value of the motor torque is obtained; The bonus value for the force application position is obtained based on whether the predicted value of the motor torque is equal to the ideal value of the motor torque. The overspeed bonus value is obtained based on whether the motor angular velocity exceeds the maximum motor angular velocity. The total reward value is obtained by weighted summation of the power position reward value and the speed reward value.

6. The motor control method for an intelligent training device based on reinforcement learning according to claim 1, characterized in that, The optimization of the motion current model for each training mode based on the state space model, action space model, and reward model to obtain the optimal motion current model for each training mode includes the following steps: Construct a policy model, which includes a policy function, a policy gradient model, a cumulative reward model, and a policy update model; Obtain the initial state as the current step state, and based on the current step state, combine it with the policy function to obtain the action to be taken; Based on the current step state and the action taken, and combined with the reward model, the reward value for the current step is obtained; Based on the action taken, obtain the next state and update the step count; Until the round termination condition is met, based on the reward value of each step and combined with the cumulative reward model, the cumulative reward value of each step in the current round is obtained, and thus the cumulative reward value of the current round is obtained; based on the reward value of each step and combined with the policy gradient model, the policy gradient of the current round is obtained; based on the policy gradient of the current round and combined with the policy update model, the policy parameters are updated. The training continues until the training termination condition is met, at which point the optimal motor control current, i.e., the optimal motion current model, is obtained for each state space. The training termination condition includes the most recent average cumulative reward value for rounds being greater than a preset reward threshold or the training reaching the maximum number of rounds.

7. A motor control system for intelligent training equipment based on reinforcement learning, characterized in that, The motor control method for intelligent training equipment based on reinforcement learning as described in any one of claims 1 to 6 includes a basic model module, a training mode model module, an overspeed adjustment module, an inference model construction module, an inference model training module, and a real-time optimization module. The basic model module constructs a motion current model based on motor operating data and motor control current; wherein, the motor operating data includes at least motor angular position, motor angular velocity, motor angular acceleration, motor stiffness, motor damping, and motor moment of inertia; The training mode model module, based on the training mode and combined with the motion current model, obtains the motion current model for each training mode; The overspeed adjustment module adjusts the motion current model of each training mode according to the difference between the motor angular velocity and the first preset angular velocity threshold when the motor angular velocity is greater than the first preset angular velocity threshold. The reasoning model construction module constructs a state space model based on motor operating data; constructs an action space model based on motor control current; and constructs a reward model based on whether the motor torque and motor angular velocity are within a reasonable range. The inference model training module optimizes the motion current model for each training mode based on the state space model, action space model, and reward model to obtain the optimal motion current model for each training mode. The real-time optimal module acquires real-time motor operation data, and based on the target training mode and the real-time motor operation data, combines the optimal motion current model of the target training mode to obtain the real-time optimal motor control current of the target training mode.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

9. A motor control device for intelligent training equipment based on reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

Images