An electric moped intelligent assist control system based on deep reinforcement learning and a control method thereof

Abstract

The application discloses an electric moped intelligent power assisting control system based on deep reinforcement learning and a control method thereof, and belongs to the technical field of electric moped control. The system comprises a bottom data acquisition module, a communication module and an AI inference module; the bottom takes an STM32 microcontroller as a core, acquires pedal torque, speed and three-axis acceleration data, and uploads the data to the AI inference module through the communication module; the AI inference module deploys a strategy network based on an improved PPO algorithm, an Actor network and a Critic network are independently introduced into a Transformer encoder to process historical time sequence states, action output adopts bounded action parameterization and sampling truncation to constrain torque boundaries, and is combined with a multi-dimensional reward function, course learning and field randomization training. The system adaptively adjusts power assistance according to a riding state, the power assistance is smooth and timely under flat road and uphill working conditions, and the riding experience is significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of electric-assisted bicycle control technology, specifically relating to an intelligent power-assisted control system and control method for electric-assisted bicycles based on deep reinforcement learning. Background Technology

[0002] Electric-assisted bicycles, as an efficient and low-carbon short-distance travel tool, have been widely used in urban commuting and leisure cycling scenarios. Their core control objective is to output appropriate motor assistance in real time based on the rider's pedal input and the vehicle's operating status, achieving a coordinated combination of human and electric power. However, existing commercial electric-assisted bicycles generally employ proportional control or rule-based control strategies, which linearly map the magnitude of pedaling torque to the motor's output torque. While these methods are simple in structure and easy to implement, they exhibit significant shortcomings in complex and changing real-world riding environments: on flat surfaces or during the initial acceleration phase, linear proportional control can easily produce a jerky feeling and excessive assistance, reducing riding comfort; when climbing hills or with increased load, the assistance response is often delayed and insufficient in amplitude, requiring the rider to still exert considerable effort; and under gliding conditions with zero or minimal pedaling torque, the system may mistakenly output unnecessary assistance, resulting in energy waste and safety hazards.

[0003] Deep reinforcement learning algorithms, such as Proximal Policy Optimization (PPO), have demonstrated superior performance in continuous control tasks, providing a new technical path for the intelligent control of electric-assisted bicycles. Theoretically, through interactive training in a simulation environment, deep reinforcement learning models can learn nonlinear assist strategies that adapt to different road conditions and riding styles. However, directly applying PPO algorithms to assist control faces several technical obstacles. First, assist decisions essentially depend on the continuous changing trend of the riding state; for example, the rise or fall slope of the pedaling torque directly relates to the timing of assist prediction. Standard PPO algorithms, based on the Markov decision process assumption, only utilize the current state for decision-making, lacking the ability to model historical time-series information and struggling to capture such dynamic features. Second, there is an unavoidable deviation between the vehicle dynamics model, road friction, and wind resistance parameters in the simulation environment and the real physical system—a generalization gap from simulation to reality. This leads to a significant performance degradation of the strategies trained in simulation after deployment on real vehicles. In addition, the output torque of the boost motor is limited by physical hardware and has strict upper and lower boundary constraints. The sampling output of the traditional Gaussian strategy cannot guarantee that it will always meet the boundary requirements. Overshoot or undershoot will affect the control effect and may damage the actuator.

[0004] In summary, there is an urgent need for an intelligent assist control scheme that can effectively integrate historical time-series information, enhance the generalization ability of simulation training, and strictly meet the output boundary constraints, so as to break through the existing technical bottlenecks and achieve safe, comfortable, and efficient riding assistance. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an intelligent power assist control system and method for electric-assisted bicycles based on deep reinforcement learning. Based on an improved PPO algorithm, it independently introduces Transformer encoders and adds residual connections to the Actor and Critic networks to model historical temporal states. Combined with bounded action parameterization, multidimensional reward functions, course learning, and domain randomization strategies, it achieves adaptive, smooth, and boundary-constrained intelligent power assist control for electric-assisted bicycles.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning includes a low-level data acquisition module, a communication module, and an AI inference module. The low-level data acquisition module, with an STM32 microcontroller as its core, integrates a pedal torque sensor, a speed sensor, and a triaxial accelerometer to collect data on human pedaling torque, speed, and vehicle acceleration during riding. The communication module is connected to both the low-level data acquisition module and the AI ​​inference module, wirelessly uploading sensor data to the AI ​​inference module and receiving control commands from it. The AI ​​inference module deploys a policy network trained based on an improved PPO algorithm, which infers and outputs assist torque based on the received sensor data. This assist torque is then converted into a control voltage and sent via the communication module, driving the power assist motor to output the corresponding assist.

[0008] Furthermore, the continuous multi-frame historical observation sequence is constructed by the AI ​​inference module in the following manner: the single-frame state vectors of the current moment and the past N-1 consecutive moments, totaling N frames, are concatenated in time sequence into an N×7-dimensional temporal state matrix, wherein the single-frame state vector is a 7-dimensional vector, and each dimension is, in order, the human pedaling torque, the driving speed, the rate of change of the human pedaling torque, the vehicle's X-axis acceleration, the vehicle's Y-axis acceleration, the vehicle's Z-axis acceleration, and the assist torque of the previous moment; both the Actor network and the Critic network use the temporal state matrix as input, and the parameters of the two networks are independent of each other.

[0009] Furthermore, the Transformer encoder module includes a multi-head self-attention layer, a first residual connection and a first normalization layer, a feedforward network layer, a second residual connection and a second normalization layer connected in sequence; the multi-head self-attention layer is used to model the temporal dependencies between each moment in the continuous multi-frame historical observation sequence; the feedforward network layer is a two-layer fully connected mapping structure, with the hidden layer dimension being larger than the input dimension; the first residual connection and the second residual connection respectively add the input of the multi-head self-attention layer and the input of the feedforward network layer to their respective outputs, which is used to retain shallow feature information and alleviate gradient vanishing.

[0010] Furthermore, in the bounded action parameterization, the strategy head outputs the action mean and shared standard deviation based on the temporal characteristics. The action mean is mapped to the output range of the assist motor via a sigmoid activation function and initialized with a negative bias to make the initial output tend to zero. After sampling from a normal distribution with the mean being the mapping result and the standard deviation being the shared standard deviation, values ​​less than the lower limit of the output boundary are truncated to the lower limit, and values ​​greater than the upper limit of the output boundary are truncated to the upper limit, so as to ensure that the assist torque is within the physical constraints of the assist motor.

[0011] Furthermore, the training of the policy network employs a multidimensional reward function, which includes: a torque matching reward term, a zero torque penalty term, a smoothing reward term, and a response reward term.

[0012] Furthermore, the reward function used in the training of the policy network includes torque matching reward, zero torque penalty, smoothing reward and response reward. Torque matching reward is used to guide the output assist torque to match the expected assist torque. Zero torque penalty is used to constrain the assist output to be zero when the human pedaling torque is zero. Smoothing reward is used to constrain the rate of change of the output assist torque. Response reward is used to encourage a rapid response to changes in pedaling torque.

[0013] Furthermore, the policy network is trained using phased learning. In the first phase, torque tracking and smooth output are learned under fixed low-speed conditions. In the second phase, speed changes are introduced to adapt the policy to dynamic conditions. In the third phase, the network is trained under random scenarios with all operating conditions, and the domain randomization of parameters such as slope, total mass, rolling resistance, and wind resistance is introduced.

[0014] This invention also provides an intelligent power assist control method for electric-assisted bicycles based on deep reinforcement learning, comprising the following steps:

[0015] Step 1: Collect pedal torque, speed and triaxial acceleration data during the riding process, calculate the rate of change of human pedal torque, and construct a time series observation vector together with the motor output torque at the previous moment;

[0016] Step 2: Input the continuous multi-frame temporal observation vectors into the policy network. The policy network is an Actor network and a Critic network based on the PPO algorithm. The Actor network and the Critic network each independently contain a Transformer encoder module to model the historical temporal state.

[0017] Step 3, the output of the strategy network adopts bounded action parameterization to output the assist torque: the strategy head outputs the action mean and shared standard deviation, the action mean is mapped to the allowable output range of the motor through the sigmoid activation function, and the action is sampled and the boundary is clipped so that the assist torque is kept within the allowable output range of the motor;

[0018] Step 4: Generate control voltage based on the obtained assist torque to drive the assist motor to output assist.

[0019] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned intelligent power assist control method for electric-assisted bicycles based on deep reinforcement learning.

[0020] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned intelligent power assist control method for electric-assisted bicycles based on deep reinforcement learning.

[0021] The beneficial effects of this invention are as follows:

[0022] Improved temporal awareness and decision smoothness: Transformer encoder modules are independently introduced into the Actor and Critic networks respectively. A multi-head self-attention mechanism is used to model continuous multi-frame historical observation sequences, enabling the policy network to perceive the changing trends of pedaling torque and the dynamic evolution of riding status, thus making more predictive assist decisions. Compared to the PPO algorithm, which relies solely on the current state, this invention significantly suppresses abrupt changes and oscillations in assist output, resulting in smooth, jerky riding on flat roads and timely and continuous assist intervention on uphill sections, greatly improving riding comfort.

[0023] The output boundary strictly satisfies hardware safety requirements: Bounded motion parameterization is employed, with the strategy head outputting the mean motion value and a shared standard deviation. The mean motion value is mapped to the motor's allowable output range (0~38 N·m) using a sigmoid activation function. After sampling, the motion is subject to boundary clipping to ensure that the assist torque always remains within the motor's allowable output range. This mechanism constrains the motion space at both the motion parameterization and sampling boundary levels, avoiding common output saturation, overshoot, or undershoot problems found in traditional Gaussian strategies or tanh mapping methods, thus ensuring the safe operation of the motor and actuators.

[0024] Enhanced generalization ability from simulation to reality: Through phased course learning and domain randomization (random sampling of parameters such as slope, total mass, rolling resistance, and wind resistance), the model is fully trained in diverse simulation scenarios, significantly narrowing the gap between the simulation environment and the real physical system; residual connections and Transformer temporal modeling further stabilize the training process. Real-vehicle testing shows that the model maintains stable and efficient assist performance on flat roads and incline conditions, demonstrating good deployment generalization ability.

[0025] The system's real-time response meets application requirements: the end-to-end delay from changes in pedaling torque to motor assist output is controlled within 50~150ms, sensor data is uploaded at a frequency of 30Hz, and with a stable wireless communication architecture, it can meet the real-time requirements for assist tracking during riding, realizing intelligent, safe, and energy-saving riding assistance control. Attached Figure Description

[0026] Figure 1 This is a hardware, software, and communication closed-loop architecture diagram of an intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning, according to the present invention.

[0027] Figure 2 This is a network structure diagram of the PPO algorithm that integrates Transformer;

[0028] Figure 3 This is a comparison chart of the assist torque mapping for different training schemes;

[0029] Figure 4 This is a time series diagram of pedal torque and strategy output for a typical cycling segment;

[0030] Figure 5 This is a comparison chart of the quality indicators of the seven solutions.

[0031] Figure 6 This is a complete system diagram from the on-site field test. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] like Figure 1 As shown, the intelligent power assist control system for electric bicycles based on deep reinforcement learning of the present invention adopts a three-layer architecture, including a bottom-layer data acquisition module, a communication module, and an AI inference module. The layers interact with each other through standard interfaces to form a complete closed loop of "acquisition-transmission-inference-control".

[0034] The underlying data acquisition module is based on a self-made PCB circuit board with an STM32F103 microcontroller, integrating signal acquisition, data processing, and communication functions. It mainly includes the following modules: a pedal torque sensor mounted on the bicycle's bottom bracket to collect the rider's pedaling torque; the sensor output voltage range is 1.5~3.0V, corresponding to a torque range of 0~100.9Nm, and is acquired through STM32's ADC1 channel 2 (PA2 pin); a speed sensor mounted on the rear wheel assist motor; its output voltage range is 0~18V, corresponding to a speed range of 0~13m / s; this voltage is reduced to 0~3.3V by a linear step-down circuit on the PCB before being acquired by STM32's ADC1 channel 3 (PA3 pin); and an accelerometer using the ADXL345 triaxial accelerometer, communicating with the STM32 via an I2C interface (PB6 / PB7) to collect X, Y, and Z axis acceleration data during riding, assisting in judging road conditions and riding posture.

[0035] The power assist motor is mounted on the rear wheel of the bicycle, with a rated power of 350W, an operating voltage of 36~48V, and a maximum torque of 38Nm. The input control voltage of the motor controller is 0~5V, which is linearly related to the output torque of the motor. The STM32 microcontroller outputs a voltage of 0~3.3V through DAC channel 1 (PA4 pin), which is amplified to 0~5V by a linear boost circuit on the PCB before driving the motor controller to output the corresponding power assist.

[0036] The PCB circuit board was designed and soldered in-house, and mainly includes a linear buck circuit, a linear boost circuit, an ESP8266 WiFi communication module, an ADXL345 accelerometer sensor module, and an STM32F103 main control chip minimum system.

[0037] The communication module of the system adopts a three-layer communication architecture to realize two-way communication of underlying sensor data upload and AI control instruction issuance. Wired communication between ESP8266 and STM32 is carried out through the USART serial port, with the baud rate set to 115200bps. The STM32 packs the collected sensor data according to the protocol format and sends it to the ESP8266. The ESP8266 is configured in TCP server mode, connected to the local WiFi network, and listens to port 8080; the AI inference client (laptop) acts as a TCP client and connects to the ESP8266 through the IP address and port. The upload format of sensor data is SENSOR_DATA_<torque voltage>_<speed voltage>_<X-axis acceleration>_<Y-axis acceleration>_<Z-axis acceleration>, with a sending frequency of 30 times per second. The issuance format of AI control instructions is AIRESULT_<DAC voltage value>. After receiving it, the ESP8266 forwards it to the STM32 through the serial port, and the STM32 parses it to control the DAC to output the corresponding motor control voltage.

[0038] The AI inference module is deployed on the laptop, written in Python language, and its main functions include: connecting to the ESP8266 through a TCP socket to receive sensor data streams; converting the received voltage signals into physical quantities, that is, converting the torque ADC sampling value into torque (Nm) and the speed voltage into speed (m / s); constructing a time-series observation vector and inputting it into the trained neural network model for inference; converting the assist torque output by the model into a DAC voltage value and sending it back to the STM32. The inference-side program uses the PyTorch framework to load model weights, supports historical frame sequence input, and ensures time-series consistency.

[0039] As shown in Figure 2 , the deep reinforcement learning model of the present invention is based on the PPO algorithm, which sets mutually independent Actor (policy) network and Critic (value) network, and independently introduces a Transformer encoder module into each of the two networks. Both the Actor network and the Critic network read the observation sequence of the latest 5 steps of history to capture the time-series evolution characteristics of the riding state. Specifically, the state input of the policy network is a 7-dimensional single-frame state vector , and its components are sequentially: human pedaling torque , driving speed , change rate of human pedaling torque , three-axis acceleration of the vehicle body and the motor output torque at the previous moment , where t represents the moment, that is:

[0040] ,

[0041] The policy network maintains a temporal state buffer, storing the state vectors of the most recent 5 time steps. , , , , They are then assembled into a 5×7 time-series state matrix. The input is used as the common input to both the Actor and Critic networks. The input to the Actor branch is exactly the same as the input to the Critic branch, but the parameters of the two networks are independent to avoid parameter competition between policy features and value features.

[0042] like Figure 2 As shown, the Actor network and the Critic network share the same input embedding, Transformer encoding, and pooling steps, but the policy head and value head are independent of each other. The following steps explain this step by step:

[0043] The first step, input embedding and position encoding: the input 5×7 time-series state matrix Mapping to a 256-dimensional hidden space through a linear embedding layer, i.e.:

[0044] ,

[0045] in, This represents the weight matrix of the embedding layer, which maps the 7-dimensional input to 256 dimensions. This represents the bias vector of the embedding layer; This represents the intermediate representation after linear embedding;

[0046] Subsequently, learnable positional encoding was added. To preserve the time sequence information, the final embedded output is obtained. :

[0047] ,

[0048] Step 2, Transformer encoder module:

[0049] The Transformer encoder consists of two sub-layers: Multi-Head Self-Attention (MHSA) and Feedforward Network (FFN). Each sub-layer is followed by residual connections and Layer Normalization (LayerNorm).

[0050] Multi-head Self-Attention (MHSA): Set up 4 attention heads, each with different dimensions. =256 / 4=64. (The rest of the text appears to be a typo and can be omitted.) Linear projections are respectively onto the query matrix Q, the key matrix K, and the value matrix V:

[0051] ,

[0052] in, , , The projection matrix is ​​learnable. The self-attention output is... :

[0053] ,

[0054] In the formula, T is the transpose, and softmax represents row-wise normalization;

[0055] The intermediate output is obtained by the first residual connection and layer normalization. :

[0056] ,

[0057] Feedforward Network (FFN): Employs two fully connected layers with 1024 hidden dimensions and ReLU activation function.

[0058] + ,

[0059] in, , For a learnable projection matrix, , For the corresponding bias term, For the output of the feedforward network;

[0060] The second residual connection and layer normalization yield the final output of the encoder:

[0061] .

[0062] Step 3, global average pooling:

[0063] right The time dimension (a total of 5 time points) is averaged to compress the sequence information into a fixed-length feature vector. :

[0064] ,

[0065] In the formula, Let represent a 256-dimensional vector at time i.

[0066] Step 4, Actor Strategy Header (Output Action):

[0067] In an Actor network, feature vectors They are processed through the mean head and the standard deviation head, respectively.

[0068] Mean Header: A linear value used to calculate the mean of actions. :

[0069] ,

[0070] The learnable projection matrix Corresponding bias term Bounded motion parameterization is employed, and the mean is mapped to the motor output boundary [0, 38] N·m using the sigmoid function. An offset of −2.0 is set to make the initial output close to 0, i.e.:

[0071] ;

[0072] here This is the center value of the final motion distribution (unit: N·m).

[0073] Standard Deviation Head: Calculates the shared standard deviation of action exploration. :

[0074] ,

[0075] The learnable projection matrix Corresponding bias term , >0 represents the standard deviation of the noise level.

[0076] The final action (i.e., the assist torque output at this moment) Sample from the normal distribution and clip to within the boundary:

[0077] ,

[0078] Where ϵ is standard normal random noise, Clip(⋅,0,38) truncates the output to [0,38] N·m to ensure that the physical limitations of the motor are strictly met.

[0079] Step 5, Critic Value Head (outputs state value):

[0080] In the Critic network, feature vectors A direct linear mapping to a state value scalar :

[0081] ,

[0082] The learnable projection matrix Corresponding bias term . Used to estimate the expected cumulative discount return in the current state, providing a baseline for policy updates.

[0083] The training objective of the policy network is to use the total loss function of the proximal policy optimization algorithm. This includes pruning strategy loss, value loss, entropy regularization term, and behavior cloning loss:

[0084] ,

[0085] in, PPO pruning strategy loss is used to limit the magnitude of strategy updates and maximize expected returns; The mean squared error loss of the value network is used to update the Critic network parameters; This is the policy entropy, used to encourage exploration and avoid premature convergence; Behavioral cloning loss is enabled during the mixed experience training or imitation learning phase to fit expert cycling data. , , These are the weighting coefficients for each type of loss.

[0086] The training process employs a three-stage learning strategy: the first stage (0-1 million steps) involves fixed low-speed conditions, focusing on learning torque tracking and smooth output; the second stage (1 million-3 million steps) introduces speed variations, reduces the smoothing penalty weight, and encourages the strategy to adapt to dynamic conditions; the third stage (after 3 million steps) involves randomized training under all conditions, with randomized changes to parameters such as gradient, total mass, rolling resistance, and wind resistance for domain randomization.

[0087] The assist control problem of electric-assisted bicycles can be modeled as a Markov decision process in a continuous action space, represented by a quintuple (S, A, P, R, γ). As mentioned earlier, the state input S of the policy network is a 7-dimensional single-frame state vector. The action space A is a one-dimensional continuous space, representing the assist torque output by the motor. The action range is [0, 38] N·m, corresponding to the maximum output torque of the motor. The state transition probability P is determined by the longitudinal dynamics model of the bicycle. This model considers the total mass of the rider and bicycle (80 kg), rolling resistance, air resistance, and slope resistance. The human torque and motor torque are converted into equivalent driving forces by dividing by the crank length of 0.175 m and the wheel radius of 0.38 m, respectively. The rolling resistance coefficient is taken as 0.008, and the frontal area is taken as 0.5 m². The discount factor γ is set to 0.99. The reward function R is a multi-dimensional reward function, including: ① torque matching reward. The assist torque used to guide the model output is matched with the expected assist torque, and the weighting coefficient is set to 80.0; ② Zero torque penalty To constrain the model so that the output assist torque is also zero when the human pedaling torque is zero, thus avoiding ineffective assistance, the weight coefficient is set to 200.0; ③ Smoothing reward The rate of change of the assist torque in the model output is used to constrain the assist output and prevent it from becoming jerky or unstable; the weighting coefficient is set to 0.0003. ④ Response reward This is used to encourage the model to respond quickly to changes in human pedaling torque, improving the responsiveness of the assist. The weighting coefficient is set to 1.5, that is: The specific formulas for each reward component are as follows:

[0088] ,

[0089] ,

[0090] ,

[0091] ,

[0092] The meanings of each parameter are as follows: This represents the assist torque (N·m) output by the motor at the current moment. The desired assist torque (N·m); The motor assist torque (N·m) at the previous moment; The torque for human pedaling (N·m); =38 N·m is the maximum output torque of the assist motor; , , , These are the weighting coefficients for the four rewards, and their values ​​are as follows: =80.0、 =200.0、 =0.0003、 =1.5.

[0093] Regarding action output, the traditional PPO algorithm uses the tanh activation function to map the network output to the [-1,1] interval, and then obtains the action value through linear transformation. This method is prone to output saturation and weakening of the learning signal. This invention adopts bounded action parameterization to model the action output: the policy head outputs the action mean and shared standard deviation. The action mean is mapped to the bounded interval [0,38] N·m through the sigmoid activation function, and initialized with a -2.0 bias to make the initial output tend to zero assist. After sampling from a normal distribution with the mean as the mapping result and the standard deviation as the shared standard deviation, the sampled values ​​are truncated to the boundary, with negative values ​​truncated to 0 and values ​​exceeding 38 truncated to 38, thereby ensuring that the output torque meets the boundary constraint of [0,38] N·m. The shared standard deviation is used to constrain the exploration degree of the action.

[0094] To verify the feasibility of the technical solution of this invention, simulation comparison experiments and field tests were conducted. Through comparative experiments of seven training schemes (including LSTM, Mamba, independent Actor-Critic, residual connection + curriculum learning, mixed real data training, and imitation learning training), the PPO+Transformer+residual connection+curriculum learning scheme adopted in this invention exhibits the optimal assist characteristics: output is close to zero at zero input, provides large assist at low speed and high torque, assist output smoothly decreases with speed increase at high speed, and provides moderate assist at high torque and high speed. In a total of 12,347 sensor-decision samples in real vehicle tests of the seven schemes, the false assist rate of this scheme was only 4.5%, the output oscillation amplitude between adjacent control cycles was 1.06 N·m, the oscillation rate was 6.7%, the response gain reached 10.8 N·m, the correlation coefficient between pedal torque and assist output was 0.83, and the maximum output was 33.2 N·m, which did not exceed the actuator boundary of 38 N·m. Figure 3 As shown, the input torque-average output torque mapping curve of this scheme is strictly monotonically increasing within the input range of 0~100 N·m, and the output approaches zero when the input is zero; as Figure 4 As shown, in a typical cycling segment, the output climbs to the matched value within 1-2 control cycles after pedaling begins, the output is continuous during sustained pedaling, and the output smoothly decays without error after pedaling stops; for example... Figure 5 As shown in (a), (b), (c), and (d), this scheme achieves the best overall assist quality in terms of output oscillation amplitude, oscillation rate, false assist rate, and response gain.

[0095] The field testing environment included flat cement roads and sloping roads. During the test, the AI ​​inference module collected sensor data in real time, performed inference through the Transformer model, and sent control commands to the STM32 to drive the assist motor. The complete system is as follows: Figure 6 As shown. Test results show that riding on flat roads is smooth and without jerking, with no extra assistance when there is zero input; the model can increase assistance in advance when going uphill, and the assistance is sufficient throughout the journey; the total delay from the change in pedaling torque to the response of the assist motor is about 50~150ms, and the sensor data is uploaded at a frequency of 30Hz, which meets the real-time requirements; the ESP8266 WiFi communication remains stable at a frequency of 30Hz.

[0096] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned intelligent power assist control method for electric-assisted bicycles based on deep reinforcement learning.

[0097] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned intelligent power assist control method for electric-assisted bicycles based on deep reinforcement learning.

[0098] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning-based intelligent power assist control system for electric-assisted bicycles, characterized in that, include: The underlying data acquisition module, with a microcontroller at its core, integrates a pedal torque sensor, a speed sensor, and a three-axis acceleration sensor to collect data on human pedaling torque, speed, and three-axis acceleration of the vehicle body during riding. The communication module is connected to the underlying data acquisition module and the AI ​​inference module respectively, and is used to wirelessly upload the sensor data to the AI ​​inference module and receive control commands issued by the AI ​​inference module. The AI ​​inference module is equipped with a policy network trained based on an improved near-end policy optimization algorithm. This network infers and outputs the assist torque based on the received sensor data, converts the assist torque into a control voltage, and sends it via the communication module. The microcontroller then drives the assist motor to output the corresponding assist torque. The policy network takes as input a continuous multi-frame historical observation sequence containing the human pedaling torque, the driving speed, the rate of change of the human pedaling torque, the vehicle's three-axis acceleration, and the assist torque from the previous moment. It includes an Actor network and a Critic network, each independently containing a Transformer encoder module. The Transformer encoder module performs temporal dependency modeling on the continuous multi-frame historical observation sequence and outputs temporal features. The action output of the policy network uses bounded action parameterization. The policy head outputs the action mean and shared standard deviation based on the temporal features. The action mean is mapped to the output range of the assist motor using a sigmoid activation function. After sampling, the sampled actions are clipped to ensure that the final output assist torque remains within the output range of the assist motor.

2. The intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning according to claim 1, characterized in that, The continuous multi-frame historical observation sequence is constructed by the AI ​​inference module in the following manner: the single-frame state vectors of the current moment and the past N-1 consecutive moments, totaling N frames, are concatenated in time sequence into an N×7-dimensional temporal state matrix, wherein the single-frame state vector is a 7-dimensional vector, and each dimension is, in order, the human pedaling torque, the driving speed, the rate of change of the human pedaling torque, the vehicle's X-axis acceleration, the vehicle's Y-axis acceleration, the vehicle's Z-axis acceleration, and the assist torque of the previous moment; both the Actor network and the Critic network take the temporal state matrix as input, and the parameters of the two networks are independent of each other.

3. The intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning according to claim 1, characterized in that, The Transformer encoder module includes a multi-head self-attention layer, a first residual connection and a first normalization layer, a feedforward network layer, a second residual connection and a second normalization layer connected in sequence; the multi-head self-attention layer is used to model the temporal dependencies between each moment in the continuous multi-frame historical observation sequence; the feedforward network layer is a two-layer fully connected mapping structure, with the hidden layer dimension being larger than the input dimension. The first residual connection and the second residual connection respectively add the input of the multi-head self-attention layer and the input of the feedforward network layer to their respective outputs, in order to preserve shallow feature information and alleviate gradient vanishing.

4. The intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning according to claim 1, characterized in that, In the bounded action parameterization, the strategy head outputs the action mean and shared standard deviation according to the temporal features. The action mean is mapped to the output range of the assist motor through the sigmoid activation function and initialized with a negative bias to make the initial output tend to zero. After sampling from a normal distribution whose mean is the mapping result and whose standard deviation is the shared standard deviation, values ​​less than the lower limit of the output boundary are truncated to the lower limit, and values ​​greater than the upper limit of the output boundary are truncated to the upper limit, so as to ensure that the assist torque is within the physical constraints of the assist motor.

5. The intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning according to claim 1, characterized in that, The policy network is trained using a multidimensional reward function, which includes: a torque matching reward term, a zero torque penalty term, a smoothing reward term, and a response reward term.

6. The intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning according to claim 5, characterized in that, The torque matching reward term is used to constrain the current output assist torque to approach the desired assist torque. The zero torque penalty term is used to constrain the current output assist torque to be zero when the human pedaling torque is less than a preset threshold. The smoothing reward term is used to constrain the current output assist torque to approach the previous output assist torque. The response reward term is used to constrain the deviation between the current output assist torque and the desired assist torque to be minimized.

7. The intelligent power assist control system for electric-assisted bicycles based on deep reinforcement learning according to claim 1, characterized in that, The strategy network is trained using a phased learning strategy, including: the first phase is trained under fixed speed conditions to learn torque tracking and smooth output; the second phase introduces speed variations to adapt the strategy to dynamic speed conditions; and the third phase is trained under randomized scenarios with all conditions, and introduces domain randomization of vehicle dynamic parameters, including gradient, total mass of rider and vehicle, rolling resistance coefficient, and air resistance coefficient.

8. A method for intelligent power assist control of electric-assisted bicycles based on deep reinforcement learning, characterized in that, The system according to any one of claims 1 to 7 comprises the following steps: Step 1: Collect data on pedal torque, speed, and three-axis acceleration of the vehicle body during the riding process, calculate the rate of change of human pedal torque, and construct a time-series observation vector together with the motor output torque at the previous moment. Step 2: Input the time-series observation vectors of multiple consecutive frames into the policy network. The policy network is an Actor network and a Critic network based on the near-end policy optimization algorithm. The Actor network and the Critic network each independently contain a Transformer encoder module. The Transformer encoder module models the historical time-series state. Step 3: The output of the strategy network adopts bounded action parameterization to output the assist torque: the strategy head outputs the action mean and shared standard deviation. The action mean is mapped to the allowable output range of the motor through the sigmoid activation function. After sampling, the action is clipped to ensure that the assist torque is kept within the allowable output range of the motor. Step four: Generate a control voltage based on the obtained assist torque to drive the assist motor to output assist.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the intelligent power assist control method for electric-assisted bicycles based on deep reinforcement learning as described in claim 8.

10. A computer-readable storage medium, characterized in that, It stores executable instructions, which, when executed by a processor, enable the processor to implement the intelligent power assist control method for electric-assisted bicycles based on deep reinforcement learning as described in claim 8.

Images