Reinforcement learning training methods, testing methods, and systems for electric bicycles

Abstract

This invention provides a reinforcement learning training method, testing method, and system for electric bicycles, comprising: inputting the acquired current path node and arrival time into a policy network to obtain the expected reward in the current state; calculating the state vector and reward function in the next state; inputting the state vector of the next state into the policy network and the evaluation network to obtain the path node and expected reward in the next state; calculating the loss value of the evaluation network based on the expected reward in the current state and the expected reward in the next state; updating the parameters of the evaluation network to minimize the loss value of the evaluation network; calculating the loss value of the policy network based on the target value in the loss value of the evaluation network and the policy gradient of the policy network; updating the network parameters of the policy network to minimize the loss value of the policy network; repeating the above steps until a preset number of training iterations or a preset performance index is reached to obtain the trained reinforcement learning algorithm.

Description

Technical Field

[0001] This invention relates to the field of control technology for electric vehicles, and in particular to reinforcement learning training methods, testing methods, and systems for electric vehicles. Background Technology

[0002] Currently, intelligent navigation algorithms for real-time traffic conditions take traffic flow into account, using it as a parameter that influences pheromones in traffic flow. This aims to guide vehicles to choose shorter, less congested routes, thus avoiding congested areas and balancing traffic load on the road network. While this algorithm uses traffic flow as a reference variable for adjustment, it is not suitable for electric bicycles. In most cases, electric bicycles can autonomously navigate through residential areas, alleyways, building gaps, and some sidewalks.

[0003] Pathless navigation algorithms using satellite positioning allow electric vehicles to have a clear destination and freely switch between driving modes (e.g., if the electric vehicle is temporarily parked on the roadside, it can switch to a person as the target or temporarily move to the motor vehicle lane, switching back to the vehicle's main route). However, satellite navigation algorithms cannot match the corresponding target. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a reinforcement learning training method, testing method and system for electric bicycles, which obtains a trained reinforcement learning algorithm through training, thereby guiding electric bicycles to pass through road sections that cars cannot pass; and uses the trained reinforcement learning algorithm to complete intelligent path selection, thereby improving user experience.

[0005] In a first aspect, embodiments of the present invention provide a reinforcement learning training method for electric bicycles, the method comprising:

[0006] Obtain the current path node of the electric vehicle and the time of arrival at the current path node;

[0007] The current path node and the time of arrival at the current path node are input into the policy network to obtain the expected reward in the current state;

[0008] Calculate the state vector and reward function for the next state;

[0009] The state vector of the next state is input into the policy network and the evaluation network to obtain the path nodes and the expected reward in the next state.

[0010] The loss value of the evaluation network is calculated based on the expected return in the current state and the expected return in the next state.

[0011] Update the parameters of the evaluation network to minimize the loss value of the evaluation network;

[0012] Calculate the policy gradient of the policy network;

[0013] The loss value of the policy network is calculated based on the target value in the loss value of the evaluation network and the policy gradient of the policy network.

[0014] Update the network parameters of the policy network to minimize the loss value of the policy network;

[0015] Repeat the above steps until the preset number of training iterations or preset performance metrics are reached, and the trained reinforcement learning algorithm is obtained.

[0016] Furthermore, the state vector and reward function for the next state are calculated, including:

[0017] Obtain the next longitude, next latitude, next speed, and next battery charge of the electric vehicle;

[0018] Calculate the state vector in the next state based on the next longitude, the next latitude, the next speed, and the next battery charge;

[0019] Obtain the time when the electric vehicle arrives at its destination;

[0020] The reward function is based on the negative value of the time it takes for the electric vehicle to reach the destination.

[0021] Furthermore, based on the expected return in the current state and the expected return in the next state, the loss value of the evaluation network is calculated, including:

[0022] The loss value of the evaluation network is calculated according to the following formula:

[0023] δ=r+γV′(s′)-V(s)

[0024] Where δ is the loss value of the evaluation network, r and γ are hyperparameters, V′(s′) is the expected reward in the next state, and V(s) is the expected reward in the current state.

[0025] Further, calculating the policy gradient of the policy network includes:

[0026] Obtain the current longitude, current latitude, current speed, and current battery level of the electric vehicle;

[0027] Calculate the state vector in the current state based on the current longitude, the current latitude, the current speed, and the current battery level;

[0028] Obtain the current action parameters of the electric vehicle, including acceleration, deceleration, turning, and charging;

[0029] Calculate the action vector in the current state based on the acceleration, deceleration, turning, and charging.

[0030] The policy gradient of the policy network is calculated based on the state vector and action vector in the current state.

[0031] Furthermore, the policy gradient of the policy network is calculated based on the state vector and action vector in the current state, including:

[0032] The policy gradient of the policy network is calculated according to the following formula:

[0033]

[0034] Where PG is the policy gradient of the policy network, a is the action vector in the current state, and s is the state vector in the current state.

[0035] Further, based on the target value in the loss value of the evaluation network and the policy gradient of the policy network, the loss value of the policy network is calculated, including:

[0036] The loss value of the policy network is calculated according to the following formula:

[0037] loss=-(r+γV′(s′))logπ(a|s)

[0038] Where r+γV′(s′) is the target value, and loss is the loss value of the policy network.

[0039] Secondly, embodiments of the present invention provide a reinforcement learning testing method for electric bicycles, the method comprising:

[0040] Collect vehicle driving information;

[0041] The vehicle driving information is identified to obtain road condition information;

[0042] The road condition information is input into the trained reinforcement learning algorithm to obtain a path decision;

[0043] The path decision is displayed.

[0044] Thirdly, embodiments of the present invention provide a reinforcement learning training system for electric bicycles, applied to a development board, the system comprising:

[0045] The current path node and time acquisition module is used to acquire the current path node of the electric vehicle and the time of arrival at the current path node;

[0046] The current state expected reward acquisition module is used to input the current path node and the time of arrival at the current path node into the policy network to obtain the expected reward in the current state.

[0047] The state vector calculation module is used to calculate the state vector and reward function for the next state.

[0048] The next state path node and expected reward acquisition module is used to input the state vector of the next state into the policy network and the evaluation network to obtain the path node and expected reward of the next state.

[0049] The evaluation network loss value calculation module is used to calculate the loss value of the evaluation network based on the expected return in the current state and the expected return in the next state.

[0050] The evaluation network parameter update module is used to update the parameters of the evaluation network so that the loss value of the evaluation network is minimized.

[0051] The policy gradient calculation module is used to calculate the policy gradient of the policy network.

[0052] The policy network loss value calculation module is used to calculate the loss value of the policy network based on the target value in the loss value of the evaluation network and the policy gradient of the policy network.

[0053] The policy network parameter update module is used to update the network parameters of the policy network to minimize the loss value of the policy network.

[0054] The training module is used to repeat the above steps. When the preset number of training iterations or preset performance metrics are reached, the trained reinforcement learning algorithm is obtained.

[0055] Fourthly, embodiments of the present invention provide a reinforcement learning testing system for electric bicycles, the system comprising:

[0056] Data acquisition device, used to collect vehicle driving information;

[0057] The development board is used to identify the vehicle's driving information to obtain road condition information; the road condition information is then input into a trained reinforcement learning algorithm to obtain a path decision.

[0058] An information transmission device is used to display the path decision.

[0059] Fifthly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described above.

[0060] This invention provides a reinforcement learning training method, testing method, and system for electric bicycles, including: obtaining the current path node and arrival time of the electric bicycle; inputting the current path node and arrival time into a policy network to obtain the expected reward in the current state; calculating the state vector and reward function for the next state; inputting the state vector of the next state into the policy network and evaluation network to obtain the path node and expected reward in the next state; calculating the loss value of the evaluation network based on the expected reward in the current state and the expected reward in the next state; updating the parameters of the evaluation network to minimize the loss value of the evaluation network; and calculating the policy network... The process involves: calculating the policy gradient; calculating the loss value of the policy network based on the target value in the evaluation network's loss value and the policy gradient of the policy network; updating the network parameters of the policy network to minimize the loss value; repeating the above steps until a preset number of training iterations or a preset performance index is reached to obtain the trained reinforcement learning algorithm; collecting vehicle driving information; identifying the vehicle driving information to obtain road condition information; inputting the road condition information into the trained reinforcement learning algorithm to obtain path decisions; training the trained reinforcement learning algorithm to guide the electric vehicle through road sections that cars cannot pass; and using the trained reinforcement learning algorithm to complete intelligent path selection, thereby improving the user experience.

[0061] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0063] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0064] Figure 1 This is a flowchart of the reinforcement learning training method for an electric vehicle provided in Embodiment 1 of the present invention;

[0065] Figure 2 This is a schematic diagram of the reinforcement learning training process provided in Embodiment 1 of the present invention;

[0066] Figure 3This is a flowchart of the reinforcement learning testing method for electric vehicles provided in Embodiment 2 of the present invention;

[0067] Figure 4 This is a schematic diagram of the reinforcement learning training system for an electric vehicle provided in Embodiment 3 of the present invention;

[0068] Figure 5 This is a schematic diagram of the reinforcement learning testing system for electric vehicles provided in Embodiment 4 of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0071] Example 1:

[0072] Figure 1 This is a flowchart of the reinforcement learning training method for electric vehicles provided in Embodiment 1 of the present invention.

[0073] Reference Figure 1 The method includes the following steps:

[0074] Step S101: Obtain the current path node of the electric vehicle and the time of arrival at the current path node;

[0075] Here, before obtaining the current path node of the electric vehicle and the time of arrival at the current path node, it is necessary to initialize the network parameters. The network parameters are randomly selected data. Since these data have losses, it is necessary to train on these data to reduce the losses.

[0076] Step S102: Input the current path node and the time to reach the current path node into the policy network to obtain the expected reward in the current state;

[0077] Step S103: Calculate the state vector and reward function for the next state;

[0078] Here, we perform an action (i.e., reach the current path node p), observe the environmental feedback, and calculate the state vector and reward function for the next state.

[0079] Step S104: Input the state vector of the next state into the policy network and the evaluation network to obtain the path nodes and expected rewards in the next state.

[0080] Step S105: Calculate the loss value of the evaluation network based on the expected return in the current state and the expected return in the next state.

[0081] Step S106: Update the parameters of the evaluation network to minimize the loss value of the evaluation network;

[0082] Here, refer to Figure 2 If the expected return in the current state approaches the target value r+γV′(s′), then the loss value δ of the evaluation network reaches its minimum.

[0083] Step S107: Calculate the policy gradient of the policy network;

[0084] Step S108: Calculate the loss value of the policy network based on the target value in the loss value of the evaluation network and the policy gradient of the policy network.

[0085] Step S109: Update the network parameters of the policy network to minimize the loss value of the policy network;

[0086] Here, the network parameters of the policy network are updated to maximize the probability of the current path node, that is, to minimize the loss value of the policy network.

[0087] Step S110: Repeat the above steps until the preset number of training iterations or preset performance metrics are reached, and the trained reinforcement learning algorithm is obtained.

[0088] Furthermore, step S103 includes the following steps:

[0089] Step S201: Obtain the next longitude, next latitude, next speed, and next battery charge of the electric vehicle;

[0090] Step S202: Calculate the state vector for the next state based on the next longitude, next latitude, next speed, and next battery charge.

[0091] Step S203: Obtain the time when the electric vehicle arrives at the destination;

[0092] Step S204: Use the negative value of the time it takes for the electric vehicle to reach the destination as the reward function.

[0093] Furthermore, step S105 includes:

[0094] Calculate the loss value of the evaluation network according to formula (1):

[0095] δ=r+γV′(s′)-V(s) (1)

[0096] Where δ is the loss value of the evaluation network, r and γ are hyperparameters, V′(s′) is the expected reward in the next state, and V(s) is the expected reward in the current state.

[0097] Furthermore, step S107 includes:

[0098] Step S301: Obtain the current longitude, current latitude, current speed, and current battery level of the electric vehicle;

[0099] Step S302: Calculate the state vector under the current state based on the current longitude, current latitude, current speed, and current battery level;

[0100] Step S303: Obtain the current action parameters of the electric vehicle, including acceleration, deceleration, turning, and charging.

[0101] Step S304: Calculate the action vector in the current state based on acceleration, deceleration, turning, and charging;

[0102] Step S305: Calculate the policy gradient of the policy network based on the state vector and action vector in the current state.

[0103] Furthermore, step S305 includes:

[0104] Calculate the policy gradient of the policy network according to formula (2):

[0105]

[0106] Where PG is the policy gradient of the policy network, a is the action vector in the current state, s is the state vector in the current state, and π(a|s) represents the action probability distribution output by the policy network (actor).

[0107] Furthermore, step S108 includes:

[0108] Calculate the loss value of the policy network according to formula (3):

[0109] loss=-(r+γV′(s′))logπ(a|s) (3)

[0110] Where r+γV′(s′) is the target value, and loss is the loss value of the policy network.

[0111] The process of constructing a reinforcement learning algorithm is as follows:

[0112] Algorithm Data Preparation: Training data is collected, including a path map, start and end points, and features of each path node (such as traffic flow and road width). Each path node is labeled to indicate whether it is on the optimal path. Entrances and exits for each residential area, alleyway, parking lot, etc., within the area are also labeled and added to the path map nodes. The labeling of each path node's optimal path status can be done manually or using existing path planning algorithms. The dataset is divided into training and testing sets. The training set is used to train the neural network model, and the testing set is used to validate the model's performance and generalization ability.

[0113] Algorithm State Definition: The state of an electric vehicle driving problem can include information such as the vehicle's position, speed, and battery level. For example, a vector can be used to represent the vehicle's state at a given moment. This vector includes information such as the vehicle's latitude and longitude, speed, and battery level. These are combined into a state vector, which is then returned to the reinforcement learning model. The state vector can be represented as s = (latitude, longitude, speed, battery). It is important to note that the state definition and the design of the generator function need to be adjusted and optimized based on the specific problem and data to ensure that the state vector contains sufficient information to reflect all or part of the environment's state.

[0114] The algorithm defines actions: Actions of an electric vehicle can include acceleration, deceleration, turning, and charging. For example, acceleration can be defined as an action, and the electric vehicle's acceleration can be considered one of these actions. Therefore, the action vector can be represented as a = (acceleration, deceleration, turn, charge). These actions are then combined into an action vector and returned to the reinforcement learning model. The design of the action definition and generation function needs to be adjusted and optimized based on the electric vehicle's driving area and data to ensure that the action vector includes all actions the agent can take and reflects all or part of the environment's state. Furthermore, the discreteness or continuity of actions, as well as factors such as the magnitude and direction of the actions, should be incorporated.

[0115] The algorithm defines a reward function: based on the goal of minimizing time, the reward function can be defined as the negative of the time required to reach the destination. Additionally, other factors, such as battery level and vehicle speed, can be considered to influence the reward function.

[0116] Designing neural networks: This problem can be solved using deep reinforcement learning algorithms, such as Deep Q Network (DQN). The structure, number of layers, activation functions, etc., of the neural network need to be selected and adjusted according to the specific problem.

[0117] This application provides an example of an actor-critic reinforcement learning algorithm, but is not limited to it, as detailed below:

[0118] The reward function measures the feedback from the environment after an agent performs an action and is one of the main criteria for evaluating agent behavior. In the electric vehicle driving problem, we want the vehicle to reach its destination as quickly as possible. Therefore, we can use the shortest time as the objective and design a reward function to encourage the agent to perform actions that minimize travel time.

[0119] Assuming that at each time step t, the distance traveled by the vehicle is d(t), the speed is V(t), and the target distance is D, then the reward function can be obtained from formula (4):

[0120] R(t)=-(Dd(t)) / V(t) (4)

[0121] The farther the vehicle is from the target, or the slower the vehicle's speed, the lower the reward value; conversely, the closer the vehicle is to the target, or the faster the vehicle's speed, the higher the reward value. Therefore, the agent will tend to take actions that allow it to quickly approach the target and increase its speed in order to obtain a higher reward.

[0122] Neural networks: This problem can be solved using deep reinforcement learning algorithms, such as Deep Q Network (DQN). The design of the neural network's structure, number of layers, activation functions, etc., needs to be selected and adjusted according to the specific problem.

[0123] Below is a simple example of a neural network design, but it is not limited to this:

[0124] Input: Vehicle state s, including vehicle speed v, battery level b, etc. The state can be represented as a vector s = [v, b].

[0125] Output: Path node p, and the time t required to reach that node. The output can be represented as a vector [p,t].

[0126] Network structure: A state-based actor-critic algorithm can be used, which includes a policy (actor) network and an evaluation (critic) network.

[0127] Policy network: Takes vehicle state as input and outputs path node p and the time t required to reach that node, representing the decision of the intelligent algorithm.

[0128] Evaluation network: Takes vehicle state and path nodes as input and outputs a value function V, which represents the expected reward of performing the action in the current state.

[0129] Optimization objective: Minimize the time cost function J, i.e., the negative value of the expected travel time.

[0130] Training the model: Using training data and reinforcement learning algorithms, a neural network model is trained to continuously optimize its performance and generalization ability. During training, appropriate hyperparameters, loss functions, and optimizers need to be selected.

[0131] By utilizing training data and reinforcement learning strategies, neural network models are trained to continuously optimize their performance and generalization ability. During training, techniques such as adding hidden layers, adjusting the number of neurons and activation functions, and adding regularization can improve the model's generalization ability and learning efficiency. Appropriate hyperparameters, loss functions, and optimizers need to be selected. The methods for constructing neural networks are well-known in the field of intelligent algorithms and will not be elaborated upon here.

[0132] Example 2:

[0133] Figure 3 This is a flowchart of the reinforcement learning testing method for electric vehicles provided in Embodiment 2 of the present invention.

[0134] Reference Figure 3 The method includes the following steps:

[0135] Step S401: Collect vehicle driving information;

[0136] Step S402: Identify vehicle driving information to obtain road condition information;

[0137] Step S403: Input the road condition information into the trained reinforcement learning algorithm to obtain the path decision;

[0138] Step S404: Display the path decision.

[0139] Using test data and evaluation metrics, the model's performance is tested and evaluated, and optimization and adjustments are made based on the evaluation results. The optimal model can be found by comparing the performance of different algorithms, network structures, and hyperparameters. Finally, the model is integrated with map data and real-time traffic data for practical testing, and the algorithm is continuously optimized during use to achieve a practical and cited level.

[0140] Compared to traditional planning and optimization algorithms, post-training reinforcement learning algorithms have the following advantages:

[0141] It does not require prior knowledge of the road network and travel time on each road segment, and can adaptively adapt to different road environments and traffic conditions. It can adjust routes and speeds in real time to avoid the impact of congestion and road conditions on travel time, improving route stability and reliability. It can also utilize historical data for learning and optimization, further enhancing the accuracy and efficiency of route planning.

[0142] Therefore, the reinforcement learning algorithm after training can bring more flexible, efficient and accurate path planning results, and has been widely used in fields such as intelligent driving of electric vehicles, shared mobility and smart transportation.

[0143] This application employs a post-trained reinforcement learning algorithm to design a path planning model for electric bicycles. This model combines factors such as vehicle state, speed, and battery level, and through continuous interaction and optimization with the environment, it can automatically learn the optimal path planning strategy. In this application, real road data and electric bicycle driving data collected in recent years are used to verify the effectiveness and accuracy of the model. Compared with traditional planning and optimization algorithms, this model enables electric bicycles to better adapt to complex road environments and traffic conditions during operation, and can adaptively plan and adjust paths, thereby achieving more efficient and accurate shortest path planning.

[0144] Furthermore, this application can also employ other neural network algorithms or traditional machine learning algorithms for algorithm application, and utilize different forms of deep learning algorithms or similar reinforcement learning algorithms for scenario application.

[0145] Example 3:

[0146] Figure 4 This is a schematic diagram of the reinforcement learning training system for an electric vehicle provided in Embodiment 3 of the present invention.

[0147] Reference Figure 4 Applied to development boards, the system includes:

[0148] The current path node and time acquisition module is used to obtain the current path node of the electric vehicle and the time of arrival at the current path node;

[0149] The current state expected reward acquisition module is used to input the current path node and the time of arrival at the current path node into the policy network to obtain the expected reward in the current state.

[0150] The state vector calculation module is used to calculate the state vector and reward function for the next state.

[0151] The next state path node and expected reward acquisition module is used to input the state vector of the next state into the policy network and the evaluation network to obtain the path node and expected reward of the next state.

[0152] The evaluation network loss value calculation module is used to calculate the loss value of the evaluation network based on the expected return in the current state and the expected return in the next state.

[0153] The evaluation network parameter update module is used to update the parameters of the evaluation network to minimize the loss value of the evaluation network.

[0154] The policy gradient calculation module is used to calculate the policy gradient of the policy network.

[0155] The policy network loss value calculation module is used to calculate the loss value of the policy network based on the target value in the loss value of the evaluation network and the policy gradient of the policy network.

[0156] The policy network parameter update module is used to update the network parameters of the policy network to minimize the loss value of the policy network.

[0157] The training module is used to repeat the above steps. When the preset number of training iterations or preset performance metrics are reached, the trained reinforcement learning algorithm is obtained.

[0158] Example 4:

[0159] Figure 5 This is a schematic diagram of the reinforcement learning testing system for electric vehicles provided in Embodiment 4 of the present invention.

[0160] Reference Figure 5 The system includes:

[0161] Data acquisition device, used to collect vehicle driving information;

[0162] The development board is used to identify vehicle driving information to obtain road condition information; the road condition information is then input into the trained reinforcement learning algorithm to obtain path decisions.

[0163] Information transmission equipment used to display route decisions.

[0164] Specifically, the system also includes a power supply device and an information transmission device. The development board serves as the algorithm carrier, and the data acquisition device collects vehicle driving information (environmental interaction data, such as road conditions, weather, traffic jams, and traffic lights) during the electric vehicle's operation and connects to the development board (algorithm carrier). The power supply device provides continuous power to the device and is connected to the development board. The information transmission device, connected to the development board, obtains route decisions and guides the user on how to execute them.

[0165] This invention provides a reinforcement learning training method, testing method, and system for electric bicycles, including: obtaining the current path node and arrival time of the electric bicycle; inputting the current path node and arrival time into a policy network to obtain the expected reward in the current state; calculating the state vector and reward function for the next state; inputting the state vector of the next state into the policy network and evaluation network to obtain the path node and expected reward in the next state; calculating the loss value of the evaluation network based on the expected reward in the current state and the expected reward in the next state; updating the parameters of the evaluation network to minimize the loss value of the evaluation network; and calculating the policy network... The process involves: calculating the policy gradient; calculating the loss value of the policy network based on the target value in the evaluation network's loss value and the policy gradient of the policy network; updating the network parameters of the policy network to minimize the loss value; repeating the above steps until a preset number of training iterations or a preset performance index is reached to obtain the trained reinforcement learning algorithm; collecting vehicle driving information; identifying the vehicle driving information to obtain road condition information; inputting the road condition information into the trained reinforcement learning algorithm to obtain path decisions; training the trained reinforcement learning algorithm to guide the electric vehicle through road sections that cars cannot pass; and using the trained reinforcement learning algorithm to complete intelligent path selection, thereby improving the user experience.

[0166] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the reinforcement learning training method and testing method for electric vehicles provided in the above embodiments.

[0167] This invention also provides a computer-readable medium having processor-executable non-volatile program code, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the reinforcement learning training method and testing method for electric vehicles described above.

[0168] The computer program product provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0170] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0171] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0173] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reinforcement learning training method for electric bicycles, characterized in that, The method includes: Obtain the current path node of the electric vehicle and the time of arrival at the current path node; The current path node and the time of arrival at the current path node are input into the policy network to obtain the expected reward in the current state; Obtain the next longitude, next latitude, next speed, and next battery charge of the electric vehicle; Calculate the state vector for the next state based on the next longitude, the next latitude, the next speed, and the next battery charge. Obtain the time when the electric vehicle arrives at its destination; The reward function is based on the negative value of the time it takes for the electric vehicle to reach the destination. The state vector of the next state is input into the policy network and the evaluation network to obtain the path nodes and the expected reward in the next state. The loss value of the evaluation network is calculated based on the expected return in the current state and the expected return in the next state. Update the parameters of the evaluation network to minimize the loss value of the evaluation network; Obtain the current longitude, current latitude, current speed, and current battery level of the electric vehicle; Calculate the state vector in the current state based on the current longitude, the current latitude, the current speed, and the current battery level; Obtain the current action parameters of the electric vehicle, including acceleration, deceleration, turning, and charging; Calculate the action vector in the current state based on the acceleration, deceleration, turning, and charging. The policy gradient of the policy network is calculated based on the state vector and action vector in the current state. The loss value of the policy network is calculated based on the target value in the loss value of the evaluation network and the policy gradient of the policy network. Update the network parameters of the policy network to minimize the loss value of the policy network; Repeat the above steps until the preset number of training iterations or preset performance indicators are reached, and the trained reinforcement learning algorithm is obtained. The trained reinforcement learning algorithm is used to guide the electric vehicle to pass through road sections that cars cannot pass, and complete the selection of intelligent paths.

2. The reinforcement learning training method for electric bicycles according to claim 1, characterized in that, The loss value of the evaluation network is calculated based on the expected return in the current state and the expected return in the next state, including: The loss value of the evaluation network is calculated according to the following formula: in, The loss value of the evaluation network is given. and For hyperparameters, The expected return in the next state. This represents the expected return under the current state.

3. The reinforcement learning training method for electric bicycles according to claim 1, characterized in that, Calculate the policy gradient of the policy network based on the state vector and action vector in the current state, including: The policy gradient of the policy network is calculated according to the following formula: in, The policy gradient of the policy network is... Let this be the action vector in the current state. Let be the state vector in the current state. This represents the probability distribution of actions output by the policy network actor.

4. The reinforcement learning training method for electric bicycles according to claim 1, characterized in that, Based on the target value in the loss value of the evaluation network and the policy gradient of the policy network, the loss value of the policy network is calculated, including: The loss value of the policy network is calculated according to the following formula: in, For the target value, The loss value of the policy network. and For hyperparameters, The expected return in the next state, Let this be the action vector in the current state. Let be the state vector in the current state. This represents the probability distribution of actions output by the policy network actor.

5. A reinforcement learning testing method for electric bicycles, characterized in that, The method includes: Collect vehicle driving information; The vehicle driving information is identified to obtain road condition information; The road condition information is input into the trained reinforcement learning algorithm to obtain a path decision; wherein the trained reinforcement learning algorithm is trained by the reinforcement learning training method for the electric vehicle described in any one of claims 1-4; The path decision is displayed.

6. A reinforcement learning training system for an electric bicycle, characterized in that, The system, applied to a development board, includes: The current path node and time acquisition module is used to acquire the current path node of the electric vehicle and the time of arrival at the current path node; The current state expected reward acquisition module is used to input the current path node and the time of arrival at the current path node into the policy network to obtain the expected reward in the current state. The state vector calculation module is used to obtain the next longitude, next latitude, next speed, and next battery level of the electric vehicle; calculate the state vector of the next state based on the next longitude, next latitude, next speed, and next battery level; obtain the time when the electric vehicle reaches the destination; and use the negative value of the time when the electric vehicle reaches the destination as the reward function. The next state path node and expected reward acquisition module is used to input the state vector of the next state into the policy network and the evaluation network to obtain the path node and expected reward of the next state. The evaluation network loss value calculation module is used to calculate the loss value of the evaluation network based on the expected return in the current state and the expected return in the next state. The evaluation network parameter update module is used to update the parameters of the evaluation network so that the loss value of the evaluation network is minimized. The policy gradient calculation module is used to obtain the current longitude, current latitude, current speed, and current battery level of the electric vehicle; calculate the state vector of the current state based on the current longitude, current latitude, current speed, and current battery level; obtain the current action parameters of the electric vehicle, including acceleration, deceleration, turning, and charging; calculate the action vector of the current state based on the acceleration, deceleration, turning, and charging; and calculate the policy gradient of the policy network based on the state vector and the action vector of the current state. The policy network loss value calculation module is used to calculate the loss value of the policy network based on the target value in the loss value of the evaluation network and the policy gradient of the policy network. The policy network parameter update module is used to update the network parameters of the policy network to minimize the loss value of the policy network. The training module is used to repeat the above steps. When the preset number of training iterations or preset performance metrics are reached, the trained reinforcement learning algorithm is obtained.

7. A reinforcement learning testing system for electric bicycles, characterized in that, The system includes: Data acquisition device, used to collect vehicle driving information; The development board is used to identify the vehicle's driving information to obtain road condition information; the road condition information is input into a trained reinforcement learning algorithm to obtain a path decision; wherein the trained reinforcement learning algorithm is trained by the reinforcement learning training method for the electric vehicle as described in any one of claims 1-4. An information transmission device is used to display the path decision.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 5.

Images