Shift strategy generation method and device, electronic equipment and storage medium

Abstract

The application provides a gear shifting strategy generation method and device, electronic equipment and storage medium, and belongs to the technical field of transportation. The method comprises the following steps: establishing a first model based on the correspondence relationship between preset variables and vehicle gear states, wherein the variables comprise state variables and action variables; establishing a step-by-step guidance reward function based on a preset gear shifting strategy target; obtaining a gear shifting curve based on a phased training method and road profile data of different speed gradients; determining a second model based on a deep reinforcement learning method, the step-by-step guidance reward function, the first model and the gear shifting curve; and generating a final gear shifting strategy based on the second model. The problem of low generalization of the gear shifting strategy obtained by the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of transportation technology, and in particular to a method, apparatus, electronic device, and storage medium for generating gear shifting strategies. Background Technology

[0002] The shifting strategy determines the optimal gear and shift timing based on the vehicle's condition, driving environment, and driver intent. This directly impacts the performance of the AMT (Automated Manual Transmission) and also affects the vehicle's power, comfort, and fuel economy. Existing technologies typically employ graphical methods, analytical methods, genetic algorithms, and dynamic programming to solve for shifting strategies.

[0003] With the development of computer technology and intelligent optimization algorithms such as deep reinforcement learning, reinforcement learning algorithms can be used to train a shifting strategy with good overall performance according to different demand scenarios. Existing technology has disclosed a dynamic optimization method for shifting strategies based on deep reinforcement learning. It uses state variables (such as vehicle speed, acceleration, and throttle opening) and action variables (such as gear position) to determine the Markov decision process of the shifting strategy and uses the DQN algorithm for solving it.

[0004] However, the training conditions of this method are relatively simple, and the dynamic model is also relatively simple. Therefore, the generalization ability of the shift strategy obtained by training is not high enough, and its performance in actual driving conditions needs further verification and improvement. Summary of the Invention

[0005] In view of this, it is necessary to provide a shift strategy generation method, apparatus, electronic device, and readable storage medium to solve the problem of low generalization of shift strategies obtained by the prior art.

[0006] To address the above problems, this invention provides a method for generating a shift strategy, comprising:

[0007] A first model is established based on the pre-defined correspondence between variables and vehicle gear status, wherein the variables include state variables and action variables;

[0008] A tiered reward function is established based on the preset shift strategy objective;

[0009] The shift curve is obtained based on a phased training method and road spectrum data with different speed gradients;

[0010] The second model is determined based on the deep reinforcement learning method, the step-guided reward function, the first model, and the shift curve;

[0011] The final shifting strategy is generated based on the second model.

[0012] In one possible implementation, the state variables include at least: vehicle speed, acceleration, engine torque demand, fuel consumption per unit distance, and engine speed;

[0013] The action variables include at least the modification values ​​of the shift speed of the shift curve under each fixed required torque.

[0014] In one possible implementation, the first model includes:

[0015]

[0016] Where GearState represents the vehicle's gear position, v represents the current actual vehicle speed, and up th and indicate the upshift speed, down th This indicates the downshift speed, Upshift represents the lookup function for upshifting, Downshift represents the lookup function for downshifting, Gear represents the current gear, and T represents the downshift speed. cmd The engine torque requirement is represented by Act(), and the action taken by the agent is represented by Δv. 1→2 The value Δv represents the modification of the shift speed for the 1st to 2nd gear shift curve. i→i+1 This represents the modification value of the shift speed for the gear progression curve from gear i to gear i+1; Δv 2→1 The value Δv represents the modification of the shift speed for the 2nd to 1st gear downshift curve. i+1→i This represents the modification value for the shift speed of the i+1 to i gear downshift curve.

[0017] In one possible implementation, the preset shift strategy objectives include: speed following, fuel consumption reduction, engine speed range limitation, running time, and overlap with prior knowledge.

[0018] In one possible implementation, the step-guided reward function includes:

[0019] R all =R v +R fuel +R fuel_spd +R t +R act

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] Among them, R all R represents the cumulative reward. v Indicates speed-following reward, R fuel Indicates a fuel consumption reduction reward, R fuel_spd Indicates the engine speed range limit reward, R t R represents the runtime reward. act Represents a reward for prior knowledge, v trgt V represents the target velocity. act This indicates the actual vehicle speed, and fuel indicates the fuel consumption per unit distance. before This indicates the fuel consumption per unit distance in the previous round, Eng. spd Indicates engine speed, t all n represents the time required to complete one round of road spectrum simulation. same n represents the number of gear selection decisions made with the same prior knowledge. all This represents the total number of gear decisions made in one round of road spectrum simulation.

[0026] In one possible implementation, the second model includes:

[0027] Action-value function network Q(s) i ,a i ;θ Q ), used to obtain cumulative rewards based on state data and action data, where s i Represents state data, a i Represents action data, θ Q This represents the network parameters of the action-value function network;

[0028] Deterministic policy network μ(s) i ;θ μ ), used to determine the action data that yields the maximum cumulative reward under different state data, where θ μ Represents the network parameters of a deterministic policy network;

[0029] Target policy network Q′(s) i ,μ′(s i ;θ′ μ );θ′ Q ), used to evaluate the cumulative reward obtainable from action data under different state data, where θ′ Q and θ′ μ All of these represent the network parameters of the target policy network.

[0030] In one possible implementation, determining the second model based on the step-guided reward function, the first model, and the shift curve includes:

[0031] Step 1: Obtain the initialized state data and noise and input them into the deterministic policy network;

[0032] Step 2: Obtain action data based on the first formula, and calculate the cumulative reward corresponding to the current action data and state data based on the step-guided reward function;

[0033] Step 3: Obtain the current shifting strategy based on the action data, the first model, and the shift curve;

[0034] Step 4: Obtain the status data corresponding to the current shifting strategy, and store the accumulated reward and status data as experience data in a preset replay area;

[0035] Step 5: Randomly sample several empirical data points from the replay area, input them into the target policy network, and calculate the target value based on the second formula;

[0036] Step 6: Calculate the loss L of the action value function network based on the target value and the third formula, and update the network parameters of the action value function network based on the gradient descent method;

[0037] Step 7: Update the network parameters of the deterministic policy network based on the gradient ascent method;

[0038] Step 8: Based on the fourth formula, calculate the network parameters θ′ of the target policy network. Q and θ′ μ The model is updated to obtain a second model, based on the updated deterministic policy network and state data s. i+1 Determine the new motion data and proceed to step 2;

[0039] The first formula includes:

[0040] a i =μ(s) i ;θ μ )+noise

[0041] Here, "noise" refers to noise.

[0042] The second formula includes:

[0043] y i =r i +γ*Q′(s i+1 ,μ′(s i+1 ;θ′ μ );θ′ Q )

[0044] Among them, y i Represents the target value, r iThis represents the cumulative reward corresponding to the current action data and state data, γ represents the preset weight value, and s i+1 This indicates the status data corresponding to the current shifting strategy;

[0045] The third formula includes:

[0046]

[0047] Where L represents the loss degree of the action value function network, and N represents the batch of sampled empirical data.

[0048] The fourth formula includes:

[0049]

[0050] in, All of these represent the network parameters used by the target policy network in this round of training. All represent the network parameters after the target policy network is updated, and τ represents the preset weight value.

[0051] The present invention also provides a shift strategy generation apparatus, the apparatus comprising:

[0052] The first model building module is used to build a first model based on the correspondence between preset variables and vehicle gear states, wherein the variables include state variables and action variables;

[0053] The stepped guided reward function module is used to establish a stepped guided reward function based on a preset shift strategy objective;

[0054] The curve optimization module is used to obtain shift curves based on a phased training method and road spectrum data with different speed gradients.

[0055] The second model building module is used to determine the second model based on the step-guided reward function, the first model, and the shift curve.

[0056] The strategy generation module is used to generate the final shifting strategy based on the second model.

[0057] The present invention also provides an electronic device, comprising:

[0058] Memory, used to store programs;

[0059] A processor, coupled to the memory, is configured to execute the program stored in the memory to implement the steps in the shift strategy generation method described in any of the above method items.

[0060] The present invention also provides a storage medium for storing a computer-readable program or instruction, wherein when the program or instruction is executed by a processor, it is able to implement the steps in the shift strategy generation method described in any one of the above method items.

[0061] The beneficial effects of this invention are as follows: This invention provides a method for generating shifting strategies. First, a first model is established based on the correspondence between state variables, action variables, and shifting states to better guide the agent to generate shifting decisions at appropriate times. Then, a step-wise guided reward function is designed to guide the agent to obtain the global optimal solution. Furthermore, a phased pre-training method is used to correct the shifting curve, resulting in a shifting curve with better accuracy and generality. Finally, deep learning methods, reward functions, shifting curves, and the first model are used to guide the agent to better learn the shifting strategy, resulting in a fully trained second model. This second model can generate the most suitable shifting strategy under different situations, solving the problem of low generalization of shifting strategies obtained by existing technologies. Attached Figure Description

[0062] Figure 1 A flowchart illustrating an embodiment of the shift strategy generation method provided by the present invention;

[0063] Figure 2 for Figure 1 A flowchart of a method according to an embodiment of step S104;

[0064] Figure 3 A schematic diagram of an embodiment of the shift strategy generation device provided by the present invention;

[0065] Figure 4 A schematic diagram of the structure of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0066] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0067] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0068] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0069] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0070] A specific embodiment of the present invention, such as Figure 1 As shown, a method for generating a shift strategy is disclosed, including:

[0071] Step S101: Establish a first model based on the preset correspondence between variables and vehicle gear status;

[0072] Step S102: Establish a tiered guided reward function based on the preset shift strategy target;

[0073] Step S103: Obtain the shift curve based on the phased training method and road spectrum data with different speed gradients;

[0074] Step S104: Determine the second model based on deep reinforcement learning methods, a step-guided reward function, the first model, and the shift curve;

[0075] Step S105: Generate the final shifting strategy based on the second model.

[0076] It should be noted that the subject of this invention is an intelligent agent, and in all embodiments of this invention, no specific form of the intelligent agent is limited.

[0077] Specifically, in this embodiment, the process of obtaining the shift curve can be as follows: first, train 100 rounds using random road spectrum data of v∈(0~15)km / h to complete the shift curve correction for 1st to 2nd gear; train 100 rounds using random road spectrum data of v∈(15~25)km / h to complete the shift curve correction for 2nd to 3rd gear; train 100 rounds using random road spectrum data of v∈(25~35)km / h to complete the shift curve correction for 3rd to 4th gear; train 100 rounds using random road spectrum data of v∈(45~55)km / h to complete the shift curve correction for 4th to 5th gear; and train 100 rounds using random road spectrum data of v∈(55~95)km / h to complete the shift curve correction for 5th to 6th gear.

[0078] Unlike existing technologies, this invention provides a method for generating shifting strategies. First, a first model is established based on the correspondence between state variables, action variables, and shifting states to better guide the agent to generate shifting decisions at appropriate times. Then, a stepped guided reward function is designed to guide the agent to obtain the global optimal solution. Furthermore, a phased pre-training method is used to correct the shifting curve, resulting in a shifting curve with better accuracy and generality. Finally, deep learning methods, reward functions, shifting curves, and the first model are used to guide the agent to better learn the shifting strategy, resulting in a fully trained second model. This second model can generate the most suitable shifting strategy under different situations, solving the problem of low generalization of shifting strategies obtained by existing technologies.

[0079] In one possible implementation, in order to enhance the generalization of the final shift strategy, the state variables include at least: vehicle speed, acceleration, engine torque demand, fuel consumption per unit distance, and engine speed.

[0080] The action variables include at least the modification values ​​of the shift speed on the shift curve under each fixed required torque.

[0081] In one possible implementation, the first model includes:

[0082]

[0083] In equation (1), GearState represents the vehicle's gear state, v represents the current actual vehicle speed, and up th and indicate the upshift speed, down th This indicates the downshift speed, Upshift represents the lookup function for upshifting, Downshift represents the lookup function for downshifting, Gear represents the current gear, and T represents the downshift speed. cmd The engine torque requirement is represented by Act(), and the action taken by the agent is represented by Δv. 1→2The value Δv represents the modification of the shift speed for the 1st to 2nd gear shift curve. i→i+1 This represents the modification value of the shift speed for the gear progression curve from gear i to gear i+1; Δv 2→1 The value Δv represents the modification of the shift speed for the 2nd to 1st gear downshift curve. i+1→i This represents the modification value for the shift speed of the i+1 to i gear downshift curve.

[0084] Understandably, the first model specifies the conditions for car gear shifting: when the vehicle speed is greater than a set value, it shifts up; when the vehicle speed is less than a set value, it shifts down.

[0085] In one possible implementation, the preset shift strategy objectives include: speed following, fuel consumption reduction, engine speed range limitation, running time, and overlap with prior knowledge.

[0086] It should be noted that, in this embodiment, the prior knowledge can be the original shift curve. By introducing the original shift curve, the agent can avoid ineffective repeated exploration during the training process, thereby improving the learning efficiency of the agent.

[0087] In one possible implementation, the step-guided reward function includes:

[0088] R all =R v +R fuel +R fuel_spd +R t +R act

[0089]

[0090] In equation (2), R all R represents the cumulative reward. v Indicates speed-following reward, R fuel Indicates a fuel consumption reduction reward, R fuel_spd Indicates the engine speed range limit reward, R t R represents the runtime reward. act Represents a reward for prior knowledge, v trgt v represents the target velocity. act This indicates the actual vehicle speed, and fuel indicates the fuel consumption per unit distance. before This indicates the fuel consumption per unit distance in the previous round. spd Indicates engine speed, t all n represents the time required to complete one round of road spectrum simulation. same n represents the number of gear selection decisions made with the same prior knowledge. all This represents the total number of gear decisions made in one round of road spectrum simulation.

[0091] It should be noted that, in this embodiment, considering that different stages in the gear shifting process may have different importance, reward functions with different weights are set to guide the agent to better learn and adapt to the gear shifting strategies at each stage.

[0092] Specifically, in this embodiment:

[0093]

[0094] When the speed error is less than 1 m / s, a positive reward of 1 point is obtained; otherwise, a negative penalty of 1 point is imposed to incentivize the agent to shift gears at an appropriate speed to meet the road spectrum requirements.

[0095]

[0096] If the fuel consumption per unit distance in this round is less than that in the previous round, a positive reward of 1 point is obtained; otherwise, a negative penalty of 0.1 points is imposed.

[0097]

[0098] When the engine speed is between 1000 and 2000 r / min, a positive reward of 1 point is obtained; otherwise, a negative penalty of 0.1 points is imposed. This method incentivizes the agent to adjust the vehicle's gears to keep the engine operating in its most efficient range.

[0099]

[0100] The time reward is set to a certain gradient and monotonically increases to guide the agent to complete as many rounds of simulation as possible.

[0101]

[0102] By setting time rewards to a certain gradient that decreases monotonically, the agent is guided to learn in this way, avoiding its over-reliance on prior knowledge and its failure to actively explore and learn.

[0103] In one possible implementation, the second model includes:

[0104] Action-value function network Q(s) i ,a i ;θ Q ), used to obtain cumulative rewards based on state data and action data, where s i Represents state data, a i Represents action data, θ Q This represents the network parameters of the action-value function network;

[0105] Deterministic policy network μ(s) i ;θ μ), used to determine the action data that yields the maximum cumulative reward under different state data, where θ μ Represents the network parameters of a deterministic policy network;

[0106] Target policy network Q′(s) i ,μ′(s i ;θ′ μ );θ′ Q ), used to evaluate the cumulative reward obtainable from action data under different state data, where θ′ Q and θ′ μ All of these represent the network parameters of the target policy network.

[0107] It's important to note that the action-value function minimizes the difference between the predicted action value and the actual reward. Gradient descent is used to update the parameters of the action-value function, making it approximate the true cumulative reward. A deterministic policy, on the other hand, directly outputs a deterministic action value based on the current state. Its goal is to maximize the action-value function's estimate of the current state. Gradient ascent is used to update the parameters of the deterministic policy, enabling it to generate better actions.

[0108] like Figure 2 As shown, in one possible implementation, step S104 includes:

[0109] Step S201: Obtain the initialized state data and noise and input them into the deterministic policy network;

[0110] Step S202: Obtain action data based on the first formula, and calculate the cumulative reward corresponding to the current action data and state data based on the step-guided reward function;

[0111] Step S203: Obtain the current shifting strategy based on the motion data, the first model, and the shift curve;

[0112] Step S204: Obtain the status data corresponding to the current shifting strategy, and store the accumulated rewards and status data as experience data in the preset replay area;

[0113] Step S205: Randomly sample several empirical data points from the replay area, input them into the target policy network, and calculate the target value based on the second formula;

[0114] Step S206: Calculate the loss L of the action value function network based on the target value and the third formula, and update the network parameters of the action value function network based on the gradient descent method;

[0115] Step S207: Update the network parameters of the deterministic policy network based on the gradient ascent method;

[0116] Step S208: Based on the fourth formula, calculate the network parameters θ′ of the target policy network. Q and θ′ μ The model is updated to obtain a second model, based on the updated deterministic policy network and state data s. i+1 Determine the new motion data and proceed to step S202;

[0117] The first formula includes:

[0118] a i =μ(s) i ;θ μ +noise (8)

[0119] Here, "noise" refers to noise.

[0120] The second formula includes:

[0121] y i =r i +γ*Q′(s i+1 ,μ′(s i+1 ;θ′ μ );θ′ Q (9)

[0122] Among them, y i Represents the target value, r i This represents the cumulative reward corresponding to the current action data and state data, γ represents the preset weight value, and s i+1 This indicates the status data corresponding to the current shifting strategy;

[0123] The third formula includes:

[0124]

[0125] Where L represents the loss degree of the action value function network, and N represents the batch of sampled empirical data.

[0126] The fourth formula includes:

[0127]

[0128] in, All of these represent the network parameters used by the target policy network in this round of training. All represent the network parameters after the target policy network is updated, and τ represents the preset weight value.

[0129] Understandably, this embodiment employs an experience replay method. During interaction with the environment, the agent stores its experiences in a replay buffer. Then, a batch of experiences is randomly sampled from the replay buffer for network training. This reduces the correlation between samples, improving training efficiency and stability. Furthermore, this embodiment incorporates random noise during the determination of the second model, effectively preventing the network from getting stuck in local optima during training.

[0130] It should be noted that in this embodiment, after obtaining the fully trained second model, the acquisition process of the second model will not end, but will enter the next round of training. The training data comes from the training process of the second model. In other words, the acquisition of the second model is a continuous self-iterative process. In each iteration, the training information acquired by the second model will be richer, and the accuracy and versatility of the shift strategy acquired by the second model will continue to increase with the number of times it is used.

[0131] like Figure 3 As shown, the present invention also provides a shift strategy generation device 30, comprising:

[0132] The first model building module 310 is used to build a first model based on the correspondence between preset variables and vehicle gear states, wherein the variables include state variables and action variables.

[0133] The reward function module 320 is used to establish a step-guided reward function based on a preset shift strategy objective;

[0134] The curve optimization module 330 is used to obtain the shift curve based on the phased training method and road spectrum data with different speed gradients.

[0135] The second model building module 340 is used to determine the second model based on deep reinforcement learning methods, a step-guided reward function, the first model, and a shift curve.

[0136] The strategy generation module 350 is used to generate the final shift strategy based on the second model.

[0137] like Figure 4 As shown, the present invention also provides an electronic device 40. The electronic device 40 includes a processor 410, a memory 420, and a display 430. Figure 4 Only some of the components of the electronic device 40 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0138] In some embodiments, processor 410 may be a central processing unit (CPU), microprocessor or other data processing chip, used to run program code stored in memory 420 or process data, such as the shift strategy generation method of the present invention.

[0139] In some embodiments, processor 410 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 410 may be local or remote. In some embodiments, processor 410 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0140] In some embodiments, memory 420 may be an internal storage unit of electronic device 40, such as a hard disk or memory of electronic device 40. In other embodiments, memory 420 may also be an external storage device of electronic device 40, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 40.

[0141] Furthermore, the memory 420 may include both internal storage units of the electronic device 40 and external storage devices. The memory 420 is used to store application software and various types of data installed on the electronic device 40.

[0142] In some embodiments, display 430 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 430 is used to display information from electronic device 40 and to display a visual user interface. Components 610-630 of electronic device 40 communicate with each other via a system bus.

[0143] In one embodiment, when the processor 410 executes the shift strategy generation program in the memory 420, the following steps can be implemented:

[0144] A first model is established based on the pre-defined correspondence between variables and vehicle gear status, wherein the variables include state variables and action variables;

[0145] A tiered reward function is established based on the preset shift strategy objective;

[0146] The shift curve is obtained based on a phased training method and road spectrum data with different speed gradients;

[0147] The second model is determined based on deep reinforcement learning methods, a step-guided reward function, the first model, and a shift curve.

[0148] The final shifting strategy is generated based on the second model.

[0149] It should be understood that when the processor 410 executes the shift strategy generation program in the memory 420, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0150] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 40 mentioned. Electronic device 40 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 40 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0151] Accordingly, this application also provides a computer-readable storage medium for storing a computer-readable program or instruction. When the program or instruction is executed by a processor, it can implement the steps or functions in the shift strategy generation method provided in the above-described method embodiments.

[0152] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0153] The shift strategy generation method, apparatus, electronic device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating a shift strategy, characterized in that, include: A first model is established based on the pre-defined correspondence between variables and vehicle gear status, wherein the variables include state variables and action variables; A tiered guided reward function is established based on preset shift strategy objectives. These preset objectives include: speed following, fuel consumption reduction, engine speed range limitation, running time, and overlap with prior knowledge. The tiered guided reward function includes: in, Indicates cumulative rewards. Indicates speed-following reward. This indicates a reward for reduced fuel consumption. This indicates a reward based on engine speed range restrictions. This indicates a runtime reward. This indicates a reward for prior knowledge. Indicates the target speed. Indicates the actual vehicle speed. fuel This indicates the fuel consumption per unit distance. fuel before This indicates the fuel consumption per unit distance in the previous round. Indicates engine speed. t all This indicates the time required to complete one round of road spectrum simulation. n same This indicates the number of gear selection decisions that are consistent with prior knowledge. n all This represents the total number of gear decisions made in one round of road spectrum simulation. The shift curve is obtained based on a phased training method and road spectrum data with different speed gradients; A second model is determined based on deep reinforcement learning methods, the stepped guided reward function, the first model, and the shift curve; the second model includes: Action Value Function Network It is used to obtain cumulative rewards based on status data and action data, whereby... s i Represents state data, a i Represents action data, This represents the network parameters of the action-value function network; Deterministic policy network This is used to determine the action data that yields the maximum cumulative reward under different state data conditions. Represents the network parameters of a deterministic policy network; Target Policy Network This is used to evaluate the cumulative reward obtainable from action data under different state data conditions, where... and All of these represent the network parameters of the target policy network; The final shifting strategy is generated based on the second model.

2. The shift strategy generation method according to claim 1, characterized in that, The state variables include at least: vehicle speed, acceleration, engine torque demand, fuel consumption per unit distance, and engine speed; The action variables include at least the modification values ​​of the shift speed of the shift curve under each fixed required torque.

3. The shift strategy generation method according to claim 2, characterized in that, The first model includes: in, GearState Indicates the vehicle's gear status. v Indicates the current actual vehicle speed. up th Indicates the upshift speed. down th Indicates downshifting speed. Upshift This represents the lookup table function for upshifting. Downshift This represents the lookup function for downshifting. Gear Indicates the current gear. T cmd This indicates the engine's required torque. Act () indicates the action taken by the agent. This indicates the modification value for the shift speed curve from 1st to 2nd gear. Indicates from i gear shift i The modification value for the shift speed of the +1 gear curve; This indicates the modification value for the shift speed of the 2nd gear downshift to 1st gear curve. express i +1 gear down i The modified value for the shift speed of the gear curve.

4. The shift strategy generation method according to claim 1, characterized in that, The determination of the second model based on the deep reinforcement learning method, the step-guided reward function, the first model, and the shift curve includes: Step 1: Obtain the initialized state data and noise and input them into the deterministic policy network; Step 2: Obtain action data based on the first formula, and calculate the cumulative reward corresponding to the current action data and state data based on the step-guided reward function; Step 3: Obtain the current shifting strategy based on the action data, the first model, and the shift curve; Step 4: Obtain the status data corresponding to the current shifting strategy, and store the accumulated reward and status data as experience data in a preset replay area; Step 5: Randomly sample several empirical data points from the replay area, input them into the target policy network, and calculate the target value based on the second formula; Step Six: Calculate the loss of the action-value function network based on the target value and the third formula. L And update the network parameters of the action value function network based on the gradient descent method; Step 7: Update the network parameters of the deterministic policy network based on the gradient ascent method; Step 8: Analyze the network parameters of the target policy network based on the fourth formula. and The model is updated to obtain a second model, based on the updated deterministic policy network and state data. s i+1 Determine the new motion data and return to step two; The first formula includes: in, noise Indicates noise The second formula includes: in, y i Indicates the target value. r i This represents the cumulative reward corresponding to the current action data and state data. This represents the preset weight value. s i+1 This indicates the status data corresponding to the current shifting strategy; The third formula includes: in, L This represents the loss degree of the action-value function network. N This indicates a batch of sampled empirical data; The fourth formula includes: in, , All of these represent the network parameters used by the target policy network in this round of training. , All represent the network parameters after the target policy network update. This indicates the preset weight value.

5. A gear shifting strategy generation apparatus, used to implement the gear shifting strategy generation method as described in any one of claims 1-4, characterized in that, The device includes: The first model building module is used to build a first model based on the correspondence between preset variables and vehicle gear states, wherein the variables include state variables and action variables; The stepped guided reward function module is used to establish a stepped guided reward function based on a preset shift strategy objective; The curve optimization module is used to obtain shift curves based on a phased training method and road spectrum data with different speed gradients. The second model building module is used to determine the second model based on the step-guided reward function, the first model, and the shift curve. The strategy generation module is used to generate the final shifting strategy based on the second model.

6. An electronic device, characterized in that, include: Memory, used to store programs; A processor, coupled to the memory, is configured to execute the program stored in the memory to implement the steps in the shift strategy generation method as described in any one of claims 1-4.

7. A storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, enable the implementation of the steps in the shift strategy generation method as described in any one of claims 1-4.

Images