Vehicle driving strategy processing method and device

Abstract

The present invention discloses a vehicle driving strategy processing method and device, which relates to the field of intelligent driving. A specific implementation of the method includes: collecting driving trajectories of multiple vehicles in different states, grouping the driving trajectories according to the strategy used in the first state of each driving trajectory, and obtaining multiple driving trajectory data sets; constructing a value function for each driving trajectory data set to calculate the confidence of each value function, and then calculating the probability activation threshold based on the confidence of each value function; taking the probability that the function value of each value function is greater than a preset value as an evaluation error, and constructing a probability activation function based on the evaluation errors and probability activation thresholds of all value functions; adjusting the network parameters of the decision model based on multiple driving trajectory data sets and the probability activation function to obtain an adjusted decision model. This implementation method segments the driving trajectory and constructs a probability activation function to ensure the reliability of the decision model under limited data conditions.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to a vehicle driving strategy processing method and device. Background Art

[0002] The self-learning decision-making method evaluates the unmanned driving strategy through the feedback obtained from the unmanned vehicles in different states and actions in the historical data, and adjusts the strategy based on the evaluation to adapt to the environment. Therefore, the uncertainty of the environmental state will directly affect the strategy return to achieve strategy adjustment. This solution does not make a priori assumptions about the environmental model, and the solution process itself does not require the state space dimension, so it is expected to be applied to the unmanned driving strategy solution in high-dimensional uncertainty scenarios.

[0003] In the process of implementing the present invention, the inventors found that insufficient training data would lead to unreliable self-learning strategies. Existing methods usually assume that the training data can cover all scenarios, but in a high-dimensional uncertain environment, complete coverage of scenarios requires a large amount of training data. The performance of an insufficiently trained self-learning decision-making method cannot be guaranteed when encountering new scenarios, so it is difficult to implement the self-learning decision-making method in autonomous driving. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a vehicle driving strategy processing method and device, which can at least solve the problem in the prior art that insufficient training data leads to insufficient training of the self-learning decision-making method, which is then not suitable for unmanned driving scenarios.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a vehicle driving strategy processing method is provided, comprising:

[0006] Collect driving trajectories of multiple vehicles in different states, group the driving trajectories according to the strategy used in the first state of each driving trajectory, and obtain multiple driving trajectory data sets;

[0007] Constructing a value function for each driving trajectory data set to calculate the confidence of each value function, and then calculating a probability activation threshold based on the confidence of each value function;

[0008] The probability that the function value of each value function is greater than a preset value is taken as an evaluation error, and a probability activation function is constructed based on the evaluation errors of all value functions and the probability activation threshold;

[0009] Based on the multiple driving trajectory data sets and the probability activation function, the network parameters of the decision model are adjusted to obtain an adjusted decision model; wherein the adjusted decision model is used to plan a driving strategy according to the current state and driving trajectory of the vehicle during vehicle driving.

[0010] Optionally, before grouping the driving trajectories according to the strategy used in the first state of each driving trajectory, the method further includes:

[0011] Using a sliding window of preset length, each driving trajectory is divided into multiple driving trajectories of preset length.

[0012] Optionally, the method further includes: counting the number of driving trajectories in each driving trajectory data set, determining a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold, and setting the evaluation error of the target driving trajectory data set to 0.

[0013] Optionally, the strategy includes one of a rule strategy and a self-learning strategy, and the confidence calculation probability activation threshold based on each value function includes:

[0014] The probability value when the performance expectation of the self-learning strategy relative to the rule strategy is maximized is calculated, and the probability value is used as the value range of the probability activation threshold; wherein the performance expectation represents the confidence difference.

[0015] Optionally, the driving strategy planning according to the current state and driving trajectory of the vehicle includes:

[0016] In response to a query result that the current state of the vehicle does not exist in the state space, determining that the vehicle driving strategy is a rule strategy;

[0017] A variable is randomly generated. While the vehicle continues to drive, the driving trajectory of the vehicle after using the rule strategy is collected to estimate the rule strategy performance value. If it is detected that the variable is greater than the cumulative value of the rule strategy performance value and 1, a strategy update operation is triggered.

[0018] To achieve the above object, according to another aspect of an embodiment of the present invention, a vehicle driving strategy processing device is provided, comprising:

[0019] A collection and grouping module is used to collect the driving trajectories of multiple vehicles in different states, and group the driving trajectories according to the strategy used in the first state of each driving trajectory to obtain multiple driving trajectory data sets;

[0020] A confidence calculation module, used to construct a value function for each driving trajectory data set to calculate the confidence of each value function, and then calculate the probability activation threshold based on the confidence of each value function;

[0021] A function construction module, used to take the probability that the function value of each value function is greater than a preset value as an evaluation error, and to construct a probability activation function based on the evaluation errors of all value functions and the probability activation threshold;

[0022] A parameter adjustment module is used to adjust the network parameters of the decision model based on the multiple driving trajectory data sets and the probability activation function to obtain an adjusted decision model; wherein the adjusted decision model is used to plan a driving strategy according to the current state and driving trajectory of the vehicle during vehicle driving.

[0023] Optionally, the collection and grouping module is further used to:

[0024] Using a sliding window of preset length, each driving trajectory is divided into multiple driving trajectories of preset length.

[0025] Optionally, the function building module is used to:

[0026] The number of driving trajectories in each driving trajectory data set is counted, a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold is determined, and an evaluation error of the target driving trajectory data set is set to 0.

[0027] Optionally, the strategy includes one of a rule strategy and a self-learning strategy, and the confidence calculation module is used to:

[0028] The probability value when the performance expectation of the self-learning strategy relative to the rule strategy is maximized is calculated, and the probability value is used as the value range of the probability activation threshold; wherein the performance expectation represents the confidence difference.

[0029] Optionally, a strategic planning module is also included to:

[0030] In response to a query result that the current state of the vehicle does not exist in the state space, determining that the vehicle driving strategy is a rule strategy;

[0031] A variable is randomly generated. While the vehicle continues to drive, the driving trajectory of the vehicle after using the rule strategy is collected to estimate the rule strategy performance value. If it is detected that the variable is greater than the cumulative value of the rule strategy performance value and 1, a strategy update operation is triggered.

[0032] To achieve the above objective, according to another aspect of an embodiment of the present invention, a vehicle driving strategy processing electronic device is provided.

[0033] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the vehicle driving strategy processing methods described above.

[0034] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned vehicle driving strategy processing methods is implemented.

[0035] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: each driving trajectory is segmented to increase the amount of training data as much as possible; a probability activation function is constructed to ensure the reliability of the rule strategy performance improvement when the self-learning strategy is activated, thereby ensuring the reliability of the decision-making model under limited data conditions.

[0036] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0038] Figure 1 is a schematic diagram of the main flow of a vehicle driving strategy processing method according to an embodiment of the present invention;

[0039] Figure 2 It is a schematic diagram of the relationship between states, strategies and behaviors;

[0040] Figure 3 It is a schematic diagram of the effect of probability activation threshold on hybrid decision;

[0041] Figure 4 is a schematic diagram of main modules of a vehicle driving strategy processing device according to an embodiment of the present invention;

[0042] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0043] Figure 6 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] See also Figure 1 , which shows a main flow chart of a vehicle driving strategy processing method provided by an embodiment of the present invention, comprising the following steps:

[0046] S101: collecting driving trajectories of multiple vehicles in different states, grouping the driving trajectories according to the strategy used in the first state of each driving trajectory, and obtaining multiple driving trajectory data sets;

[0047] S102: constructing a value function for each driving trajectory data set to calculate the confidence of each value function, and then calculating a probability activation threshold based on the confidence of each value function;

[0048] S103: taking the probability that the function value of each value function is greater than a preset value as an evaluation error, and constructing a probability activation function based on the evaluation errors of all value functions and the probability activation threshold;

[0049] S104: Based on the multiple driving trajectory data sets and the probability activation function, adjust the network parameters of the decision model to obtain an adjusted decision model; wherein the adjusted decision model is used to plan a driving strategy according to the current state and driving trajectory of the vehicle during vehicle driving.

[0050] In the above implementation, for step S101, the real-world scene has the characteristics of high dimensionality and strong uncertainty, so it is very difficult for unmanned vehicles to fully sample all scenes. According to statistics, about 14 billion kilometers of driving data are required to verify only one driving strategy in a real driving scene, and the amount of data required for strategy training is even greater.

[0051] Since the vehicle interacts with surrounding vehicles and roads when driving, the vehicle driving strategy decision needs to comprehensively consider the vehicle's motion state, surrounding vehicle motion state and lane passability. The state is the vehicle's driving state, including 1) the vehicle's current motion state information, such as speed and acceleration, 2) surrounding vehicle's current motion state information, such as longitudinal distance, lateral distance and relative speed with the vehicle. If there are no surrounding vehicles, the distance is set to infinity and the relative speed is set to 0. 3) Lane passability information, such as left lane marking information and right lane marking information. The passable mark is 1, and the unpassable mark (such as the opposite lane, non-motorized vehicle lane, and curb) is 0.

[0052] For example, when walking through a maze, a vehicle always likes to go left. This is a behavior pattern. Vehicles also follow certain behavior patterns during driving, such as turning around, changing lanes, turning left, turning right, going straight, etc., corresponding to different strategies, represented by π, and the value function is also closely related to it. When the vehicle always runs with strategy π, the value function shows that there is a correlation between the expected return value that can be obtained in the current state s and its subsequent state.

[0053] For example, see Figure 2As shown in the figure, the hollow circles represent states and the solid circles represent behaviors. In state s, the vehicle faces three choices, and the strategy π is what prompts the vehicle to choose among the three behaviors. It represents a probability distribution, that is, the probability of choosing different behaviors from all behaviors. After choosing behavior a, it will switch to a new state s' with different possibilities, and the conversion probability is p. In the process of conversion, it will also obtain the corresponding reward from the environment, that is, r.

[0054] When a vehicle is driving on a real road, the length of its driving trajectory is usually not fixed, so the decision model needs to use a sliding window of preset length to collect the required trajectory τ π (s), and the driving trajectory under different states is obtained as shown in the following formula:

[0055]

[0056] τ π (s i ):={s i ,a i ,...,s k ,a k}

[0057] k=min(i+H-1,n)

[0058] In the formula, s is the state, Z s is the terminal state set, ω π is the trajectory of the vehicle using strategy π. Figure 2 As can be seen from the description, each trajectory may consist of multiple states and strategies. Therefore, according to the action (i.e. strategy) used in the first state in the trajectory, two sub-datasets (rule strategy dataset and self-learning strategy dataset) are obtained, as shown below:

[0059] D b (s): = {τ(s1 = s, a1 = π b (s1))}

[0060] D r1 (s): = {τ(s1 = s, a1 = π r1 (s1))}

[0061] Where D b (s) contains the trajectory data of the vehicle in the running state s, the first action using the rule strategy; D r1(s) contains the trajectory data of the vehicle in the running state s, and the first action uses the self-learning strategy. For example, if there are 2 states and 2 strategies, 4 driving trajectory datasets D(s,a) will be obtained, including driving trajectory dataset 1-state 1 and strategy 1, driving trajectory dataset 2-state 1 and strategy 2, driving trajectory dataset 3-state 1 and strategy 2, and driving trajectory dataset 4-state 2 and strategy 2.

[0062] For step S102, after the driving trajectory data set D(s,a) is generated, the evaluation error of each driving trajectory data set is calculated; wherein the evaluation error represents the deviation between the estimated value and the true value, such as the estimated probability is 90%, but the true value is 1, then the deviation is 10%.

[0063] Define the value function Q π (s,u π ), represents the expected performance of the vehicle using state s and strategy π in the future. However, when the training data is limited, there are errors in the estimation of the value function. If the value function containing errors is directly used to adjust the driving strategy model in the decision model, it may lead to incorrect strategy use. In fact, the real value function only exists in the concept and cannot be directly observed. Considering that when the vehicle motion state can be fully sampled, the deviation between the estimated value and the true value of the value function is small, so a function can be constructed to express the deviation. The estimation method based on Monte Carlo sampling here requires sufficient sampling of each state and trajectory under each strategy. However, due to the strong uncertainty of the environment, the high-dimensional characteristics of the environment, and the insufficient amount of data, it is impossible to guarantee that the sufficient sampling requirements are met. Therefore, this method is needed to obtain the confidence of the value function.

[0064] This scheme preferably uses the Lindberg-Feller theorem to estimate the confidence of the value function of each driving trajectory data set. The main idea is that the mean of a large number of independent random variables, after appropriate standardization, converges to a normal distribution. The formula is as follows:

[0065]

[0066]

[0067] σ 2 =Var(G π (s))

[0068] In the formula, Φ(·) represents the cumulative distribution function of the standard normal distribution, G π(s) represents the discounted return value of the trajectory. Through the above formula, the true value distribution of the value function is described as a cumulative distribution function, and as the number of sampling increases, its estimated value gradually approaches the true value. This formula can also be used to obtain the confidence of the value function by integration. Define the true value function use Represents the true value function The probability that is greater than λ will As the evaluation error. Here, based on the rule strategy dataset D(s t ,a b )calculate And based on the self-learning strategy dataset D(s t ,a r1 )calculate

[0069] For step S103, after generating the evaluation error of each driving trajectory data set, an activation function that takes into account the value function error must be constructed first, which is called a probability activation function in this solution. Considering that in actual application, the self-learning strategy will be iteratively updated during the training process, the true value of the self-learning value function is estimated conservatively based on the trajectory data obtained by the self-learning strategy, that is, the calculated value of the probability activation function is larger than the true value, but it does not undermine the reliability of the hybrid decision-making to improve the performance of the strategy.

[0070] Design of hybrid decision probability activation function: expected value of policy performance under limited data may not be equal to the value function Q(s,u π ), when the driverless vehicle is driving in some scenes, there are the following situations:

[0071]

[0072]

[0073] In the above formula, for the rule strategy π b And the self-learning strategy π r1 , when according to the self-learning strategy value function Q(s,u r1 ) and the rule strategy value function Q(s,u b ) When the probability activation function is activated, the self-learning strategy performs worse than the rule strategy, and the self-learning strategy is mistakenly activated. Since the value function cannot be accurately estimated, the performance of the self-learning strategy is better than the expectation of the rule strategy, which is expressed as the probability activation function:

[0074]

[0075] In the formula, P(·) represents the probability of an event occurring, c thresTo obtain the hybrid decision probability activation threshold, the confidence level needs to be calculated theoretically. Based on the above formulas (1) and (2), the relationship between the probability activation function and the sampled data set is as follows:

[0076]

[0077] In the formula, when the number of sampling is insufficient, that is, n d ≤n thres =30, let κ(Q(s,u),λ)=0. Probability activation threshold c thres The design principle is to ensure that the self-learning strategy has the greatest expectation of improving the performance of the rule strategy based on the current observations and training data, that is, (4):

[0078]

[0079] Furthermore, according to the above formulas (1) and (3), we can get:

[0080]

[0081] Ideally, the expected performance of the two strategies is a constant value. However, due to the existence of estimation errors, it is impossible to accurately estimate the expected performance of the two. Therefore, the probabilistic activation function is designed in the form of probability. The strategy evaluation process can be introduced into the activation function to solve the evaluation error caused by insufficient training, avoid the false activation of the self-learning strategy due to insufficient sampling due to limited data, and achieve high-reliability decision-making under limited data conditions.

[0082] See also Figure 3 As shown, the probability activation threshold c thres It can realize the transition from rule-based strategy to fully self-learning strategy. Assuming that the probability activation threshold value ranges from 0 to 1, when the threshold is 0, the probability activation function always holds, so the self-learning strategy will always be used to drive the vehicle. When the threshold is 1, the probability activation function always does not hold, so the rule-based strategy will always be used to drive the vehicle. In the values ​​of 0 to 1, the hybrid decision framework is used for driving. When the threshold is 0.5, the performance improvement is expected to be the highest, so the reliability of the hybrid strategy to improve the rule strategy can be guaranteed. A higher threshold will make the strategy improvement conservative, while a lower threshold may make the strategy improvement unreliable.

[0083] For step S104, determine the probability activation threshold c under limited data conditions thres After that, based on the probability activation threshold c thres To design the decision model, this solution refers to the deep Q learning algorithm to establish the decision model. The iterative calculation of the Q value uses the Bellman equation, and its update principle is shown in the following formula, where α is the learning rate:

[0084] Q(s k ,a k )←

[0085] Q(s k ,a k )+α[r(s k+1 )+γmax a Q(s k+1 ,a)-Q(s k ,a k )]

[0086] In the above formula, the Q function (also known as the right tail function of the standard normal distribution) needs to be updated according to the rewards generated by the state and action strategy. However, in the definition of the autonomous driving decision-making problem, the state space is usually continuous, so it is impossible to repeatedly visit a specific state. For this reason, the storage and update of the Q function will be completed based on a neural network. This framework is called the deep Q learning method, and its update principle is shown in the following formula:

[0087]

[0088] Q + (s k ,a k )=r(s k+1 )+γmax a Q(s k+1 ,a,θ - )

[0089] In the formula, θ and θ - They represent the Q network parameters being adjusted and the Q network parameters stored historically. - It will iterate a certain number of times n up = Update θ after 100 times. This setting makes the network update more stable. (Q(s k ,a k ,θ j )-Q + (s k ,a k )) 2 represents the training error, Q + (s k ,a k ) is calculated by the Bellman equation. Since the value of the Q function is updated based on the data set D(s,a), data sampling determines the quality of the policy update. In this algorithm, the self-learning policy update is divided into two stages: the rule policy verification stage and the self-learning exploration stage.

[0090] The algorithm here describes the process of self-learning exploration and strategy generation. Represents a dynamically updated dataset D(s1,a b ) estimated strategy performance, nd (D(s,a b )) represents the number of trajectories in the dataset:

[0091] i=0;

[0092] ω=0;

[0093] D(s,a b )=0;

[0094] D(s,a)=0;

[0095] while i≤total number of training times do

[0096] i=1+1;

[0097] if s t ∈Z s , then

[0098] Randomly sample ξ←U(0,1);

[0099] If n d (D(s,a b ))≥n thres and then

[0100] Generate exploration action a explore ;

[0101] Use action a explore Interact with the environment;

[0102] else

[0103] Use action a b Interact with the environment;

[0104] From ω, extract τ={s1,a1,r1,...,s H ,a H ,r H};

[0105] if a1=π b (s1) then

[0106] Add the trajectory τ to D(s1,a b );

[0107] else

[0108] Add the trajectory τ to D(s1,a=a1);

[0109] Delete (s1,a1) from ω

[0110] else

[0111] for i=l to n ω do

[0112] From ω, extract τ = {s i ,a i ,...,s nω ,a nω};

[0113] if a i =π b (s) then

[0114] Add the trajectory τ to D(s1,a b );

[0115] else

[0116] Add the trajectory τ to D(s1,a=a i );

[0117] ω=θ;

[0118] for i=l to H do

[0119] Use action a b Interact with the environment

[0120] Specifically, in the first stage, the driverless vehicle will drive using only rule-based strategies to obtain Q(s,a b ) update; in the second stage, the self-learning strategy actively tries different actions from the rule strategy to obtain a better strategy. The switching between the two stages is achieved by designing a suppression of the self-learning exploration range. In the algorithm framework proposed in this chapter, there are two suppression conditions that limit the exploration range: 1) When the rule strategy has not been verified for a long enough time, that is, n d (D(s,a b ))<n thres , no strategy exploration will be performed; 2) When the rule strategy performs well enough, no strategy exploration will be performed. In contrast, the exploration scheme of traditional self-learning algorithms is usually independent of other strategies.

[0121] Since the exploration scope usually determines the efficiency of strategy exploration, the larger the exploration scope means the lower the efficiency of strategy exploration. The method proposed in this scheme can also help improve the exploration efficiency. In addition, this scheme prefers to use the e-greedy method for strategy exploration. Other strategy exploration methods are also feasible in this scheme.

[0122] The reason for setting the first inhibition condition is that the probability activation function needs to estimate the value function of the rule strategy first. When the rule strategy does not obtain the confidence of the value function estimate, the system will not be able to determine whether the current self-learning strategy can be effectively improved. Therefore, when encountering an unfamiliar state, it is necessary to give priority to driving based on the rule strategy to ensure driving reliability. The second method of setting the inhibition condition is to randomly generate a variable ξ~U(0,1) when the first condition is met, that is, ξ obeys the average distribution between 0 and 1. When the variable satisfies the constraint condition so that The system will actively explore.

[0123] In this way, the self-learning exploration probability is -Q(s,a b ), this probability is related to the rule strategy value function. The larger the rule strategy value function, the weaker the exploration motivation. Under the reward function condition for probability constraints, the Q value represents the probability that the strategy satisfies the constraint. Therefore, this method means that the more likely the danger is to occur based on the rule strategy, the more motivated the system is to actively explore. This method can improve the exploration efficiency while ensuring reliability.

[0124] In addition, since the state space is continuous, the system uses a fixed sampling window method to obtain the data set corresponding to a certain state and calculate the corresponding number of sampling times. The scale of the sampling window is b(max(D(s))-min(D(s))), b∈(0,1). The value of b needs to be designed based on experience. In this scheme, 0.1 is preferred.

[0125] When the driverless vehicle visits a certain state s during driving t When , the decision model under limited data conditions is generated according to the probability activation function:

[0126]

[0127]

[0128]

[0129] This decision model can be used to plan driving strategies based on the vehicle's current state and driving trajectory.

[0130] The method provided by the embodiment of the present invention can still ensure effective determination of the strategy under the condition of limited training data:

[0131] 1. Use a sliding window of preset length to split each driving trajectory into multiple driving trajectories of preset length to maximize the amount of training data;

[0132] 2. Construct a probability activation function to ensure the reliability of the rule strategy performance improvement when the self-learning strategy is activated, thereby achieving the reliability of the decision-making model under limited data conditions;

[0133] 3. Design a self-learning decision-making system based on the deep Q-learning framework. While considering the rule strategy for strategy search, it also considers the probability activation function's demand for the confidence of the strategy value function, so that when the confidence of the rule strategy value function does not meet the requirements or perform well, it can inhibit the strategy exploration of the self-learning model.

[0134] See also Figure 4 , which shows a schematic diagram of main modules of a vehicle driving strategy processing device 400 provided by an embodiment of the present invention, including:

[0135] The collection and grouping module 401 is used to collect the driving trajectories of multiple vehicles in different states, and group the driving trajectories according to the strategy used in the first state of each driving trajectory to obtain multiple driving trajectory data sets;

[0136] A confidence calculation module 402 is used to construct a value function for each driving trajectory data set to calculate the confidence of each value function, and then calculate the probability activation threshold based on the confidence of each value function;

[0137] A function construction module 403 is used to take the probability that the function value of each value function is greater than a preset value as an evaluation error, and to construct a probability activation function based on the evaluation errors of all value functions and the probability activation threshold;

[0138] The parameter adjustment module 404 is used to adjust the network parameters of the decision model based on the multiple driving trajectory data sets and the probability activation function to obtain an adjusted decision model; wherein the adjusted decision model is used to plan the driving strategy according to the current state and driving trajectory of the vehicle during the vehicle driving process.

[0139] In the implementation device of the present invention, the collection and grouping module 401 is also used for:

[0140] Using a sliding window of preset length, each driving trajectory is divided into multiple driving trajectories of preset length.

[0141] In the implementation device of the present invention, the function construction module 403 is used to:

[0142] The number of driving trajectories in each driving trajectory data set is counted, a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold is determined, and an evaluation error of the target driving trajectory data set is set to 0.

[0143] In the implementation device of the present invention, the strategy includes one of a rule strategy and a self-learning strategy, and the confidence calculation module 402 is used to:

[0144] The probability value when the performance expectation of the self-learning strategy relative to the rule strategy is maximized is calculated, and the probability value is used as the value range of the probability activation threshold; wherein the performance expectation represents the confidence difference.

[0145] The implementation device of the present invention also includes a strategy planning module, which is used to:

[0146] In response to a query result that the current state of the vehicle does not exist in the state space, determining that the vehicle driving strategy is a rule strategy;

[0147] A variable is randomly generated. While the vehicle continues to drive, the driving trajectory of the vehicle after using the rule strategy is collected to estimate the rule strategy performance value. If it is detected that the variable is greater than the cumulative value of the rule strategy performance value and 1, a strategy update operation is triggered.

[0148] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0149] Figure 5 An exemplary system architecture 500 to which embodiments of the present invention may be applied is shown, including terminal devices 501 , 502 , 503 , a network 504 and a server 505 (only an example).

[0150] Terminal devices 501, 502, 503 can be various electronic devices with display screens and supporting web browsing, and various communication client applications are installed. Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc.

[0151] The network 504 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0152] The server 505 may be a server that provides various services. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 505. Accordingly, the device is generally set in the server 505. It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0153] Reference below Figure 6, which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0154] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0155] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0156] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0157] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0158] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor including a collection grouping module, a confidence calculation module, a function construction module, and a parameter adjustment module. The names of these modules do not constitute a limitation on the modules themselves in certain cases, for example, the parameter adjustment module may also be described as a "parameter adjustment and model usage module".

[0160] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes the vehicle driving strategy processing method of the present solution.

[0161] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A vehicle driving strategy processing method, characterized in that: include: Collect driving trajectories of multiple vehicles in different states, group the driving trajectories according to the strategy used in the first state of each driving trajectory, and obtain multiple driving trajectory data sets; A value function is constructed for each driving trajectory data set to calculate the confidence of each value function, and then a probability activation threshold is calculated based on the confidence of each value function; wherein the value function represents the expected performance of the vehicle usage status and strategy in the future; the design principle of the probability activation threshold is to ensure that the self-learning strategy has the greatest expectation of improving the performance of the rule strategy based on the current observation and training data; The probability that the function value of each value function is greater than a preset value is taken as an evaluation error, and a probability activation function is constructed based on the evaluation errors of all value functions and the probability activation threshold; wherein the probability activation function is an activation function that takes into account the value function error; when the probability activation threshold is the minimum value of the probability activation threshold value range, the probability activation function is always valid, and the self-learning strategy is used to drive the vehicle; when the probability activation threshold is the maximum value of the probability activation threshold value range, the probability activation function is always invalid, and the rule-based strategy is used to drive the vehicle; when the probability activation threshold is greater than the minimum value of the probability activation threshold value range and less than the maximum value, a hybrid decision framework is used for driving; the probability activation threshold value range is 0-1; Based on the multiple driving trajectory data sets and the probability activation function, the network parameters of the decision model are adjusted to obtain an adjusted decision model; wherein the adjusted decision model is used to plan a driving strategy according to the current state and driving trajectory of the vehicle during vehicle driving.

2. The method according to claim 1, characterized in that Before grouping the driving trajectories according to the strategy used in the first state in each driving trajectory, the method further includes: Using a sliding window of preset length, each driving trajectory is divided into multiple driving trajectories of preset length.

3. The method according to claim 1 or 2, characterized in that: Also includes: The number of driving trajectories in each driving trajectory data set is counted, a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold is determined, and an evaluation error of the target driving trajectory data set is set to 0.

4. The method according to claim 1 or 2, characterized in that: The strategy includes one of a rule strategy and a self-learning strategy, and the confidence calculation probability activation threshold based on each value function includes: The probability value when the performance expectation of the self-learning strategy relative to the rule strategy is maximized is calculated, and the probability value is used as the value range of the probability activation threshold; wherein the performance expectation represents the confidence difference.

5. The method according to claim 1, characterized in that The driving strategy planning according to the current state and driving trajectory of the vehicle includes: In response to a query result that the current state of the vehicle does not exist in the state space, determining that the vehicle driving strategy is a rule strategy; A variable is randomly generated. While the vehicle continues to drive, the driving trajectory of the vehicle after using the rule strategy is collected to estimate the rule strategy performance value. If it is detected that the variable is greater than the cumulative value of the rule strategy performance value and 1, a strategy update operation is triggered.

6. A vehicle driving strategy processing device, characterized in that: include: A collection and grouping module is used to collect the driving trajectories of multiple vehicles in different states, and group the driving trajectories according to the strategy used in the first state of each driving trajectory to obtain multiple driving trajectory data sets; A confidence calculation module is used to construct a value function for each driving trajectory data set to calculate the confidence of each value function, and then calculate the probability activation threshold based on the confidence of each value function; wherein the value function represents the expected performance of the vehicle usage status and strategy in the future; the design principle of the probability activation threshold is to ensure that the self-learning strategy has the greatest expectation of improving the performance of the rule strategy based on the current observation and training data; A function construction module, used to take the probability that the function value of each value function is greater than a preset value as an evaluation error, and construct a probability activation function based on the evaluation errors of all value functions and the probability activation threshold; wherein the probability activation function is an activation function that takes into account the value function error; when the probability activation threshold is the minimum value of the probability activation threshold value range, the probability activation function is always valid, and the self-learning strategy is used to drive the vehicle; when the probability activation threshold is the maximum value of the probability activation threshold value range, the probability activation function is always invalid, and the rule-based strategy is used to drive the vehicle; when the probability activation threshold is greater than the minimum value of the probability activation threshold value range and less than the maximum value, a hybrid decision framework is used for driving; the probability activation threshold value range is 0-1; A parameter adjustment module is used to adjust the network parameters of the decision model based on the multiple driving trajectory data sets and the probability activation function to obtain an adjusted decision model; wherein the adjusted decision model is used to plan a driving strategy according to the current state and driving trajectory of the vehicle during vehicle driving.

7. The device according to claim 6, characterized in that The function building module is used to: The number of driving trajectories in each driving trajectory data set is counted, a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold is determined, and an evaluation error of the target driving trajectory data set is set to 0.

8. The device according to claim 6, characterized in that The strategy includes one of a rule strategy and a self-learning strategy, and the confidence calculation module is used to: The probability value when the performance expectation of the self-learning strategy relative to the rule strategy is maximized is calculated, and the probability value is used as the value range of the probability activation threshold; wherein the performance expectation represents the confidence difference.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images