Vehicle control method and device

By collecting vehicle environment information, using strategy models and greedy algorithms to obtain lane adjustment strategies, and combining deep reinforcement learning and update models, the problem of low lane change in vehicles in complex environments in the existing technology is solved, and the vehicle is realized in a variety of ways Safe lane change in the scene.

CN119078833APending Publication Date: 2024-12-06BEIJING CO WHEELS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310667230.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-06
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing vehicle lane change methods are mainly suitable for fixed lane change scenarios, and the lane cannot be adjusted effectively in emergencies or complex environments, resulting in reduced vehicle safety.

Method used

By collecting the environment information around the vehicle, using the strategy model and greedy algorithm to obtain the estimated values ​​of multiple lane adjustment strategies, select the target lane adjustment strategy, and update the strategy model based on the deep reinforcement learning algorithm and Q-Learning to realize that the vehicle is in multiple scenarios in a variety of scenarios lane change control.

Benefits of technology

It improves the driving safety of vehicles in various lane changes scenarios, avoids the limitation that vehicles can only change lanes in fixed scenarios, and enhances the ability to deal with complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119078833A_ABST
    Figure CN119078833A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a vehicle control method and device, and relates to the technical field of data processing. The method comprises the steps that firstly, vehicle surrounding environment information is collected, and an initial vehicle state is obtained; inputting the initial vehicle state into a strategy model, and obtaining a plurality of lane adjustment strategies and an estimated value corresponding to each lane adjustment strategy of the plurality of lane adjustment strategies; the estimated value is used for representing the adaptation degree of the lane adjustment strategy and the initial vehicle state; obtaining a target lane adjustment strategy from the plurality of lane adjustment strategies based on a greedy algorithm and the estimated values corresponding to the lane adjustment strategies; and controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy. The embodiment of the invention is used for improving the vehicle driving safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a vehicle control method and device. Background Art

[0002] With the rapid development of autonomous driving technology, vehicles need to automatically adjust lanes in various complex environments while driving. Existing lane adjustment methods mainly deal with fixed lane change scenarios. Only in fixed lane change scenarios can the corresponding lane adjustment strategy be obtained. As a result, in the event of an accident or in other complex scenarios, the corresponding lane adjustment strategy cannot be obtained to achieve the vehicle's lane change, which greatly reduces the safety of the vehicle. Summary of the invention

[0003] In view of this, an embodiment of the present application provides a vehicle control method and device for improving vehicle driving safety.

[0004] In a first aspect, an embodiment of the present application provides a vehicle control method, including:

[0005] Collect information about the vehicle's surroundings and obtain the initial vehicle status;

[0006] Inputting the initial vehicle state into a strategy model, obtaining a plurality of lane adjustment strategies and an estimated value corresponding to each lane adjustment strategy of the plurality of lane adjustment strategies; the estimated value is used to characterize the adaptability of the lane adjustment strategy to the initial vehicle state;

[0007] Obtaining a target lane adjustment strategy based on a greedy algorithm and estimated values ​​corresponding to each lane adjustment strategy;

[0008] The vehicle is controlled to perform lane adjustment according to the target lane adjustment strategy.

[0009] As an optional implementation of the embodiment of the present application, after controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy, the method further includes:

[0010] Acquire an adjusted vehicle state; the adjusted vehicle state is a state of the vehicle after adjustment according to the target lane adjustment strategy;

[0011] Acquire a feedback value corresponding to the adjusted vehicle state; the feedback value is feedback obtained by controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy;

[0012] generating a quaternary group based on the adjusted vehicle state, the initial vehicle state, the target lane adjustment strategy, and the feedback value;

[0013] The four-tuple is saved in a database.

[0014] As an optional implementation of the embodiment of the present application, the method further includes:

[0015] When a preset condition is met, obtaining a plurality of quadruple groups from the database;

[0016] Obtaining corresponding estimated values ​​based on the adjusted vehicle states in each quadruple, and determining the maximum estimated value in each quadruple to obtain a first quadruple; the first quadruple is the quadruple to which the adjusted state with the maximum estimated value belongs;

[0017] Obtaining an expected estimated value based on the maximum estimated value and the feedback value in the first quaternary group;

[0018] updating the policy model based on the expected estimate;

[0019] After the target network model reaches a preset update number, the updating of the policy model is terminated.

[0020] As an optional implementation manner of the embodiment of the present application, obtaining the expected estimated value based on the maximum estimated value and the feedback value in the first quaternary group includes:

[0021] The expected estimated value is obtained based on a target network model, the maximum estimated value, and a feedback value in the first quadruple; wherein the target network model is the same as the strategy model.

[0022] As an optional implementation manner of the embodiment of the present application, the obtaining the expected estimated value based on the target network model, the maximum estimated value, and the feedback value in the first quadruple includes:

[0023] The expected estimated value is obtained according to the target network model, the maximum estimated value, the feedback value in the first quadruple, and the following formula:

[0024] Q′+α*(R+γ*max_Q-Q′)→Q′

[0025] Among them, Q′ is the expected estimated value, max_Q is the maximum estimated value, R is the feedback value, γ is the discount factor, which indicates the discount degree of the feedback value, and the value range is: 0≤γ≤1; α is the learning rate, and the value range is 0≤α≤1, which indicates the update speed of the expected estimated value.

[0026] As an optional implementation of the embodiment of the present application, the updating of the policy model based on the expected estimated value includes:

[0027] The model parameters of the policy model are updated based on a gradient descent algorithm and the expected estimated value.

[0028] In a second aspect, an embodiment of the present application provides a vehicle control device, including:

[0029] A first acquisition unit, used to collect vehicle surrounding environment information and obtain an initial vehicle state;

[0030] a processing unit, configured to input the initial vehicle state into a strategy model, and obtain a plurality of lane adjustment strategies and an estimated value corresponding to each lane adjustment strategy of the plurality of lane adjustment strategies; the estimated value is used to characterize the degree of adaptation of the lane adjustment strategy to the initial vehicle state;

[0031] A second acquisition unit, configured to acquire a target lane adjustment strategy based on a greedy algorithm and estimated values ​​corresponding to each lane adjustment strategy;

[0032] A control unit is used to control the vehicle to perform lane adjustment according to the target lane adjustment strategy.

[0033] As an optional implementation of the embodiment of the present application, the vehicle control device also includes a storage unit, which is specifically used to obtain an adjusted vehicle state; the adjusted vehicle state is the state of the vehicle after adjustment according to the target lane adjustment strategy; obtain a feedback value corresponding to the adjusted vehicle state; the feedback value is feedback obtained by controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy; generate a quadruple based on the adjusted vehicle state, the initial vehicle state, the target lane adjustment strategy and the feedback value; and save the quadruple to a database.

[0034] As an optional implementation of the embodiment of the present application, the vehicle control device also includes an updating unit, which is specifically used to obtain multiple quadruplets from the database when a preset condition is met; obtain corresponding estimated values ​​based on the adjusted vehicle states in each quadruplet, and determine the maximum estimated value in each quadruplet to obtain a first quadruplet; the first quadruplet is the quadruplet to which the adjusted state with the largest estimated value belongs; based on the maximum estimated value and the feedback value in the first quadruplet, an expected estimated value is obtained; based on the expected estimated value, the strategy model is updated, and after the target network model reaches a preset number of updates, the update of the strategy model is terminated.

[0035] As an optional implementation of the embodiment of the present application, the updating unit is also used to: obtain the expected estimated value based on the target network model, the maximum estimated value and the feedback value in the first quadruple; wherein the target network model is the same as the strategy model.

[0036] As an optional implementation of the embodiment of the present application, the updating unit is specifically used to obtain the expected estimated value according to the target network model, the maximum estimated value, the feedback value in the first quadruple, and the following formula:

[0037] Q′+α*(R+γ*max_Q-Q′)→Q′

[0038] Among them, Q′ is the expected estimated value, max_Q is the maximum estimated value, R is the feedback value, γ is the discount factor, which indicates the discount degree of the feedback value, and the value range is: 0≤γ≤1; α is the learning rate, and the value range is 0≤α≤1, which indicates the update speed of the expected estimated value.

[0039] As an optional implementation of the embodiment of the present application, the updating unit is specifically used to update the model parameters of the strategy model based on the gradient descent algorithm and the expected estimated value.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the vehicle control method described in any one of the above embodiments when executing the computer program.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the vehicle control method described in any one of the above embodiments.

[0042] In a fifth aspect, an embodiment of the present application provides a vehicle, comprising: the vehicle control device described in the second aspect or the electronic device described in the third aspect.

[0043] The vehicle control method provided in the embodiment of the present application is specifically as follows: first, the initial vehicle state is obtained by collecting vehicle surrounding environment information; the initial vehicle state is input into the strategy model to obtain multiple lane adjustment strategies and the estimated values ​​corresponding to each lane adjustment strategy; the estimated values ​​are used to characterize the degree of adaptability of the lane adjustment strategy to the initial vehicle state; based on the greedy algorithm and the estimated values ​​corresponding to each lane adjustment strategy, the target lane adjustment strategy is obtained from the multiple lane adjustment strategies; the vehicle is controlled to adjust the lane according to the target lane adjustment strategy. The embodiment of the present application obtains the initial vehicle state based on the collected vehicle environment information; then, based on the initial vehicle state, a lane adjustment strategy that is adapted to the current initial vehicle state is obtained, and the vehicle lane change is completed based on the lane adjustment strategy, thereby avoiding the situation where the vehicle can only change lanes according to the corresponding lane change strategy in a fixed scenario, so that the vehicle can cope with a variety of lane change scenarios and improve vehicle driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 One of the step flow charts of the vehicle control method provided in the embodiment of the present application;

[0047] Figure 2 A diagram showing a usage scenario of the vehicle control method provided in an embodiment of the present application;

[0048] Figure 3 The second step flow chart of the vehicle control method provided in the embodiment of the present application;

[0049] Figure 4 The third step flow chart of the vehicle control method provided in the embodiment of the present application;

[0050] Figure 5 A system architecture diagram of a vehicle control method provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a vehicle control device provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of a vehicle control device provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0057] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0058] The embodiment of the present application is a vehicle control method based on the DQN (Deep Q Network) algorithm, using Q-Learning algorithm, greedy algorithm, deep neural network (Deep Neural Network, DNN) technology, etc. Through the vehicle control method of the embodiment of the present application, it is possible to obtain a target lane adjustment strategy based on the vehicle's surrounding environment information under the condition that the vehicle changes lanes, so as to adjust the vehicle to complete the lane change.

[0059] Among them, the DQN algorithm is a classic algorithm of deep reinforcement learning. Usually, deep reinforcement learning is mostly used in scenarios that require interaction with the environment. That is, given a state of an environment, the program selects a corresponding behavior according to a certain policy. After executing this action, the environment will change, that is, the state will be converted to a new state, and the program will get a reward value after each action is executed. The program updates the strategy according to the size of the reward value, so that after all steps are executed, that is, when the state reaches the terminal state, the sum of the reward values ​​obtained is the largest.

[0060] The Q-Learning algorithm is a values-based algorithm in reinforcement learning. The final feedback is a table, namely the Q-value table. Each row of this table represents a state, each column of each row represents an action, and each value represents the maximum expected reward value that can be obtained if the action is taken in this state. The Q-value table can be used to find the optimal behavior in each state, and then the maximum expected reward can be obtained by finding all the optimal actions.

[0061] Based on the above content, the embodiment of the present application provides a vehicle control method, referring to Figure 1 As shown, the vehicle control method includes the following steps S101-S104:

[0062] S101, collecting vehicle surrounding environment information and obtaining an initial vehicle state.

[0063] In some embodiments, the initial vehicle state may be a plurality of parameters representing various environmental information such as vehicle speed, lane width, road curvature, traffic flow, etc.

[0064] In some embodiments, in the above step S101, the method for collecting vehicle surrounding environment information and obtaining the initial vehicle state may include the following steps 1 and 2:

[0065] Step 1: Collect vehicle surrounding environment information.

[0066] In some embodiments, the collected vehicle surrounding environment information can be obtained through a sensor module installed on the vehicle. Specifically, the sensor module can obtain the vehicle speed through a vehicle speed sensor; the current environmental information collected by sensors such as cameras, lidars, millimeter-wave radars, etc. is subjected to multi-sensor fusion, and the lane width and road curvature are obtained after calculation.

[0067] Step 2: preprocess the vehicle surrounding environment information to obtain the initial vehicle state.

[0068] In some embodiments, the preprocessing may include performing data fusion, filtering, noise reduction and other operations on the vehicle surrounding environment information to improve the quality of the vehicle surrounding environment information to ensure the accuracy of the initial vehicle state;

[0069] S102: Input the initial vehicle state into a strategy model to obtain a plurality of lane adjustment strategies and an estimated value corresponding to each of the plurality of lane adjustment strategies.

[0070] The estimated value is used to characterize the degree of adaptability between the lane adjustment strategy and the initial vehicle state.

[0071] In some embodiments, the strategy model may be a model obtained by training a preset machine learning model based on a sample data set, and the sample data set may include: multiple sample data, the sample data including: a sample initial vehicle state, multiple lane adjustment strategies corresponding to the sample initial vehicle state, and estimated values ​​corresponding to each lane adjustment strategy of the multiple lane adjustment strategies; wherein, the preset machine learning model may be other machine learning models such as a deep neural network DNN model.

[0072] In some embodiments, the lane adjustment strategy may be a strategy that includes a series of actions for controlling the current vehicle to change lanes. The lane adjustment strategy may be used to control the vehicle's steering, acceleration, deceleration and other actions to achieve automatic lane adjustment.

[0073] For example, when the vehicle changes lanes from lane 1 to lane 2, the corresponding lane adjustment strategy may include four actions, namely: reducing the speed, turning on the right turn signal, rotating the steering wheel, and returning the steering wheel to the center.

[0074] S103: Obtain a target lane adjustment strategy based on a greedy algorithm and estimated values ​​corresponding to each lane adjustment strategy.

[0075] Specifically, the greedy algorithm (Greedy Algorithm), also known as the greedy algorithm, is a common method for finding the optimal solution to the problem. This method mode generally divides the solution process into several steps, but each step applies the greedy principle, selects the best or optimal option (the most favorable local option) under the current state, and on this basis, obtains the stacked final result which is also the best or optimal solution; each time the corresponding result is obtained according to the initial state of the vehicle and the strategy model, the greedy algorithm can ensure that the optimal lane adjustment strategy is used as the target lane adjustment strategy among the multiple lane adjustment strategies according to the initial state of the vehicle, that is, the lane adjustment strategy with the largest estimated value corresponding to the lane adjustment strategy obtained this time; and saves this estimated value, so that when the same initial state of the vehicle is input into the strategy model next time, the estimated value of the optimal lane adjustment strategy is used as the minimum screening criterion for screening, and then the optimal lane adjustment strategy of this time is obtained.

[0076] In some embodiments, the lane adjustment strategy with the largest estimated value is usually selected with a probability of 1-ε, and a lane adjustment strategy is randomly selected with a probability of ε, where ε is a value between 0 and 1; at the beginning of training, ε can be set to a larger value, and ε will gradually decrease over time. For example, when ε is 0.1, it means that the lane adjustment strategy with the largest estimated value will be selected with a probability of 90%, and the lane adjustment strategy will be randomly selected with a probability of 10%. At the beginning of training, since it is uncertain which lane adjustment strategy has a better estimated value, a lot of training is required for exploration. As the number of training increases, it is relatively certain which lane adjustment strategy corresponds to the estimated value that is better; the exploration can be reduced, and the value of ε can be gradually reduced, that is, the lane adjustment strategy is mainly obtained based on the estimated value.

[0077] S104: Control the vehicle to perform lane adjustment according to the target lane adjustment strategy.

[0078] In some embodiments, reference Figure 2 As shown, it is a schematic diagram of controlling vehicle lane change based on the vehicle control method provided in an embodiment of the present application. There is a vehicle 22 driving normally in front of lane 1 where the current vehicle 21 is located, and there is no vehicle driving in front and behind lane 2 located on the right side of lane 1. At this time, the lane change condition is met, and the current vehicle 21 can be changed from lane 1 to lane 2. The lane adjustment strategy for changing the vehicle 21 to lane 2 can be: reducing the speed, turning on the right turn signal, rotating the steering wheel, and returning the steering wheel to the center. After adjusting the vehicle according to the lane adjustment strategy, the lane change of the vehicle 21 is completed, and then the vehicle 21 can drive normally.

[0079] The vehicle control method provided in the embodiment of the present application is specifically as follows: first, the initial vehicle state is obtained by collecting vehicle surrounding environment information; the initial vehicle state is input into the strategy model to obtain multiple lane adjustment strategies and the estimated values ​​corresponding to each lane adjustment strategy; the estimated values ​​are used to characterize the degree of adaptability of the lane adjustment strategy to the initial vehicle state; based on the greedy algorithm and the estimated values ​​corresponding to each lane adjustment strategy, the target lane adjustment strategy is obtained from the multiple lane adjustment strategies; the vehicle is controlled to adjust the lane according to the target lane adjustment strategy. The embodiment of the present application obtains the initial vehicle state based on the collected vehicle environment information; then, based on the initial vehicle state, a lane adjustment strategy that is adapted to the current initial vehicle state is obtained, and the vehicle lane change is completed based on the lane adjustment strategy, thereby avoiding the situation where the vehicle can only change lanes according to the corresponding lane change strategy in a fixed scenario, so that the vehicle can cope with a variety of lane change scenarios and improve vehicle driving safety.

[0080] As an extension and refinement of the above embodiment, the present application embodiment provides a vehicle control method, referring to Figure 3 As shown, after the above step S104 (controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy), the vehicle control method includes the following steps S301 to S306:

[0081] S301. Obtain adjusted vehicle status.

[0082] The adjusted vehicle state is the state of the vehicle after adjustment according to the target lane adjustment strategy.

[0083] In some embodiments, the vehicle control method strategy model based on the deep reinforcement learning algorithm obtains the initial vehicle state by observing and collecting the surrounding environment of the vehicle. The strategy model then obtains the lane adjustment strategy that best suits the initial environment of the vehicle based on the initial vehicle state. At this time, a feedback value will be obtained, and because the vehicle changes lanes, the vehicle state will also change accordingly. Therefore, a new vehicle state, that is, the adjusted vehicle state, will be obtained, and it will continue to be executed to perform deep reinforcement learning and improve the model operation effect. Based on the above, it is necessary to obtain the adjusted vehicle state after controlling the vehicle to adjust the lane according to the target lane adjustment strategy.

[0084] S302: Obtain a feedback value corresponding to the adjusted vehicle state.

[0085] The feedback value is feedback obtained by controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy.

[0086] In some embodiments, when obtaining the feedback value, multiple aspects need to be considered, for example, the safety, lane change time, comfort, etc. of the vehicle after the vehicle changes lanes according to the target lane change strategy. Specifically, positive feedback is given when the lane adjustment is successful and safe, and negative feedback is given when a collision occurs or the vehicle deviates seriously from the lane.

[0087] S303: Generate a quaternary group based on the adjusted vehicle state, the initial vehicle state, the target lane adjustment strategy, and the feedback value.

[0088] S304: Save the quadruple to a database.

[0089] In some embodiments, multiple quads in the database are stored in an experience buffer (ExperienceReplay Buffer), which can store n quads, each of which can be called experience. A certain scheme is used to interact with the environment, and multiple experiences are obtained and stored in the experience buffer. The experiences in the experience buffer may come from different schemes. The replay buffer will delete old data only when it is full. Each time, a batch of experience data (i.e., a batch of quads) is randomly extracted, and this batch of quads is input into the target network, multiple random gradients are calculated, and the network of the policy model is updated with the average of the gradients.

[0090] As an extension and refinement of the above embodiment, the present application embodiment provides a vehicle control method, referring to Figure 4 As shown, including the process of optimizing the strategy model, the vehicle control method includes the following steps S401 to S404:

[0091] S401. When a preset condition is met, obtain a plurality of quadruples from the database.

[0092] In some embodiments, the preset condition may be a fixed duration. For example, when the policy model is updated, a batch of quadruplets may be extracted from the database every 5 minutes to obtain the expected estimated value.

[0093] S402: Acquire corresponding estimated values ​​based on the adjusted vehicle states in each quadruple, and determine the maximum estimated value in each quadruple to acquire a first quadruple.

[0094] The first quaternary group is the quaternary group to which the adjusted state with the largest estimated value belongs.

[0095] S403: Obtain an expected estimated value based on the maximum estimated value and the feedback value in the first quaternary group.

[0096] In some embodiments, the above step S403 may be implemented by the following steps:

[0097] The expected estimated value is obtained based on the target network model, the maximum estimated value, and the feedback value in the first quadruple.

[0098] The target network model is the same as the policy model.

[0099] In some embodiments, the target network model is to avoid the instability caused by obtaining the maximum estimated value while updating the policy model when obtaining the maximum estimated value based on the adjusted vehicle state. In order to enable real-time update, it will cause trajectory confusion and inconvenience for model training. Therefore, a target network is introduced to obtain the maximum estimated value from the target network, and then obtain the expected estimated value to update the policy model.

[0100] Specifically, the method for obtaining the expected estimated value based on the target network model, the maximum estimated value and the feedback value in the first quaternary group may be:

[0101] The expected estimated value is obtained according to the target network model, the maximum estimated value, the feedback value in the first quadruple, and the following formula:

[0102] Q′+α*(R+γ*max_Q-Q′)→Q′

[0103] Among them, Q′ is the expected estimated value, max_Q is the maximum estimated value, R is the feedback value, γ is the discount factor, which indicates the discount degree of the feedback value, and the value range is: 0≤γ≤1; α is the learning rate, and the value range is 0≤α≤1, which indicates the update speed of the expected estimated value.

[0104] S404: Update the policy model based on the expected estimated value.

[0105] In some embodiments, the above step S404 may be implemented by the following steps:

[0106] The model parameters of the policy model are updated based on a gradient descent algorithm and the expected estimated value.

[0107] Specifically, the gradient descent algorithm is an optimization algorithm for finding the minimum value of a function. Gradient descent is usually used to make the value of the loss function as small as possible, that is, to make the predicted value (actual output) of the neural network and the label value (expected output) as close as possible. In this process, the network parameters (weights and biases of each layer) will be adjusted, which is exactly the training principle of the neural network.

[0108] In the embodiment of the present application, the following formula is used to minimize the loss function to obtain the neural network parameters for updating the strategy model:

[0110] θ=E[(R+γ*max_a′Q(s′,a′,θ′)-Q(s,a,θ))]

[0111] Among them, θ′ represents the parameters of the target network, which is used to calculate the expected estimate to improve training stability, s′ represents the new vehicle state, and a′ represents the lane adjustment strategy corresponding to s′.

[0112] S405: After the target network model reaches a preset update number, the updating of the policy model is terminated.

[0113] Specifically, when the parameters of the strategy model converge or reach a predetermined number of training times, the strategy model has learned how to select and output the lane adjustment strategy with the highest degree of adaptability according to a given vehicle state.

[0114] In combination with the above embodiments, refer to Figure 5 As shown, it is a system architecture diagram based on the vehicle control method, including:

[0115] The information collection module 51 is used to collect the surrounding environment information of the current vehicle. Specifically, the surrounding environment information of the vehicle can be obtained through various sensors.

[0116] The data processing module 52 is used to pre-process the vehicle surrounding environment information to obtain the initial vehicle state.

[0117] The strategy model 53 is used to receive the initial vehicle state, and obtain a target lane adjustment strategy based on the initial vehicle state through inference and calculation by the strategy model.

[0118] The vehicle control module 54 is configured to receive the target lane adjustment strategy and control the vehicle to complete lane adjustment based on the target lane adjustment strategy.

[0119] The updating and optimizing module 55 is used to update the model parameters of the strategy model based on the historical data stored in the database to optimize the strategy model.

[0120] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present invention also provides a vehicle control device, which corresponds to the above method embodiment. For ease of reading, this embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the vehicle control device in this embodiment can correspond to all the contents in the above method embodiment.

[0121] The embodiment of the present application provides a vehicle control device, Figure 6 is a schematic diagram of the structure of the vehicle control device, such as Figure 7 As shown, the vehicle control device 600 includes:

[0122] The first acquisition unit 601 is used to collect vehicle surrounding environment information and obtain an initial vehicle state;

[0123] A processing unit 602 is used to input the initial vehicle state into a strategy model, and obtain a plurality of lane adjustment strategies and an estimated value corresponding to each lane adjustment strategy of the plurality of lane adjustment strategies; the estimated value is used to characterize the adaptability of the lane adjustment strategy to the initial vehicle state;

[0124] A second acquisition unit 603, configured to acquire a target lane adjustment strategy based on a greedy algorithm and estimated values ​​corresponding to each lane adjustment strategy;

[0125] The control unit 604 is used to control the vehicle to perform lane adjustment according to the target lane adjustment strategy.

[0126] As an optional implementation of the present application, refer to Figure 6 As shown, the vehicle control device also includes a storage unit 605, which is specifically used to obtain an adjusted vehicle state; the adjusted vehicle state is the state of the vehicle after adjustment according to the target lane adjustment strategy; obtain a feedback value corresponding to the adjusted vehicle state; the feedback value is feedback obtained by controlling the vehicle to adjust the lane according to the target lane adjustment strategy; generate a four-tuple based on the adjusted vehicle state, the initial vehicle state, the target lane adjustment strategy and the feedback value; and save the four-tuple to a database.

[0127] As an optional implementation of the present application, refer to Figure 7 As shown, the vehicle control device further includes an updating unit 606, which is specifically used to obtain multiple quadruplets from the database when the preset conditions are met; obtain corresponding estimated values ​​based on the adjusted vehicle states in each quadruplet, and determine the maximum estimated value in each quadruplet to obtain the first quadruplet; the first quadruplet is the quadruplet to which the adjusted state with the largest estimated value belongs; obtain an expected estimated value based on the maximum estimated value and the feedback value in the first quadruplet; update the strategy model based on the expected estimated value; and end the update of the strategy model after the target network model reaches the preset update times. .

[0128] As an optional implementation of the embodiment of the present application, the updating unit 606 is specifically used to obtain the expected estimated value based on the target network model, the maximum estimated value and the feedback value in the first quadruple; wherein the target network model is the same as the strategy model.

[0129] As an optional implementation of the embodiment of the present application, the updating unit is specifically used to obtain the expected estimated value according to the target network model, the maximum estimated value, the feedback value in the first quadruple, and the following formula;

[0130] Among them, is the expected estimated value, is the maximum estimated value, is the feedback value, is the discount factor, which indicates the degree of discount on the feedback value, and has a value range of: 1; is the learning rate, has a value range of, and indicates the update speed of the expected estimated value.

[0131] As an optional implementation of the embodiment of the present application, the updating unit 606 is specifically used to update the model parameters of the strategy model based on the gradient descent algorithm and the expected estimated value.

[0132] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device provided in this embodiment includes: a memory 801 and a processor 802, wherein the memory 801 is used to store a computer program; and the processor 802 is used to execute the vehicle control method provided in the above embodiment when executing the computer program.

[0133] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the vehicle control method provided in the above embodiment.

[0134] Based on the same inventive concept, an embodiment of the present application further provides a vehicle, which includes the vehicle control device provided by the above embodiment or the electronic device provided by the above embodiment.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0136] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0137] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0138] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A vehicle control method, characterized in that: include: Collect information about the vehicle's surroundings and obtain the initial vehicle status; Inputting the initial vehicle state into a strategy model, obtaining a plurality of lane adjustment strategies and an estimated value corresponding to each lane adjustment strategy; the estimated value is used to characterize the degree of adaptability between the lane adjustment strategy and the initial vehicle state; Based on the greedy algorithm and the estimated values ​​corresponding to the lane adjustment strategies, obtaining a target lane adjustment strategy from the multiple lane adjustment strategies; The vehicle is controlled to perform lane adjustment according to the target lane adjustment strategy.

2. The method according to claim 1, characterized in that After controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy, the method further includes: Acquire an adjusted vehicle state; the adjusted vehicle state is a state of the vehicle after adjustment according to the target lane adjustment strategy; Acquire a feedback value corresponding to the adjusted vehicle state; the feedback value is feedback obtained by controlling the vehicle to perform lane adjustment according to the target lane adjustment strategy; generating a quaternary group based on the adjusted vehicle state, the initial vehicle state, the target lane adjustment strategy, and the feedback value; The four-tuple is saved in a database.

3. The method according to claim 2, characterized in that The method further comprises: When a preset condition is met, obtaining a plurality of quadruple groups from the database; Obtaining corresponding estimated values ​​based on the adjusted vehicle states in each quadruple, and determining the maximum estimated value in each quadruple to obtain a first quadruple; the first quadruple is the quadruple to which the adjusted state with the maximum estimated value belongs; Obtaining an expected estimated value based on the maximum estimated value and the feedback value in the first quaternary group; updating the policy model based on the expected estimate; After the target network model reaches a preset update number, the updating of the policy model is terminated.

4. The method according to claim 3, characterized in that The obtaining of the expected estimated value based on the maximum estimated value and the feedback value in the first quaternary group includes: The expected estimated value is obtained based on a target network model, the maximum estimated value, and a feedback value in the first quadruple; wherein the target network model is the same as the strategy model.

5. The method according to claim 4, characterized in that The obtaining the expected estimated value based on the target network model, the maximum estimated value, and the feedback value in the first quadruple includes: The expected estimated value is obtained according to the target network model, the maximum estimated value, the feedback value in the first quadruple, and the following formula: Q′+α*(R+γ*max_Q-Q′)→Q′ Among them, Q′ is the expected estimated value, max_Q is the maximum estimated value, R is the feedback value, γ is the discount factor, which indicates the discount degree of the feedback value, and the value range is: 0≤γ≤1; α is the learning rate, and the value range is 0≤α≤1, which indicates the update speed of the expected estimated value.

6. The method according to claim 3, characterized in that The updating of the policy model based on the expected estimated value comprises: The model parameters of the policy model are updated based on a gradient descent algorithm and the expected estimated value.

7. A vehicle control device, characterized in that: include: A first acquisition unit, used to collect vehicle surrounding environment information and obtain an initial vehicle state; a processing unit, configured to input the initial vehicle state into a strategy model, and obtain a plurality of lane adjustment strategies and an estimated value corresponding to each lane adjustment strategy of the plurality of lane adjustment strategies; the estimated value is used to characterize the degree of adaptation of the lane adjustment strategy to the initial vehicle state; A second acquisition unit, configured to acquire a target lane adjustment strategy based on a greedy algorithm and estimated values ​​corresponding to each lane adjustment strategy; A control unit is used to control the vehicle to perform lane adjustment according to the target lane adjustment strategy.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the vehicle control method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the vehicle control method according to any one of claims 1 to 6.

10. A vehicle, characterized in that: include: The vehicle control device according to claim 7 or the electronic device according to claim 8.