Intelligent automobile lane-changing decision method and device, electronic equipment and storage medium

Abstract

The intelligent automobile lane-changing decision method and device, the electronic device and the storage medium provided by the embodiments of the present disclosure comprise: constructing a virtual lane-changing scene by using a simulation method; constructing a decision model for determining a mapping function between a state and a reward, obtaining a Q table for evaluating the lane-changing feasibility of an intelligent automobile based on the mapping function, and obtaining a lane-changing decision, i.e., "lane-changing" or "not lane-changing"; based on a greedy algorithm, making the intelligent automobile produce a random lane-changing behavior in the constructed virtual lane-changing scene, storing the state and the reward of the current lane-changing behavior in an experience replay pool, when the number of state-reward pairs in the experience replay pool reaches a minimum sample number, randomly sampling a plurality of state-reward pairs from the experience replay pool for single-step training of the decision model, storing a new state-reward pair generated in the training process in the experience replay pool, repeatedly performing the above training process until a maximum training number is reached, and obtaining a lane-changing decision model. The lane-changing decision generated by the present disclosure is high in safety and wide in applicability.

Description

Technical Field

[0001] This disclosure pertains to the field of autonomous driving technology, and particularly relates to a method, apparatus, and storage medium for generating lane-changing decision models for intelligent vehicles. Background Technology

[0002] Autonomous driving methods mainly include environmental perception, driving decision-making, and vehicle control. Among these, lane changing (lane switching) decisions are a common lateral movement behavior of vehicles. The methods are mainly divided into rule-based and machine learning methods, with the latter further divided into supervised learning and reinforcement learning branches. Essentially, both establish a mapping relationship between perceived information and decision-making.

[0003] Rule-based methods are mainly in the form of finite state machines (FSMs), which rely heavily on rules and thresholds set by engineers and are poorly adaptable to complex traffic environments. In addition, the rules are complex and there are many calibration parameters. Moreover, safety is often the core objective, and the strategies are often too conservative, which seriously affects traffic efficiency.

[0004] Learning-based methods primarily utilize supervised learning and reinforcement learning from machine learning, which are highly adaptable. However, the former relies on collecting large amounts of real-world road data for training, resulting in significant costs and time consumption, and poses safety issues such as poor interpretability due to its "black box" nature. The latter relies on training agents in simulated environments to achieve similar results, but reinforcement learning requires selecting specific technologies such as DDPG and DQN frameworks based on the discreteness of input and output variables, the key variables involved in the specific problem, the applicable scenarios, and parameters. Existing solutions for safe lane-changing decisions for vehicles are few and do not consider lane-changing under different traffic phases, resulting in poor generalization ability of their decisions under different traffic densities and speeds in the target lane.

[0005] Deep reinforcement learning (DRL) is a method combining reinforcement learning and deep neural networks (DNNs). It uses a DNN to fit the state-action reward Q-function (value function) to adapt it to continuous state variables (corresponding to vehicle surrounding information) as decision input. The difference between single-step reinforcement learning and general reinforcement learning lies in the computational cost compared at each step of the exploration strategy. The former selects the action with the maximum feedback in a single step, while the latter selects the action with the maximum feedback over multiple steps. Currently, there are few techniques that utilize single-step reinforcement learning to generate lane-changing decisions. Summary of the Invention

[0006] This disclosure aims to address at least one of the technical problems existing in the prior art.

[0007] Therefore, the lane-changing decision-making method for intelligent vehicles provided in the first aspect of this disclosure has the advantages of high training efficiency, strong adaptability, good safety robustness, strong engineering applicability, strong interpretability of training parameter adjustment, and high safety.

[0008] The lane-changing decision-making method for intelligent vehicles provided in the first aspect of this disclosure includes:

[0009] 1) A virtual lane-changing scenario is constructed using simulation methods. The virtual lane-changing scenario includes lanes, intelligent vehicles, and randomly generated traffic flow. The lanes include at least the lane where the intelligent vehicle is located and the target lane into which the intelligent vehicle is expected to change lanes.

[0010] 2) Construct a decision model based on a single-step deep Q-neural network to determine the mapping function between the state and the reward. Based on the mapping function, obtain a Q-table for evaluating the feasibility of lane changing for intelligent vehicles, and thus obtain the lane changing decision, i.e., "change lane" or "do not change lane".

[0011] 3) In the constructed virtual lane-changing scenario, the intelligent vehicle generates random lane-changing behavior based on a greedy algorithm. The state and reward of the current lane-changing behavior are recorded to form a current state-reward pair. The current state-reward pair is stored in the experience replay pool. When the number of state-reward pairs in the experience replay pool reaches the minimum number of samples, a number of state-reward pairs are randomly sampled from the experience replay pool as a batch of training samples to train the decision model step by step. The new state-reward pairs generated during the training process are stored in the experience replay pool. The above training process is repeated until the maximum number of training times is reached to obtain the lane-changing decision model.

[0012] 4) Obtain the state of the actual lane-changing scenario in which the intelligent vehicle is located, and input the state into the lane-changing decision model to obtain the lane-changing decision of the intelligent vehicle.

[0013] In some embodiments, in step 1), the parameters of the randomly generated traffic flow include: average vehicle speed of the lane, traffic flow density of the lane, and center-to-center distance between two adjacent vehicles in the lane.

[0014] In some embodiments, the maximum number of training iterations is determined according to the following steps:

[0015] Different traffic flow conditions are added to the constructed virtual lane-changing scenario. For each condition, the intelligent vehicle is made to perform lane-changing tests in different relative states with vehicles in the target lane. The number of collisions after the intelligent vehicle performs the lane-changing action is counted. The number of lane-changing tests when the number of collisions converges to close to 0 is taken as the total number of lane-changing tests for each condition. The maximum value among the total number of lane-changing tests for all conditions is taken as the maximum number of training iterations.

[0016] In some embodiments, the maximum number of training iterations is determined by following these steps:

[0017] 311) Add different traffic flow conditions to the constructed virtual lane-changing scenario.

[0018] Based on traffic flow theory, traffic flow conditions of free phase, unobstructed phase, forced phase and congested phase are established in the virtual lane changing scenario, and the operating parameters of each traffic phase are set respectively, including traffic flow speed, traffic density and center distance between two vehicles in the same lane.

[0019] 312) For each traffic flow condition, conduct several lane-changing tests. Each lane-changing test includes the following steps:

[0020] 3121) At the initial moment of the test, all vehicles are stationary, the center distance between adjacent vehicles in the target lane is the center distance set in step 311), and the initial value of the distance between the vehicle and the vehicle in front in the current lane is set.

[0021] 3122) Start the vehicles in the target lane and control the vehicles in the target lane to reach the traffic flow speed set in step 311). Throughout the process, the vehicles in the target lane maintain a constant distance between each other. After a delay, start the vehicle and control the vehicle to reach the traffic flow speed set in step 311).

[0022] 3123) After the vehicle reaches the traffic flow speed set in step 311), the vehicle in front in the vehicle lane begins to decelerate. When the vehicle in front decelerates to the first speed value, the vehicle also decelerates after a delay and follows the vehicle in front. When the vehicle in front decelerates to the second speed value, the vehicle begins to judge the feasibility of changing lanes. After judging that it is possible to change lanes, it executes the lane changing action.

[0023] 313) For each traffic flow condition, after a certain number of lane change tests, the number of collisions that occur after the vehicle performs the lane change action in the certain number of lane change tests is counted. The number of lane change tests when the number of collisions converges to close to 0 is taken as the total number of lane change tests for each condition. The maximum value among the total number of lane change tests for all conditions is taken as the maximum number of training tests.

[0024] In some embodiments, in steps 3122) and 3123), the delay time of the vehicle is randomly set.

[0025] In some embodiments, step 3) specifically includes the following steps:

[0026] 31) Set the cycle number to j and initialize j = 1; initialize the network parameters θ of the decision model; construct the Q-table and initialize all Q values ​​in the Q-table to 0; set the maximum number of training iterations to M; construct the experience replay pool D and initialize it as an empty set; set the minimum number of samples in the experience replay pool D to N. start Let N be the number of state-reward pairs randomly collected from the experience replay pool D each time; let ε be the probability parameter of the greedy algorithm. j Let its initial and final values ​​be ε. start and ε endLet ε be the probability decay parameter of the greedy algorithm. decay ;

[0027] 32) If j < M, then proceed to step 33); if j = M, then the training ends and the lane-changing decision model is obtained.

[0028] 33) Randomly generate state-reward pairs based on a greedy algorithm and store them in the experience replay pool D, specifically including:

[0029] 331) In a virtual lane-changing scenario, the state is randomly initialized, and the lane-changing action of the vehicle is initialized to not changing lanes. A traffic flow simulating human driving behavior is randomly generated in the target lane, and the state s is updated. j Let the expression for the state be:

[0030]

[0031] In the formula, v e and These are the vehicle speed and its normalized threshold, respectively; v l and These are the vehicle speed of the vehicle ahead in the target lane and its normalized threshold, respectively; v f and These represent the speed of the following vehicle within the target lane and its normalized threshold, respectively; d l and These are the net distance between the vehicle and the vehicle ahead in the target lane, and their normalized threshold values, respectively; d f and These are the net distance between the vehicle and the following vehicle in the target lane, and the normalized threshold of that net distance, respectively.

[0032] 332) Generate random numbers rnd∈(0,1), if rnd<ε j Then, a random number rnd2 ∈ (0, 1) is generated. When rnd2 > 1 / 2, the lane-changing action of the vehicle is selected as lane changing, and step 334) is executed. When rnd2 ≤ 1 / 2, the lane-changing action of the vehicle is selected as not changing lanes, and step 335) is executed. If rnd > ε j Then proceed to step 333);

[0033] 333) Read the state s from the current Q table. j The corresponding Q value Q(s) j If Q(s) j If Q(s) > 0, then the lane-changing action of the vehicle is selected as lane change, and step 334 is executed. j If )≤0, then the lane-changing action of the vehicle is selected as no lane change, and step 332 is executed;

[0034] 334) Enable the vehicle to perform a lane change operation in a virtual lane change scenario and collect the current state s. jand the corresponding reward r j The reward r is set based on whether a collision occurs after the lane change operation. j The expression is as follows:

[0035]

[0036] In the formula, r s As a reward for a successful lane change, r f A reward for failing to change lanes;

[0037] State s j and reward r j State reward pair (s) j r j Store the data in the experience replay pool D and proceed to step 335.

[0038] 335) Determine whether the number of state reward pairs in the experience replay pool D has reached the minimum sample size N. start If the condition is met, proceed to step 34); otherwise, proceed to step 35.

[0039] 34) Randomly sample several state-reward pairs from the experience replay pool as a batch of training samples to train the decision model, specifically including:

[0040] 341) Randomly sample N state-reward pairs (s) from the experience replay pool D. i r i This constitutes the current training sample set X. i , i∈[1,N];

[0041] 342) Change the current training sample set X i Each state s in i Input the decision model to obtain the corresponding Q value Q(s) i This enables the decision-making model to learn the mapping relationship between state and reward;

[0042] 343) Update parameters

[0043] Update table Q according to the following formula:

[0044] Q(s i )←(1-η)Q(s i )+ηr i

[0045] Where η is the learning rate;

[0046] The network parameters of the decision model are updated according to the following formula to reduce the loss function of the decision model:

[0047]

[0048] Update the probability parameter ε of the greedy algorithm according to the following formula. j :

[0049]

[0050] Perform step 35);

[0051] 35) Let j = j + 1, then return to step 32).

[0052] In some embodiments, the lane change success reward r is determined according to the following steps. s And the lane change failure reward r f :

[0053] Using a method similar to logistic regression, let p(r) s |s)>-r f / (r s -r f When Q(s) = p(r) is satisfied s |s)r s +p(r f |s)r f When >0, we get r f / r s =-5, let r s =1, r f = -5, where p(r) s |s) indicates that a lane change was successful in state s, i.e., a reward r was obtained. s The probability, p(r) f |s) indicates that a lane change failed in state s, i.e., a collision occurred and a reward r was obtained. f The probability of.

[0054] The intelligent vehicle lane-changing decision-making device provided in the second aspect of this disclosure includes:

[0055] The first module is configured to construct a virtual lane-changing scenario using a simulation method. The virtual lane-changing scenario includes lanes, intelligent vehicles, and randomly generated traffic flow. The lanes include at least the lane where the intelligent vehicle is located and the target lane into which the intelligent vehicle is expected to change lanes.

[0056] The second module is configured to construct a decision model based on a single-step deep Q-neural network to determine the mapping function between the state and the reward. Based on the mapping function, a Q-table is obtained to evaluate the feasibility of lane changing for intelligent vehicles, thereby obtaining a lane-changing decision, namely "change lane" or "do not change lane".

[0057] The third module is configured to enable the intelligent vehicle to generate random lane-changing behavior based on a greedy algorithm in the constructed virtual lane-changing scenario, record the state and reward of the current lane-changing behavior to form a current state-reward pair, and store the current state-reward pair in the experience replay pool. When the number of state-reward pairs in the experience replay pool reaches the minimum number of samples, a number of state-reward pairs are randomly sampled from the experience replay pool as a batch of training samples to train the decision model step by step, and new state-reward pairs generated during the training process are stored in the experience replay pool. The above training process is repeated until the maximum number of training times is reached to obtain the lane-changing decision model.

[0058] The fourth module is configured to obtain the state of the actual lane-changing scenario in which the intelligent vehicle is located, and input the state into the lane-changing decision model to obtain the lane-changing decision of the intelligent vehicle.

[0059] The electronic device provided in the third aspect embodiment of this disclosure includes:

[0060] At least one processor, and a memory communicatively connected to said at least one processor;

[0061] The memory stores instructions executable by the at least one processor, the instructions being configured to execute an intelligent vehicle lane-changing decision method provided according to any embodiment of the first aspect of this disclosure.

[0062] The computer-readable storage medium provided in the fourth aspect of this disclosure is characterized in that the computer-readable storage medium stores computer instructions for causing the computer to execute the intelligent vehicle lane-changing decision method provided in any embodiment of the first aspect of this disclosure.

[0063] The embodiments disclosed herein have the following features and beneficial effects:

[0064] 1. Save on control strategy development costs: Reduce the workload, calibration time, and economic costs for personnel calibrating control parameters.

[0065] 2. Fast and high-quality training and validation of strategies: Compared with the method of relying on real vehicles to collect a large amount of data on the road to train the control parameters of the neural network, the virtual environment can provide a large amount of simulation scenario data to train the agent to output reliable control strategies; the virtual environment constructs a variety of traffic flow scenarios that can better reflect the actual traffic flow of human drivers, ensuring the generalization ability of the decision module; at the same time, the vehicle adopts an academically recognized accurate dynamic model, which simulates the lane changing process more realistically and ensures that the decision model has strong adaptability to lane changing processes of different speeds and trajectories.

[0066] 3. The DQN algorithm was adjusted for lane-changing decisions: a single-step reward value inspired by the principle of logistic regression was adopted, and the state reached during training convergence was close to the theoretically set lane-changing success rate. Compared with the general multi-step reward DQN, the single-step DQN method converges the number of collisions rapidly under the same number of training cycles, indicating that this method is efficient and safe for lane-changing decisions.

[0067] 3. Easy to deploy: Neural networks have a simple structure and good real-time performance, and can be quickly deployed in MCUs (Microcontroller Units) with minimal increase in memory requirements for the control chip. Attached Figure Description

[0068] Figure 1 This is an overall flowchart of the intelligent vehicle lane-changing decision-making method provided in the first aspect of this disclosure.

[0069] Figure 2 This is a schematic diagram of the structure of a decision model based on a single-step deep Q-neural network constructed in the method provided in the first aspect of this disclosure.

[0070] Figure 3 This is a state diagram of a lane-changing scenario in the method provided by the first aspect of this disclosure.

[0071] Figure 4 This is a flowchart of the method for training a decision model provided in the first aspect of this disclosure.

[0072] Figure 5 This is a graph showing the change in the number of collisions during the trial process for determining the maximum number of cycles in the method provided in the first aspect of this disclosure, as a function of the number of training rounds.

[0073] Figure 6 This is a graph showing the change in the number of collisions during lane-changing decisions generated by the single-step DQN method provided in the first aspect of this disclosure and the existing DQN method, as a function of the number of training iterations.

[0074] Figure 7 This is a schematic diagram of the structure of an electronic device provided in a third aspect embodiment of the present disclosure. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0076] Conversely, this application covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined by the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.

[0077] See Figure 1 The intelligent vehicle lane-changing decision-making method provided in the first aspect of this disclosure includes:

[0078] 1) Construct a virtual lane-changing scenario using simulation methods. The virtual lane-changing scenario includes lanes, intelligent vehicles, and randomly generated traffic flow. The lanes include at least the lane where the intelligent vehicle is located and the target lane into which the intelligent vehicle is expected to change.

[0079] 2) Construct a decision model based on a single-step deep Q-neural network to determine the mapping function between the state and the reward. Based on this mapping function, obtain the Q-table for evaluating the feasibility of lane changing for intelligent vehicles, and thus obtain the lane changing decision, i.e., "change lane" or "do not change lane".

[0080] 3) In the constructed virtual lane-changing scenario, the intelligent vehicle generates random lane-changing behavior based on a greedy algorithm. The state and reward of the current lane-changing behavior are recorded to form a current state-reward pair. The current state-reward pair is stored in the experience replay pool. When the number of state-reward pairs in the experience replay pool reaches the minimum number of samples, a number of state-reward pairs are randomly sampled from the experience replay pool as a batch of training samples to train the decision model step by step. New state-reward pairs generated during the training process are stored in the experience replay pool. The above training process is repeated until the maximum number of training times is reached to obtain the lane-changing decision model. The maximum number of training times is determined according to the following steps: Different traffic flow conditions are added to the constructed virtual lane-changing scenario. For each condition, the intelligent vehicle is tested for lane changing in different relative states with vehicles in the target lane. The number of collisions after the intelligent vehicle performs the lane-changing action is counted. The number of lane-changing tests when the number of collisions converges to close to 0 is taken as the total number of lane-changing tests for each condition. The maximum value among the total number of lane-changing tests for all conditions is taken as the maximum number of training times.

[0081] 4) Obtain the state of the actual lane-changing scenario in which the intelligent vehicle is located, and input the state into the lane-changing decision model to obtain the lane-changing decision of the intelligent vehicle.

[0082] In some embodiments, the CarMaker for Simulink module in CarMaker software is used to construct a virtual lane-changing scenario to simulate intelligent vehicle driving and generate traffic environment. Specifically, a two-lane intelligent vehicle model and the vehicle itself are established in CarMaker. The vehicle travels in one lane, randomly changing its initial speed. Traffic flow is randomly generated in the other lane (i.e., the target lane the vehicle is expected to switch into). The intelligent vehicle perceives the movement of the vehicle and the vehicles in front and behind in the target lane in real time and reacts accordingly, autonomously adjusting its speed or changing lanes. Since the safety of lane-changing decisions mainly depends on whether the vehicle changes lanes, the speed adjustment and lane-changing trajectory are determined by the default following and lane-changing model in CarMaker software. Therefore, this disclosure only focuses on training intelligent lane-changing decisions (0-1 decision on whether to change lanes), that is, issuing a lane-changing command or not to change lanes. The specific lane-changing actions (such as lane-changing speed and lane-changing trajectory) adopted by the intelligent vehicle after receiving the issued lane-changing command are not within the protection scope of this disclosure.

[0083] Furthermore, the parameters of the traffic flow randomly generated within the lane include: the average vehicle speed within the lane (km / h), the traffic flow density within the lane (veh / km), and the center-to-center distance between two adjacent vehicles within the lane (m).

[0084] In some embodiments, see Figure 2 The constructed decision model based on a single-step deep Q-neural network includes an input layer, a hidden layer, and an output layer connected sequentially. The state s is input into the input layer and encoded. Features are extracted from the encoded result through the hidden layer, and the extracted features are decoded through the output layer to obtain the Q-value Q(s) corresponding to state s. Optionally, the input layer contains 5 nodes; the hidden layer consists of two fully connected layers, each containing 50 nodes, with the activation function tanh; the output layer contains 1 node, which corresponds to "lane change" when Q(s) ≥ 0 and "no lane change" when Q(s) < 0, with the activation function purelin. This embodiment employs a neural network based on single-step reinforcement learning, which greatly simplifies the problem, shortens training time, and yields a more robust decision model with the same number of training iterations.

[0085] In some embodiments, see Figure 3 Step 3) specifically includes the following steps:

[0086] 31) Set the loop number to j and initialize j = 1; initialize the network parameters θ of the decision model; construct the Q-table and initialize all Q values ​​in the Q-table to 0; set the maximum number of training iterations to M = 6000; construct the experience replay pool D and initialize it to an empty set; set the minimum number of samples N in the experience replay pool D. start=200, let N = 32 be the number of state reward pairs randomly collected from the experience replay pool D each time; let ε be the probability parameter of the greedy algorithm. j Let its initial and final values ​​be ε. start =0.9 and ε end =0, let ε be the probability decay parameter of the greedy algorithm. decay =200;

[0087] 32) If j < M, then proceed to step 33); if j = M, then the training ends and the lane-changing decision model is obtained.

[0088] 33) Randomly generate state-reward pairs based on a greedy algorithm and store them in the experience replay pool D, specifically including:

[0089] 331) In a virtual lane-changing scenario, the state is randomly initialized, and the lane-changing action of the vehicle is initialized to no lane change; see [link to relevant documentation]. Figure 4 This is a virtual lane-changing scenario constructed according to an embodiment of the present disclosure. The scenario contains two lanes: the current lane where the controlled vehicle (i.e., the self-driving vehicle) is located and the target lane into which the self-driving vehicle is expected to switch. Within the target lane, a traffic flow simulating human driving behavior is randomly generated, and the state s is updated. j Let the expression for the state be:

[0090]

[0091] In the formula, v e and Let the vehicle speed and its normalized threshold be respectively. v l and Let be the speed of the vehicle ahead in the target lane and its normalized threshold, respectively. v f and Let be the speed of the following vehicle in the target lane and its normalized threshold, respectively. d l and Let be the net distance between the vehicle and the vehicle ahead in the target lane, and its normalized threshold. d f and Let be the net distance between the vehicle and the following vehicle in the target lane, and its normalized threshold. v e (unit: m / s), v l (unit: m / s), d l (unit: m), v f (unit: m / s) and d f (Unit: m) Randomly generated;

[0092] 332) Generate random numbers rnd∈(0,1), if rnd≤εj Then, a random number rnd2 ∈ (0, 1) is generated. When rnd2 > 1 / 2, the lane-changing action of the vehicle is selected as lane changing, and step 334) is executed. When rnd2 ≤ 1 / 2, the lane-changing action of the vehicle is selected as not changing lanes, and step 335) is executed. If rnd > ε j Then proceed to step 333);

[0093] 333) Read the state s from the current Q table. j The corresponding Q value Q(s) j If Q(s) j If Q(s) > 0, then the lane-changing action of the vehicle is selected as lane change, and step 334 is executed. j If )≤0, then the lane-changing action of the vehicle is selected as no lane change, and step 332 is executed;

[0094] 334) Enable the vehicle to perform a lane change operation in a virtual lane change scenario and collect the current state s. j and the corresponding reward r j The reward r is set based on whether a collision occurs after the lane change operation. j The expression is as follows:

[0095]

[0096] In the formula, r s As a reward for a successful lane change, r f A reward for failing to change lanes;

[0097] State s j and reward r j State reward pair (s) j r j Store the data in the experience replay pool D and proceed to step 335.

[0098] 335) Determine whether the number of state reward pairs in the experience replay pool D has reached the minimum sample size N. start If the condition is met, proceed to step 34); otherwise, proceed to step 35.

[0099] 34) Randomly sample several state-reward pairs from the experience replay pool as a batch of training samples to train the decision model, specifically including:

[0100] 341) Randomly sample N = 32 state-reward pairs (s) from the experience replay pool D. i r i This constitutes the current training sample set X. i , i∈[1,N];

[0101] 342) Change the current training sample set X i Each state s in iInput the decision model to obtain the corresponding Q value Q(s) i This enables the decision-making model to learn the mapping relationship between state and reward;

[0102] 343) Update parameters

[0103] Update table Q according to the following formula:

[0104] Q(s i )←(1-η)Q(s i )+ηr i

[0105] Where η is the learning rate, which can optionally be η = 0.001;

[0106] The network parameters of the decision model are updated according to the following formula to reduce the loss function of the decision model:

[0107]

[0108] Update the probability parameter ε of the greedy algorithm according to the following formula. j :

[0109]

[0110] Perform step 35);

[0111] 35) Let j = j + 1, then return to step 32).

[0112] Further, determine the maximum number of training iterations M in step 31) according to the following steps:

[0113] 311) Add different traffic flow conditions to the constructed virtual lane-changing scenario.

[0114] Based on traffic flow theory, four traffic flow scenarios—free phase, unobstructed phase, forced phase, and congested phase—are established in CarMaker. These four scenarios realistically reflect different road conditions. The parameters corresponding to the four scenarios are shown in Table 1.

[0115] Table 1 Traffic Flow Condition Parameters

[0116]

[0117] In Table 1, traffic flow speed reflects the average speed of vehicles in the road, and traffic density reflects the number of vehicles per kilometer.

[0118] 312) For each traffic flow condition, conduct several lane-changing tests. Each lane-changing test includes the following steps:

[0119] 3121) At the initial moment of the test, all vehicles are stationary. The center distances between adjacent vehicles in the target lane are shown in Table 1. The distance between the vehicle in the current lane and the vehicle in front is 5m.

[0120] 3122) Start the vehicle in the target lane and control it to reach the target speed, i.e., the traffic flow speed set in Table 1, under the default following and lane-changing model in the CarMaker software. During the entire process, the vehicles in the target lane maintain a constant distance. After a delay period (a random delay time is used for each lane-changing test to ensure that the relative state between the vehicle and the vehicles in the target lane is different in different tests), start the vehicle and control it to reach the target speed, i.e., the traffic flow speed set in Table 1, under the default following and lane-changing model in the CarMaker software.

[0121] 3123) After the autonomous vehicle reaches the speed of the traffic flow, the vehicle in front in the autonomous vehicle lane begins to decelerate to 40% of the traffic flow speed. The autonomous vehicle follows the vehicle in front by reducing its speed according to the randomly set delay time. After the speed of the vehicle in front is lower than 50% of the traffic flow speed, the autonomous vehicle begins to judge the feasibility of changing lanes. After judging that it is possible to change lanes, it executes the lane change action.

[0122] 313) For each traffic flow condition, after every 100 lane change tests, the number of collisions after the vehicle performs the lane change action is counted. The number of lane change tests when the number of collisions converges to close to 0 is taken as the total number of lane change tests for each condition. The maximum value among the total number of lane change tests for all conditions is taken as the maximum number of training tests M.

[0123] According to one embodiment of this disclosure, see Figure 5 When the average total number of cycles (also known as the number of training rounds) under each traffic phase is greater than 3000, it can be guaranteed that the number of collisions per 100 lane-changing decisions converges to close to 0; under extreme conditions (such as the initial state of a congested phase), 6000 cycles can also guarantee that the number of collisions is 0, so M is taken as 6000. According to the method for determining the maximum number of training rounds M provided in the embodiments of this disclosure, the generated lane-changing decisions have high safety even under extreme congestion conditions, and do not require more training time than general methods while strictly avoiding collisions.

[0124] In some embodiments, the lane change success reward r is determined according to the following steps. s And lane change failure reward r f :

[0125] Since the Q-value in single-step reinforcement learning depends on the single-step reward value r s and r fThe expected average frequency of lane changes is considered. Based on literature review, an average lane change success rate of 80% is considered an effective lane change decision model. This embodiment employs a similar logistic regression method, letting p(r) = ... s |s)>-r f / (r s -r f At this time, Q(s) = p(r) s |s)r s +p(r f |s)r f >0, get r f / r s = -5, where p(r) s |s) indicates that a lane change was successful in state s, i.e., a reward r was obtained. s The probability, p(r) f |s) indicates that a lane change failed in state s, i.e., a collision occurred and a reward r was obtained. f The probability of r. Preferably, let r s =1, r f = -5.

[0126] According to the r provided in this disclosure s With r f The method for determining the lane-changing strategy can ensure that the lane-changing strategy obtained by the agent's reinforcement learning converges to the expected lane-changing success rate, while also ensuring that the total number of iterations required for convergence is within a reasonable range.

[0127] To verify the effectiveness of the intelligent vehicle lane-changing decision-making method provided in the first aspect of this disclosure, the relationship between the number of collisions per 100 lane-changing iterations and the number of training cycles is compared between existing multi-step DQN and the single-step DQN used in this embodiment. (See [link to relevant documentation]). Figure 6 It can be seen that under the same number of iterations, the number of collisions in the multi-step DQN method does not converge to 0 significantly asymptotically, and it still cannot guarantee absolute safety under a large number of iterations. In contrast, the single-step DQN method converges to 0 more quickly, indicating that the training method of this embodiment is more efficient for lane-changing decision-making problems and can guarantee lane-changing safety.

[0128] The intelligent vehicle lane-changing decision-making device provided in the second aspect of this disclosure includes:

[0129] The first module is configured to construct a virtual lane-changing scenario using simulation methods. The virtual lane-changing scenario includes lanes, intelligent vehicles, and randomly generated traffic flow. The lanes include at least the lane where the intelligent vehicle is currently located and the target lane.

[0130] The second module is configured to construct a decision model based on a single-step deep Q-neural network to determine the mapping function between the state and the reward. Based on this mapping function, a Q-table is obtained to evaluate the feasibility of lane changing, thereby obtaining the lane changing decision, i.e., "change lanes" or "do not change lanes".

[0131] The third module is configured to induce random lane-changing behavior of the intelligent vehicle based on a greedy algorithm in the constructed virtual lane-changing scenario, record the state and reward of the current lane-changing behavior to form a current state-reward pair, and store the current state-reward pair in the experience replay pool. When the number of state-reward pairs in the experience replay pool reaches the minimum sample size, a number of state-reward pairs are randomly sampled from the experience replay pool as a batch of training samples to train the decision model step by step, and new state-reward pairs generated during the training process are stored in the experience replay pool. The above training process is repeated until the maximum number of training times is reached to obtain the lane-changing decision model. The maximum number of training times is determined according to the following steps: different traffic flow conditions are added to the constructed virtual lane-changing scenario. For each condition, the intelligent vehicle is made to perform lane-changing tests in different relative states with vehicles in the target lane. The number of collisions after the intelligent vehicle performs the lane-changing action is counted. The number of lane-changing tests when the number of collisions converges to close to 0 is taken as the total number of lane-changing tests for each condition. The maximum value among the total number of lane-changing tests for all conditions is taken as the maximum number of training times.

[0132] The fourth module is configured to obtain the state of the actual lane-changing scenario in which the intelligent vehicle is located, and input the state into the lane-changing decision model to obtain the lane-changing decision of the intelligent vehicle.

[0133] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing a computer program thereon, which is executed by a processor to perform the intelligent vehicle lane-changing decision method of the above embodiments.

[0134] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. It should be noted that the electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0135] like Figure 7As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the electronic device. The processing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0136] Typically, the following devices can be connected to I / O interface 105: input devices 106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 109. Communication device 109 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0137] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 109, or installed from a storage device 108, or installed from a ROM 102. When the computer program is executed by the processing device 101, it performs the functions defined in the methods of embodiments of this disclosure.

[0138] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0139] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0140] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: construct a virtual lane-changing scenario using a simulation method, the virtual lane-changing scenario including lanes, intelligent vehicles, and randomly generated traffic flow, wherein the lanes include at least the lane currently occupied by the intelligent vehicle and the target lane; construct a decision model based on a single-step deep Q-neural network to determine the mapping function between state and reward, and obtain a Q-table for evaluating the feasibility of lane changing based on the mapping function, thereby obtaining a lane-changing decision, i.e., "change lanes" or "do not change lanes"; in the constructed virtual lane-changing scenario, induce the intelligent vehicle to perform random lane-changing behavior based on a greedy algorithm, record the state and reward of the current lane-changing behavior to form a current state-reward pair, store the current state-reward pair in an experience replay pool, and when the number of state-reward pairs in the experience replay pool reaches the minimum sample size, retrieve the current state-reward pair from the experience replay pool. Several state-reward pairs are randomly sampled from the pool as training samples to train the decision model step by step. New state-reward pairs generated during training are stored in the experience replay pool. The above training process is repeated until the maximum number of training iterations is reached to obtain the lane-changing decision model. The maximum number of training iterations is determined according to the following steps: Different traffic flow conditions are added to the constructed virtual lane-changing scenario. For each condition, the intelligent vehicle is subjected to lane-changing tests in different relative states with vehicles in the target lane. The number of collisions after the intelligent vehicle performs the lane-changing action is counted. The number of lane-changing tests when the number of collisions converges to close to 0 is taken as the total number of lane-changing tests for each condition. The maximum value among the total number of lane-changing tests for all conditions is taken as the maximum number of training iterations. The state of the intelligent vehicle in the real lane-changing scenario is obtained and the state is input into the lane-changing decision model to obtain the lane-changing decision of the intelligent vehicle.

[0141] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and Python, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0143] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0144] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0145] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning paper or other media, followed by editing, interpreting or otherwise processing as necessary, and then stored in computer memory.

[0146] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0147] Those skilled in the art will understand that all or part of the steps carried in the methods of the above embodiments can be implemented by a program instructing related hardware. The developed program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0149] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for intelligent lane-changing decision of an automobile, characterized in that, include: 1) A virtual lane-changing scenario is constructed using simulation methods. The virtual lane-changing scenario includes lanes, intelligent vehicles, and randomly generated traffic flow. The lanes include at least the lane where the intelligent vehicle is located and the target lane into which the intelligent vehicle is expected to change. 2) Construct a decision model based on a single-step deep Q-neural network to determine the mapping function between the state and the reward. Based on the mapping function, obtain a Q-table to evaluate the feasibility of lane changing for intelligent vehicles, thereby obtaining the lane changing decision, i.e., "change lane" or "do not change lane". 3) In the constructed virtual lane-changing scenario, the intelligent vehicle generates random lane-changing behavior based on a greedy algorithm. The state and reward of the current lane-changing behavior are recorded to form a current state-reward pair. The current state-reward pair is stored in the experience replay pool. When the number of state-reward pairs in the experience replay pool reaches the minimum number of samples, a number of state-reward pairs are randomly sampled from the experience replay pool as a batch of training samples to train the decision model step by step. The new state-reward pairs generated during the training process are stored in the experience replay pool. The above training process is repeated until the maximum number of training times is reached to obtain the lane-changing decision model. 4) Obtain the state of the actual lane-changing scenario in which the intelligent vehicle is located, and input the state into the lane-changing decision model to obtain the lane-changing decision of the intelligent vehicle. 2.The intelligent automobile lane-changing decision method according to claim 1, characterized in that, In step 1), the randomly generated traffic flow parameters include: average vehicle speed per lane, traffic flow density per lane, and center-to-center distance between two adjacent vehicles within the lane. 3.The intelligent automobile lane-changing decision method according to claim 1, characterized in that, The maximum number of training iterations is determined according to the following steps: Different traffic flow conditions are added to the constructed virtual lane-changing scenario. For each condition, the intelligent vehicle is made to perform lane-changing tests in different relative states with vehicles in the target lane. The number of collisions after the intelligent vehicle performs the lane-changing action is counted. The number of lane-changing tests when the number of collisions converges to close to 0 is taken as the total number of lane-changing tests for each condition. The maximum value among the total number of lane-changing tests for all conditions is taken as the maximum number of training iterations. 4.The intelligent automobile lane-changing decision method according to claim 1, characterized in that, Determine the maximum number of training iterations by following these steps: 311) Add different traffic flow conditions to the constructed virtual lane-changing scenario. Based on traffic flow theory, traffic flow conditions of free phase, unobstructed phase, forced phase and congested phase are established in the virtual lane changing scenario, and the operating parameters of each traffic phase are set respectively, including traffic flow speed, traffic density and center distance between two vehicles in the same lane. 312) For each traffic flow condition, conduct several lane-changing tests. Each lane-changing test includes the following steps: 3121) At the initial moment of the test, all vehicles are stationary, the center distance between adjacent vehicles in the target lane is the center distance set in step 311), and the initial value of the distance between the vehicle and the vehicle in front in the current lane is set. 3122) Start the vehicles in the target lane and control the vehicles in the target lane to reach the traffic flow speed set in step 311). Throughout the process, the vehicles in the target lane maintain a constant distance between each other. After a delay, start the vehicle and control the vehicle to reach the traffic flow speed set in step 311). 3123) After the vehicle reaches the traffic flow speed set in step 311), the vehicle in front in the vehicle lane begins to decelerate. When the vehicle in front decelerates to the first speed value, the vehicle also decelerates after a delay and follows the vehicle in front. When the vehicle in front decelerates to the second speed value, the vehicle begins to judge the feasibility of changing lanes. After judging that it is possible to change lanes, it executes the lane changing action. 313) For each traffic flow condition, after a certain number of lane change tests, the number of collisions that occur after the vehicle performs the lane change action in the certain number of lane change tests is counted. The number of lane change tests when the number of collisions converges to close to 0 is taken as the total number of lane change tests for each condition. The maximum value among the total number of lane change tests for all conditions is taken as the maximum number of training tests.

5. The intelligent lane-changing decision method for an intelligent vehicle according to claim 4, wherein, In steps 3122) and 3123), the delay time of the vehicle is randomly set. 6.The intelligent automobile lane-changing decision method according to claim 1, characterized in that, Step 3) specifically includes the following steps: 31) Set the cycle number to j and initialize j = 1; initialize the network parameters θ of the decision model; construct the Q-table and initialize all Q values ​​in the Q-table to 0; set the maximum number of training iterations to M; construct the experience replay pool D and initialize it as an empty set; set the minimum number of samples in the experience replay pool D to N. start Let N be the number of state-reward pairs randomly collected from the experience replay pool D each time; let ε be the probability parameter of the greedy algorithm. j Let its initial and final values ​​be ε. start and ε end Let ε be the probability decay parameter of the greedy algorithm. decay ; 32) If j < M, then proceed to step 33); if j = M, then the training ends and the lane-changing decision model is obtained. 33) Randomly generate state-reward pairs based on a greedy algorithm and store them in the experience replay pool D, specifically including: 331) In the virtual lane-changing scenario, the state is randomly initialized, and the lane-changing action of the ego vehicle is initialized as no lane-changing. The traffic flow simulating human driving behavior is randomly generated in the target lane, and the state s is updated j The expression of the state is: In the formula, v e and These are the vehicle speed and its normalized threshold, respectively; v l and These are the vehicle speed of the vehicle ahead in the target lane and its normalized threshold, respectively; v f and These represent the speed of the following vehicle within the target lane and its normalized threshold, respectively; d l and These are the net distance between the vehicle and the vehicle ahead in the target lane, and their normalized threshold values, respectively; d f and These are the net distance between the vehicle and the following vehicle in the target lane, and the normalized threshold of that net distance, respectively. 332) Generate random numbers rnd∈(0,1), if rnd<ε j Then, a random number rnd2 ∈ (0, 1) is generated. When rnd2 > 1 / 2, the lane-changing action of the vehicle is selected as lane changing, and step 334) is executed. When rnd2 ≤ 1 / 2, the lane-changing action of the vehicle is selected as not changing lanes, and step 335) is executed. If rnd > ε j Then proceed to step 333); 333) Read state s from current Q table j Corresponding Q value Q(s j ), if Q(s j )>0, select the lane-changing action of the ego vehicle as lane-changing, and execute step 334), if Q(s j )≤0, select the lane-changing action of the ego vehicle as no lane-changing, and execute step 332); 334) make the ego vehicle perform a lane-changing operation in a virtual lane-changing scenario, collect the current state s j and the corresponding reward r j , set the reward r j according to whether a collision occurs after the lane-changing operation, the expression is as follows: In the formula, r s is a reward for a successful lane change, r f is a reward for a failed lane change; The state s j and the reward r j are stored as a state-reward pair (s j , r j ) into the experience replay pool D, and step 335) is performed. 335) determining whether the number of state rewards in the experience replay pool D reaches the minimum sample number N start If yes, step 34) is performed, otherwise step 35) is performed. 34) Randomly sample several state-reward pairs from the experience replay pool as a batch of training samples to train the decision model, specifically including: 341) Randomly sample N state-reward pairs (s i , r i ) from the experience replay pool D to form the current training sample set X i , i ∈ [1, N]; 342) obtaining a current training sample set X i , and a state s i in the current training sample set X i ; inputting the state s i into a decision model to obtain a corresponding Q value Q(s i ); and making the decision model learn a mapping relationship between the state and the reward. 343) Update parameters Update table Q according to the following formula: Q(s i )←-(1-η)Q(s i )+ηr i Where η is the learning rate; The network parameters of the decision model are updated according to the following formula to reduce the loss function of the decision model: The probability parameter ε of the greedy algorithm is updated according to the following formula j : Perform step 35); 35) Let j = j + 1, then return to step 32).

7. The intelligent lane-changing decision method for an automobile according to claim 6, wherein, The lane change success reward r is determined according to the following steps s And the lane change failure reward r f : Using a method similar to logistic regression, let p(r) s |s)>-r f / (r s -r f When Q(s) = p(r) is satisfied s |s)r s +p(r f |s)r f When >0, we get r f / r s =-5, let r s =1, r f = -5, where p(r) s |s) indicates that a lane change was successful in state s, i.e., a reward r was obtained. s The probability, p(r) f |s) indicates that a lane change failed in state s, i.e., a collision occurred and a reward r was obtained. f The probability of.

8. An intelligent lane-changing decision device for an automobile, characterized in that, include: The first module is configured to construct a virtual lane-changing scenario using a simulation method. The virtual lane-changing scenario includes lanes, intelligent vehicles, and randomly generated traffic flow. The lanes include at least the lane where the intelligent vehicle is located and the target lane into which the intelligent vehicle is expected to change lanes. The second module is configured to construct a decision model based on a single-step deep Q-neural network to determine the mapping function between the state and the reward, and to obtain a Q-table for evaluating the feasibility of lane changing for intelligent vehicles based on the mapping function, thereby obtaining a lane changing decision, namely "change lane" or "do not change lane". The third module is configured to enable the intelligent vehicle to generate random lane-changing behavior based on a greedy algorithm in the constructed virtual lane-changing scenario, record the state and reward of the current lane-changing behavior to form a current state-reward pair, and store the current state-reward pair in the experience replay pool. When the number of state-reward pairs in the experience replay pool reaches the minimum number of samples, a number of state-reward pairs are randomly sampled from the experience replay pool as a batch of training samples to train the decision model step by step, and new state-reward pairs generated during the training process are stored in the experience replay pool. The above training process is repeated until the maximum number of training times is reached to obtain the lane-changing decision model. The fourth module is configured to obtain the state of the actual lane-changing scenario in which the intelligent vehicle is located, and input the state into the lane-changing decision model to obtain the lane-changing decision of the intelligent vehicle.

9. An electronic device, comprising: include: At least one processor, and a memory communicatively connected to said at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the intelligent lane-changing decision method of the smart vehicle according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to execute the intelligent lane-changing decision method of the smart vehicle according to any one of claims 1-7.

Images