An intelligent lane changing method and system, an electronic device, and a storage medium

Abstract

Embodiments of the present application provide a smart lane changing method and system, an electronic device, and a storage medium. The smart lane changing method comprises: acquiring vehicle driving data and environment information of a vehicle; assigning the vehicle driving data and the environment information of the vehicle to corresponding parameters of a pre-trained reinforcement learning model, and outputting control actions of acceleration and steering angle; correcting the control actions of acceleration and steering angle based on legality rules; and performing smart lane changing by a lane changing vehicle based on the control actions of acceleration and steering angle after correction. The technical solution of the embodiments of the present application can achieve smoother and safer lane changing.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and autonomous driving technology, and more specifically, to an intelligent lane-changing method and system, electronic devices, and storage media. Background Technology

[0002] Autonomous driving is the future development goal, and the current transition from assisted driving to autonomous driving is necessary. Lane changing is a fundamental and critical issue in the field of autonomous driving, and it is a challenging task. Autonomous vehicles need to vigilantly observe the vehicles in front of them in their own lane and the vehicles in the adjacent target lane, and take appropriate lane-changing actions based on the potential adversarial or cooperative responses shown by these related vehicles.

[0003] Chinese patent document CN113928321A discloses a technology entitled "A method and apparatus for lane-changing decision-making based on end-to-end deep reinforcement learning". In this technology, a deep reinforcement learning network is initialized; image information captured by a camera in front of the autonomous vehicle is input into the deep reinforcement learning network to obtain training data; the deep reinforcement learning network is trained based on the training data to obtain a lane-changing decision-making model; the lane-changing decision-making model is used to establish a direct and coherent mapping relationship between image information and lane-changing decisions; and a correct and safe lane-changing decision is made based on the current position of the autonomous vehicle through the lane-changing decision-making model.

[0004] Chinese patent document CN113581182A discloses a technology entitled "A method and system for lane-changing trajectory planning of autonomous vehicles based on reinforcement learning". In this technology, a state space, action space, state transition, reward function and safety correction model are constructed at one time. The model training steps are as follows: after constructing the deflection response model, training data is collected, reinforcement learning models are trained, and then each model is tested accordingly to realize lane-changing trajectory planning in different scenarios.

[0005] The two techniques mentioned above have the following drawbacks: the algorithm input only considers the situation of the vehicle in front and the vehicle in front in the target lane, without considering the situation of the vehicle behind in the target lane, which leads to safety hazards; in addition, the algorithm input only considers the situation of the vehicle at the current moment and the situation of the target lane, without considering that the driving process is a continuous time sequence process, so the vehicle is not smooth when changing lanes. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides an intelligent lane-changing method and system, electronic device, computer-readable storage medium, computer program product, and intelligent vehicle. This enables smoother and safer lane changes for vehicles.

[0007] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0008] According to one aspect of the embodiments of this application, an intelligent lane-changing method is provided, comprising:

[0009] Acquire vehicle driving data and vehicle environment information. The vehicle driving data includes the vehicle's own situation data, and the vehicle's environment information includes the situation data of the vehicle in front ...

[0010] The vehicle driving data and the information about the vehicle's environment are assigned to the corresponding parameters of the pre-trained reinforcement learning model, which outputs control actions such as acceleration and steering angle.

[0011] The control actions of acceleration and steering angle are modified based on legality rules;

[0012] The lane-changing vehicle performs intelligent lane changing based on the corrected acceleration and steering angle control actions.

[0013] In the technical solutions provided in the embodiments of this application, the reinforcement learning model predefines parameters, including: state space, action space, and reward / penalty function.

[0014] In the technical solution provided by the embodiments of this application, the state space is defined as: defining the current status data of the lane-changing vehicle as C. t C t = [Speed ​​of the vehicle changing lanes, steering angle of the vehicle changing lanes, distance of the vehicle changing lanes from the center line of the target lane, speed of the vehicle in front, longitudinal distance between the front of the vehicle changing lanes and the rear of the vehicle in front];

[0015] Define the current information on the vehicles ahead and behind in the target lane as T. t T t = [Speed ​​of the vehicle in front in the target lane, longitudinal distance between the front of the vehicle changing lanes and the rear of the vehicle in front in the target lane, speed of the vehicle behind in the target lane, longitudinal distance between the rear of the vehicle changing lanes and the front of the vehicle behind in the target lane];

[0016] T in the target lane t For time-series information, define 2τ historical states, where τ is the time step and the current state s is the current state. t Defined as:

[0017] s t =[C t ,T t ,T t-τ ,T t-2τ ]

[0018] Among them, C t Including current data on the vehicle changing lanes and data on the vehicle in front, T t ,T t-τ ,T t-2τ It is the temporal status of vehicles in the target lane, extracting two historical experiences.

[0019] In the technical solution provided in the embodiments of this application, the action space is defined as: the acceleration a of the lane-changing vehicle. v The interval range is [-t1m / s] 2 ,t1m / s 2 ], when a v The value of is in the interval [-t1m / s] 2 When [0], the lane-changing vehicle decelerates; a v The value of is in the interval [0, t1m / s] 2 At that time, the lane-changing vehicle accelerates; the rate of change of the steering angle of the lane-changing vehicle, α y The range is [-p° / s, p° / s]. A value in the range [-p° / s, 0] indicates a left turn operation, and a value in the range [0, p° / s] indicates a right turn operation.

[0020] The motion space of the lane-changing vehicle is defined as a continuous motion space, and the speed transformation formula is: V t+τ =V t +a v *τ; The formula for changing the direction angle is: θ t+τ =θ t +a y *τ.

[0021] In the technical solutions provided in the embodiments of this application, the reward and penalty functions include lane change reward and penalty, distance reward and penalty, comfort reward and penalty, and collision reward and penalty;

[0022] The formula for lane-changing rewards and penalties is r. l =-abs[L t -Y], where abs represents taking the absolute value; L t Y represents the lateral coordinate position of the center point of the lane-changing vehicle at time t; Y is the lateral coordinate of the center line of the target lane.

[0023] The distance reward / penalty formula is as follows:

[0024] Where, r d For distance bonus; D is the distance to the vehicle in front; the maximum safe distance is defined as:

[0025]

[0026] Among them, Vt t' is the current vehicle speed; t' is the reaction time; u is the speed of the vehicle in front; d s Basic safety threshold;

[0027] The comfort reward / penalty formula is as follows:

[0028] The collision reward / penalty formula is r t =w l r l +w d r d +w c r c +w f r f ,

[0029] Among them, w l ,w s ,w c ,w f This is the penalty coefficient.

[0030] In the technical solutions provided by the embodiments of this application, the modified control actions for processing the acceleration and steering angle include: speed constraints and steering angle constraints;

[0031] When defining the velocity constraint, if the V output by the reinforcement learning model... t+τ =V t +a v If V in τ t+τ ≤V′ t+τ The correction is made using the following formula:

[0032] a v =(V′) t+τ -V t ) / τ.

[0033] When defining the steering angle constraint, if the reinforcement learning model outputs θ t+τ =θ t +a y If θ in τ t+τ ≥θ′ t+τ The correction is made using the following formula:

[0034] a y =(θ′) t+τ -θ t ) / τ.

[0035] In the technical solutions provided in the embodiments of this application, training the reinforcement learning model includes:

[0036] Initialize the actor and critic networks in the online net, and copy the network parameters of the actor and critic networks in the online net to the actor and critic networks in the target net;

[0037] Actor networks and environments in Online networks t Interaction: Real-world actor gets state s t Get action a t To exert an action on the environment t Environment t Return to the state s of the next moment t+1 and reward r t Using quadruplets (s t ,a t ,r t ,s t+1 This represents the data from the above process, and the quadruple is placed into the experience pool;

[0038] Update the critic network in the Online net: The critic network is based on the acquired (s t ,a t ,r t ,s t+1 s in ) t and a t Obtain reality Q(s) t ,a t Let Q = Q(S,a) and the actor network in the target net obtains (s) value. t ,a t ,r t ,s t+1 s in ) t+1 We obtain action a', and the Critic network in the Target Net is based on the acquired s t+1 And a' to get Q(s) t+1 The objective Q value is Q' = r + gamma × Q(s). t+1 ,a') value;

[0039] Update the actor network in the online net; update the actors based on the update formula to maximize the Q-value output.

[0040] Regularly update the actors in the target net and the critics in the target net;

[0041] The actor network in the online net is cyclically sampled and updated.

[0042] According to one aspect of the embodiments of this application, an intelligent lane-changing system is provided, comprising:

[0043] The signal acquisition module is used to acquire vehicle driving data and vehicle environment information. The vehicle driving data includes the vehicle's own situation data and the vehicle's environment information includes the situation data of the vehicle in front of the vehicle in front of the vehicle and the situation information of the vehicle in front of the target lane on both sides of the target lane.

[0044] The intelligent algorithm module is used to output control actions of acceleration and steering angle based on the pre-trained reinforcement learning model and the assigned parameters, wherein the vehicle driving data and the information of the vehicle's environment are assigned to the parameters;

[0045] The motion correction module is used to correct the control actions that process the acceleration and steering angle.

[0046] The action execution module is used to convert the corrected control actions into operation information, and the lane-changing vehicle performs intelligent lane changing based on the operation information.

[0047] According to one aspect of the embodiments of this application, an electronic device is provided, comprising:

[0048] One or more processors;

[0049] A storage device for storing one or more programs that, when executed by one or more processors, enable the electronic device to implement the intelligent lane-changing method described above.

[0050] According to one aspect of the embodiments of this application, a computer-readable storage medium is also provided, on which computer calibration instructions are stored, which, when executed by a computer's processor, cause the computer to perform the above-described intelligent lane-changing method.

[0051] According to one aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described intelligent lane-changing method.

[0052] According to one aspect of the embodiments of this application, an intelligent vehicle is also provided, including the intelligent lane-changing system described above.

[0053] In some embodiments of this application, the technical solutions involve acquiring vehicle driving data and environmental information. The vehicle driving data includes the vehicle's own information, and the environmental information includes information on the vehicle ahead and the vehicles in the target lanes on either side. The vehicle driving data and environmental information are assigned to corresponding parameters of a pre-trained reinforcement learning model. The pre-trained reinforcement learning model outputs control actions for acceleration and steering angle based on the assigned parameters. The control actions for acceleration and steering angle are then corrected, and the vehicle intelligently changes lanes based on these corrected control actions. Some embodiments primarily target autonomous driving scenarios requiring lane changes. The algorithm's input is data collected by vehicle sensors, and the output is longitudinal acceleration and steering angle change rate, corresponding to changes in vehicle speed and steering wheel direction, respectively. This controls the vehicle to achieve intelligent lane changing while avoiding collision hazards, resulting in smoother and safer lane changes. From a technical perspective, the input to a pre-trained reinforcement learning model includes vehicle driving data and information about the vehicle's environment. The vehicle driving data includes the vehicle's own situation data, while the environmental information includes the situation data of the vehicle in front of it, the situation of the vehicles in the target lanes on both sides, and the situation of the vehicles in the target lanes behind it. Therefore, in some embodiments, lane changing is made safer. Furthermore, from a technical perspective, the reinforcement learning model predefines three parameters: a state space, an action space, and a reward / penalty function. The state space defines multiple parameters, including the speed of the vehicle changing lanes, the steering angle of the vehicle changing lanes, the distance of the vehicle changing lanes from the centerline of the target lane, the speed of the vehicle in front, and the longitudinal distance between the front and rear of the vehicle changing lanes (i.e., the current lane situation of the vehicle changing lanes). The defined parameters also include the speed of the vehicle in the target lane, the longitudinal distance between the front and rear of the vehicle changing lanes, the speed of the vehicle following the vehicle in the target lane, and the longitudinal distance between the rear of the vehicle changing lanes and the front of the vehicle following the vehicle in the target lane (i.e., the target lane situation). Since driving is a continuous process, the temporal information of vehicles in the target lane helps in analyzing their dynamic trajectories. Therefore, some embodiments will use the T-wave data of the target lane. t (The information on the vehicles ahead and behind in the target lane is T) t Transforming this into a time-series information helps extract the continuous changes in vehicles in the target lane, enabling vehicles changing lanes to change lanes more smoothly.

[0054] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0056] Figure 1 This is a flowchart illustrating an exemplary embodiment of the intelligent lane-changing method of this application;

[0057] Figure 2 This is a block diagram of an intelligent lane-changing system illustrated in an exemplary embodiment of this application;

[0058] Figure 3 This is a logic diagram of an intelligent lane-changing system shown in an exemplary embodiment of this application;

[0059] Figure 4 This is a diagram illustrating a deep reinforcement learning framework, as shown in an exemplary embodiment of this application.

[0060] Figure 5 This is an exemplary embodiment illustrating the DDPG algorithm network framework diagram. Detailed Implementation

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they approximate examples of systems and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0063] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the actual situation.

[0064] First, it's important to clarify that reinforcement learning, a branch of machine learning inspired by behavioral psychology, focuses on how agents take different actions in an environment to maximize cumulative rewards. Reinforcement learning primarily consists of an agent, environment, state, action, and reward. After an agent performs an action, the environment transitions to a new state, providing a reward signal (positive or negative). Subsequently, the agent, based on the new state and the reward feedback from the environment, executes a new action according to a specific strategy. This process represents the interaction between the agent and the environment through state, action, and reward. Through reinforcement learning, the agent can understand what actions to take in what state to maximize its reward. Since the interaction with the intelligent sports environment is similar to the interaction between humans and their environment, reinforcement learning can be considered a general learning framework applicable to solving problems in general artificial intelligence. Therefore, reinforcement learning is also known as a machine learning method for general artificial intelligence. Reinforcement learning is a label-free learning method that uses a reward function to determine the merits of performing a certain action in a given state. The learning process involves changing the strategy for performing actions based on reward signals, ultimately forming a strategy that maximizes the reward.

[0065] The DDPG algorithm combines techniques from deterministic policy gradient algorithms and DQN algorithms. DDPG mainly includes experience backtracking, target network, and noise exploration. Experience backtracking: The agent puts the obtained experience data into the ReplayBuffer, and updates network parameters using batch sampling. Target network: In addition to the actor and critic networks, a target actor network and a target critic network are applied to estimate the target. When updating the target network, a soft update method is used to avoid excessively rapid parameter updates. Noise exploration: The large actions output by the deterministic policy are deterministic actions, lacking exploration of the environment. During the training phase, noise is added to the actions output by the actor network, thus giving the agent a certain degree of exploration ability.

[0066] The technical solutions of this application involve technologies such as reinforcement learning, which are specifically illustrated through the following embodiments:

[0067] Figure 1 This is a flowchart illustrating an exemplary embodiment of the intelligent lane-changing method of this application.

[0068] Reference Figure 1As shown, this exemplary embodiment mainly targets autonomous driving scenarios that require lane changing. The pre-trained model uses data collected by vehicle sensors and outputs longitudinal acceleration and steering angle change rate, which correspond to changes in vehicle speed and steering wheel steering, respectively, thereby controlling the vehicle to achieve intelligent lane changing while avoiding collisions. This embodiment mainly uses the Deep Deterministic Policy Gradient Reinforcement Learning (DDPG) algorithm in deep reinforcement learning to allow the vehicle to autonomously explore and learn in a simulation environment to improve its strategy for dealing with intelligent lane changing scenarios.

[0069] In this embodiment, a state space, action space, and reward space are defined in reinforcement learning. The vehicle then interacts with the environment based on actions, continuously explores, and learns autonomously to acquire a strategy capable of handling intelligent lane-changing scenarios. Specifically, the intelligent lane-changing method in this embodiment includes: S01, acquiring vehicle driving data and environmental information. The vehicle driving data includes the vehicle's own situation data, and the environmental information includes the situation data of the vehicle in front of the vehicle and the situation data of the vehicles in the target lanes on both sides; S02, assigning the vehicle driving data and environmental information to the corresponding parameters of the pre-trained reinforcement learning model, outputting control actions of acceleration and steering angle; S03, correcting the control actions of acceleration and steering angle based on legality rules; S04, the vehicle intelligently changes lanes based on the corrected control actions of acceleration and steering angle. Specifically, the first step is to acquire vehicle driving data and environmental information through a signal acquisition module. This data is primarily obtained from sensors installed at the front and rear of the vehicle. In this embodiment, the necessary data includes information about the vehicle in the current lane equipped with the intelligent lane-changing system, the vehicle in front, and the vehicles in the target lane. These acquired signal values ​​are then used as input to a reinforcement learning algorithm. The pre-trained reinforcement learning model outputs control actions for acceleration and steering angle. These actions are then processed through an action correction template to obtain the final control actions for the vehicle to automatically perform intelligent lane-changing.

[0070] In this embodiment, the reinforcement learning model needs to be trained first, and the state space, action space and reward space in reinforcement learning are defined first.

[0071] Figure 4 This is a diagram illustrating a deep reinforcement learning framework, as shown in an exemplary embodiment of this application.

[0072] Reference Figure 4As shown, the reinforcement learning model predefines parameters including the state space, action space, and reward / penalty function. The three key design elements of reinforcement learning are the state space design, action space design, and reward / penalty function design. These three parameters directly determine the problem being solved and the quality of the training results. In this embodiment, the time step is τ. The time to begin merging into another lane in a typical lane change is approximately two to three seconds, so τ can be set to 0.5 seconds. The time step can also be set according to the situation. Let t represent the current moment and t+τ represent the next moment.

[0073] The state space is defined as follows: The current state of the lane-changing train is defined as C. t C t = [Speed ​​of the vehicle changing lanes, steering angle of the vehicle changing lanes, distance of the vehicle changing lanes from the center line of the target lane, speed of the vehicle in front, longitudinal distance between the front of the vehicle changing lanes and the rear of the vehicle in front];

[0074] Define the current information on the vehicles ahead and behind in the target lane as T. t T t = [Speed ​​of the vehicle in front in the target lane, longitudinal distance between the front of the vehicle changing lanes and the rear of the vehicle in front in the target lane, speed of the vehicle behind in the target lane, longitudinal distance between the rear of the vehicle changing lanes and the front of the vehicle behind in the target lane];

[0075] T in the target lane t For time-series information, define 2τ historical states, where τ is the time step and the current state s is the current state. t Defined as:

[0076] s t =[C t ,T t ,T t-τ ,T t-2τ ]

[0077] Among them, C t Including current data on the vehicle changing lanes and data on the vehicle in front, T t ,T t-τ ,T t-2τ It is the temporal status of vehicles in the target lane, extracting two historical experiences.

[0078] The state space is constructed from the information collected by sensors regarding the current lane and the target lane. The collected signals mainly include the speed and steering angle of vehicles changing lanes in the current lane. Other signals include the distance between the current vehicle and the vehicle in front, the speed of the vehicle in front, the speed and longitudinal distance of the vehicle in front in the target lane, and the speed and longitudinal distance of the vehicle behind. The state space is defined as follows: The current lane situation at time t is defined as C. t= [Speed ​​of the lane-changing vehicle, steering angle of the lane-changing vehicle, distance of the lane-changing vehicle from the center line of the target lane, speed of the preceding vehicle, longitudinal distance between the front and rear of the lane-changing vehicle]; Define the state of the target lane at time t as T. t = [Speed ​​of the vehicle in front, longitudinal distance between the front and rear of the vehicle changing lanes, speed of the vehicle behind, longitudinal distance between the rear and front of the vehicle changing lanes]; the above eight values ​​are necessary parameters. Since the vehicle cab is a continuous temporal process, the temporal information of vehicles in the target lane is helpful in analyzing the dynamic trajectory of vehicles. Therefore, this embodiment uses the T-axis of the target lane... t Transforming this into time-series information helps extract continuous changes in vehicle traffic within the target lane. As an optimization, the number of historical states retrieved backwards can be adjusted as a parameter; further optimization can be achieved by combining an LSTM module with a DDPG.

[0079] Therefore, we define a 2τ historical state, with the state s at time t. t The definition is as follows:

[0080] s t =[C t ,T t ,T t-τ ,T t-2τ ]

[0081] The state s at time t t It contains two parts, C t Includes information about vehicles changing lanes and the vehicle in front in the current lane at the current moment, used to extract information about the vehicle in the current lane and the vehicle in front; T t ,T t-τ ,T t-2τ It represents the temporal state of vehicles in the target lane. Two historical experiences were extracted to extract the situation of the vehicles in front and behind in the target lane.

[0082] The motion space is defined as: the acceleration a of the lane-changing vehicle. v The interval range is [-t1m / s] 2 ,t1m / s 2 ], when a v The value of is in the interval [-t1m / s] 2 When [0], the lane-changing vehicle slows down; a v The value of is in the interval [0, t1m / s] 2 When the lane-changing vehicle accelerates, the rate of change of the lane-changing vehicle's steering angle a y The range is [-p° / s, p° / s]. Values ​​in the range [-p° / s, 0] represent left turns, and values ​​in the range [0, p° / s] represent right turns. The motion space of the lane-changing vehicle is defined as a continuous motion space, and the speed transformation formula is: V t+τ =V t +a v*τ; The formula for changing the direction angle is: θ t+τ =θ t +a y *τ. The action space is the output of the reinforcement learning algorithm. For lane-changing scenarios, controlling the vehicle primarily involves learning how to adjust speed and steering angle to reach the destination. Therefore, the action output consists of two-dimensional vehicle control signals: longitudinal vehicle acceleration and the rate of change of the steering angle. During lane changes, longitudinal acceleration mainly controls the vehicle's throttle and brakes, while the rate of change of the steering angle controls the steering wheel's angle. The longitudinal vehicle acceleration a... v The range is [-3m / s] 2 3m / s 2 Specifically, the value is in the range [-3m / s]. 2 [0, 0] represents deceleration operation, with values ​​ranging from [0, 3m / s]. 2 [This refers to acceleration.] For example, when the value is 0, the vehicle speed remains unchanged, maintaining the current speed. Similarly, the steering angle mainly controls the magnitude of lane changes, equivalent to the angle of the steering wheel, with the steering angle change rate 'a'... y The range is [-3° / s, 3° / s]. Values ​​in the range [-3° / s, 0] represent left turns, and values ​​in the range [0, 3° / s] represent right turns. The steering angle should be constrained by speed; the faster the speed, the smaller the steering angle should be for lane changes, and vice versa. Furthermore, this embodiment uses a continuous motion space design, employing the DDPG algorithm for continuous motion spaces. Therefore, motion space a... v and a y The definition is as follows:

[0083] a v ∈[-3m / s 2 3m / s 2 ], a y ∈[-3° / s, 3° / s]

[0084] Therefore, the formula for velocity transformation is: V t+τ =V t +a v *τ; The formula for changing the direction angle is: θ t+τ =θ t +a y *τ. The reward and penalty functions include lane-change rewards and penalties, distance rewards and penalties, comfort rewards and penalties, and collision rewards and penalties;

[0085] The formula for rewarding or penalizing lane changes is r. l =-abs[L t -Y], where abs represents taking the absolute value; L tY represents the lateral coordinate position of the center point of the lane-changing vehicle at time t; Y is the lateral coordinate of the center line of the target lane.

[0086] The distance reward / penalty formula is as follows:

[0087] Where, r d For distance bonus; D is the distance to the vehicle in front; the maximum safe distance is defined as:

[0088]

[0089] Among them, V t t' is the current vehicle speed; t' is the reaction time; u is the speed of the vehicle in front; d s Basic safety threshold;

[0090] The formula for comfort reward and punishment is as follows:

[0091] The collision reward / penalty formula is r t =w l r l +w d r d +w c r c +w f r f ,

[0092] Among them, w l ,w s ,w c ,w f This is the penalty coefficient.

[0093] Reward and punishment function r t As a crucial module guiding reinforcement learning, it directly impacts the learning process. A reward and penalty function is customized for intelligent lane changing scenarios. The reward function consists of the following parts: First, lane-changing reward and penalty, indicating that the closer the center of the changing vehicle is to the center line of the target lane, the better; second, distance reward and penalty, maintaining a safe distance from the vehicle in front throughout the process; third, comfort reward and penalty, as comfort typically depends on the rate of change of longitudinal acceleration and steering angle, and a smooth transition should be maintained as much as possible during lane changing; and fourth, collision penalty, where a collision will result in a severe penalty.

[0094] Lane change reward formula: r l =-abs[L t -Y], where abs represents taking the absolute value, L tThis represents the lateral coordinate position of the center point of the lane-changing vehicle (considering the entire vehicle as the center point of the diagonal of a rectangle) at time t. Y represents the lateral coordinate of the target lane centerline. The farther the vehicle is from the target lane centerline, the smaller this value and the more severe the penalty; the closer the vehicle is to the target lane centerline, the larger this value.

[0095] Distance reward / penalty formula: Where r d For distance-based rewards, D is the distance to the vehicle in front. A reward is given if the distance is greater than the safe distance, and a penalty is imposed if the distance is less than the safe distance. The safe distance is generally related to the speed of the vehicle in front, and a maximum safe distance is defined as follows: Where V t t' is the current vehicle speed, t' is the reaction time, u is the speed of the vehicle in front, and d is the speed of the vehicle ahead. s The basic safety threshold is generally set at 5 meters. It can be seen that the safe distance is directly proportional to speed and inversely proportional to the speed of the vehicle in front. The key point is that when a vehicle changing lanes crosses the center line of the lane, the vehicle in front of it will jump to become the vehicle in front of the target lane.

[0096] Comfort reward / penalty: In order to maintain a smooth transition during lane changes, the changes in speed and steering angle must be smooth. The greater the changes in speed and steering, the worse the vehicle's comfort will be, and the more it will be penalized.

[0097] Collision rewards and penalties: where r c The collision penalty is a binary reward, for example, when no collision occurs, r... c When a collision occurs, r is 0. c It is -100.

[0098] Therefore, the entire reward and punishment function formula is as follows:

[0099] r t =w l r l +w d r d +w c r c +w f r f

[0100] Where w l ,w s ,w c ,w f is the penalty coefficient, used to adjust the proportion of each reward / penalty in the reward function. Where w c r c The reward function needs to be two or three orders of magnitude smaller than other reward and penalty items to ensure safety.

[0101] The control actions for handling acceleration and steering angle have been revised, including: velocity constraints and steering angle constraints; when defining velocity constraints, if the V output by the reinforcement learning model... t+τ =V t +a v If V in τ t+τ ≤V′ t+τ The correction is made using the following formula:

[0102] a v =(V′) t+τ -V t ) / τ.

[0103] When defining the steering angle constraint, if the θ output by the reinforcement learning model... t+τ =θ t +a y If θ in τ t+τ ≥θ′ t+τ Correction is made using a formula:

[0104] a y =(θ′) t+τ -θ t ) / τ.

[0105] The acceleration and steering angle change rate output by the reinforcement learning network directly control the vehicle's speed changes and steering wheel steering. In the early stages of training, many of the output actions of the neural network are illegal. For example, outputting a large steering angle at high speeds could easily lead to a rollover. Outputting excessive acceleration could cause the vehicle to travel too fast, exceeding the speed of the vehicle in front in the target lane, potentially causing a rear-end collision. Such output actions are dangerous and are deemed illegal. This embodiment incorporates a necessary action correction module to forcibly prevent these events from occurring; this module can be customized.

[0106] Speed ​​constraints require that the current vehicle speed be greater than or equal to the speed of the vehicle following in the target lane to ensure safe lane changing without affecting the speed of the following vehicle. The constraint formula is as follows:

[0107] V′ t+τ ≥V behind ;

[0108] The algorithm output V t+τ =V t +a v If V in τ t+τ ≤V′ t+τ Therefore, the algorithm's actions need to be modified, and the modification formula is: a v =(V′) t+τ -V t ) / τ.

[0109] Steering angle constraints establish a relationship between current vehicle speed and steering angle: higher speeds should correspond to smaller steering angles, and lower speeds should correspond to larger steering angles. The constraint formula is as follows: The calculation formula involves V t+τ In this embodiment, the acceleration is corrected first, followed by the steering angle. If the acceleration is corrected, then V... t+τ Replace with V′ t+τ Perform the calculation. The algorithm outputs θ. t+τ =θ t +a y If θ in τ t+τ ≥θ′ t+τ Therefore, the algorithm's actions need to be modified, and the modification formula is: a y =(θ′) t+τ -θ t ) / τ.

[0110] Figure 5 This is an exemplary embodiment illustrating the DDPG algorithm network framework diagram.

[0111] In one embodiment of this application, reference is made to Figure 5 As shown, training the reinforcement learning model includes: initializing the actor and critic networks in the online net; copying the network parameters of the actor and critic networks in the online net to the actor and critic networks in the target net; and configuring the actor network in the online net with the environment. t Interaction: Real-world actor gets state s t Get action a t To exert an action on the environment t Environment t Return to the state s of the next moment t+1 and reward r t Using quadruplets (s t ,a t ,r t ,s t+1 ) represents the data from the above process and puts the quadruplets into the experience pool; updates the critic network in the Online net: the critic network is based on the acquired (s t ,a t ,r t ,s t+1 s in ) t and a t Obtain reality Q(s) t ,a tLet Q = Q(S,a) and the actor network in the target net obtains (s) value. t ,a t ,r t ,s t+1 s in ) t+1 We obtain action a', and the Critic network in the Target Net is based on the acquired s t+1 And a' to get Q(s) t+1 The objective Q value is Q' = r + gamma × Q(s). t+1 , a') value; update the actor network in the online net; update the actor based on the update formula to maximize the Q value output; periodically update the actor in the target net and the critic in the target net; cyclically sample and update the actor network in the online net.

[0112] The network architecture is mainly based on the DPG algorithm architecture. First, the Actor and Critic networks in the Online Net are initialized. Then, the network parameters of the Actor and Critic networks in the Online Net are copied to the Actor and Critic networks in the Target Net, resulting in four networks with the same structure.

[0113] Using Actor networks and environments in Online Net t To interact, that is, to input state s t In reality, the Actor gets the action a t To exert an action on the environment t The environment will return the state to you for the next moment. t+1 and reward r t Therefore, we use the quadruple (s t ,a t ,r t ,s t+1 This quad (s) represents the data for this process. t ,a t ,r t ,s t+1 This indicates that in state s t At that time, take action a t The reward r received t and the next state s t+1 This quadruple is called a transition, and it's placed in the experience pool. Thus, we know that the Actor network interacts with the environment and obtains sample data (s). t ,a t ,r t,s t+1 () function.

[0114] Update the Critic network in Online Net. Retrieve samples (s) from real-world Actors from the experience pool. t ,a t ,r t ,s t+1 ) to train. (s) t ,a t ,r t ,s t+1 s in ) t and a t The input is fed into the Critic network to obtain the actual Q(s) t ,a t Let Q = Q(S,a) as the value of ). Then set (s) t ,a t ,r t ,s t+1 s in ) t+1 Inputting this into the Actor network of the Target Net yields action a'. And then... t+1 Together with a', it is input into the Critic network in the Target Net to obtain Q(s) t+1 Therefore, the target Q value is Q' = r + gamma × Q(s). t+1 The goal is to make the Q-value of the critic output in the Online Net equal to the target Q' value. This becomes a supervised learning problem, where Q' is treated as the label, and we update the actual critic to make the output Q as close as possible to the label Q'.

[0115] Update the Actor network in the Online Net. We then update the Actor network in the Online Net because the action output by the Actor network gives the Q-value in the critic network. We update the Actor according to the DDPG gradient update formula to maximize the Q-value output.

[0116] Update the Actor and Critic in the Target Net. Then, periodically use the parameters of these two networks to update the Actor and Critic in the Target Net.

[0117] Then, the Actor network in Online Net is used for sampling, followed by updates and sampling again. It's a cyclical process. Specific parameters require extensive experimentation for tuning.

[0118] Figure 2This is a block diagram of an intelligent lane-changing system illustrated in an exemplary embodiment of this application.

[0119] Figure 3 This is a logic diagram of an intelligent lane-changing system illustrated in an exemplary embodiment of this application.

[0120] Reference Figure 3 As shown, the first step is to initiate a command that sends a lane-changing request to the neural network. The neural network then outputs action a. v and a y Then, the action is corrected to obtain the corrected action a. v and a y .

[0121] The intelligent lane-changing system includes: a signal acquisition module, used to acquire vehicle driving data and environmental information of the vehicle. The vehicle driving data includes the vehicle's own status data, and the vehicle's environmental information includes the status data of the vehicle in front of the vehicle in front of the vehicle and the status information of the vehicle in front of the target lane on both sides of the target lane.

[0122] The intelligent algorithm module is used by the pre-trained reinforcement learning model to output control actions of acceleration and steering angle based on the assigned parameters. The vehicle driving data and the information of the vehicle's environment are assigned to the parameters.

[0123] The motion correction module is used to correct control actions that process acceleration and steering angle.

[0124] The action execution module is used to convert the corrected control actions into operation information, and the lane-changing vehicle performs intelligent lane changing based on the operation information.

[0125] More specifically, refer to Figure 2 As shown, the intelligent lane-changing system is mainly divided into five modules: signal acquisition module, start command module, intelligent algorithm module, action correction module, and action execution module.

[0126] The signal acquisition module primarily collects information about the current driving situation and the surrounding environment. In lane-changing scenarios, based on the driver's actual lane-changing process, the main information they focus on is obtaining information about vehicles behind and in front of them in the target lane from the rearview mirror. They will then change lanes when the timing is right. Therefore, based on prior knowledge, this embodiment needs to collect information about vehicles changing lanes and vehicles in front in the current lane, as well as vehicles in front and behind in the target lane. These parameters can be obtained through sensors installed on the lane-changing vehicle.

[0127] The activation command module primarily determines whether to initiate a lane change and activates the intelligent lane-changing algorithm. This activation command originates from a driver's request to change lanes, a navigation-guided lane change request, or the autonomous driving path planning system. Upon receiving the activation command, the intelligent lane-changing algorithm begins making intelligent lane-changing decisions based on the current lane's conditions and the target lane's conditions. This step can incorporate prior knowledge; if suitable lane-changing conditions are not met, the algorithm can abandon the lane-changing attempt.

[0128] The intelligent algorithm module is primarily used for deploying algorithms. During the training phase, this module builds a reinforcement learning training algorithm that feeds the necessary information collected by the acquisition module into the neural network for training after real-time processing. Finally, the trained model can be deployed in vehicles that need it for assisted intelligent lane-changing driving.

[0129] The action correction module is primarily used to determine whether the actions output by the algorithm are legal. While any algorithm's output will inevitably have some fluctuations and errors, the accuracy requirements for algorithm output are extremely high in the field of autonomous driving. When an algorithm outputs a poor action value, the resulting danger can be irreversible. Furthermore, since reinforcement learning's training mechanism involves continuous trial and error, incorporating prior rules to constrain action output can ensure more reasonable action outputs and reduce the number of trial and error rounds.

[0130] The motion execution module is primarily used for assisting driving or operating the vehicle. This module converts the output of the motion correction module into operational information, connects it to the vehicle's operating system, and can directly control the vehicle's direction, throttle, and brakes to achieve the effect of assisted driving.

[0131] Embodiments of this application also provide an electronic device, one or more processors, and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the intelligent lane-changing method described above.

[0132] Embodiments of this application also provide a computer-readable storage medium storing computer-readable instructions, which, when executed by a computer's processor, cause the computer to perform the aforementioned intelligent lane-changing method. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, flash memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0133] In the flowcharts and block diagrams in the accompanying drawings, each block may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the intelligent lane-changing method described above. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the intelligent lane-changing method provided in the various embodiments described above.

[0135] Another aspect of this application provides an intelligent vehicle, including the aforementioned intelligent lane-changing system.

[0136] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0137] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0138] It should be understood that the above content is only a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A smart lane-changing method, characterized in that, include: Acquire vehicle driving data and vehicle environment information. The vehicle driving data includes the vehicle's own situation data, and the vehicle's environment information includes the situation data of the vehicle in front ... The vehicle driving data and the information about the vehicle's environment are assigned to the corresponding parameters of the pre-trained reinforcement learning model, which outputs control actions such as acceleration and steering angle. The control actions of acceleration and steering angle are modified based on legality rules; The lane-changing vehicle performs intelligent lane changing based on the corrected acceleration and steering angle control actions; The reinforcement learning model predefines parameters, including a state space and an action space. The state space is defined as: defining the current state data of the lane-changing vehicle as C. t C t =[Speed ​​of the lane-changing vehicle, steering angle of the lane-changing vehicle, distance of the lane-changing vehicle from the center line of the target lane, speed of the vehicle in front, longitudinal distance between the front of the lane-changing vehicle and the rear of the vehicle in front]; Define the current information on the vehicles ahead and behind in the target lane as T. t T t =[Speed ​​of the vehicle in front in the target lane, longitudinal distance between the front of the vehicle changing lanes and the rear of the vehicle in front in the target lane, speed of the vehicle behind in the target lane, longitudinal distance between the rear of the vehicle changing lanes and the front of the vehicle behind in the target lane]; T in the target lane t For timing information, set 2 Historical status Set the time step to 2. Historical state refers to the state s at the current moment. t Settings and These two data points represent the current state s. t Defined as: in, This includes current data on the vehicle changing lanes and data on the vehicle in front. It is the temporal status of vehicles in the target lane. Two historical experiences are extracted to extract the situation of the vehicles in front and behind in the target lane. The motion space is defined as: the acceleration a of the lane-changing vehicle. v The interval range is [-t1m / s] 2 ,t1m / s 2 ], when a v The value of is in the interval [-t1m / s] 2 When [0], the lane-changing vehicle decelerates; a v The value of is in the interval [0, t1m / s] 2 At that time, the lane-changing vehicle accelerates; the rate of change of the steering angle of the lane-changing vehicle, α y The interval range is [-p o / s,p o / s], taking values ​​in the interval [-p o ` / s,0` represents a left turn operation, with values ​​in the range [0, p]. o / s] is a right turn operation; The motion space of the lane-changing vehicle is defined as a continuous motion space, and the speed transformation formula is: , Indicates the velocity at the next moment; The formula for transforming the direction angle is: , Indicates the direction angle at the next moment; The control actions for processing the acceleration and steering angle are corrected, including: speed constraints and steering angle constraints; When defining the velocity constraint, if the reinforcement learning model outputs... If The correction is made using the following formula: ； When defining the steering angle constraint, if the reinforcement learning model outputs... If The correction is made using the following formula: 。 2. The intelligent lane-changing method according to claim 1, characterized in that, The predefined parameters of the reinforcement learning model also include a reward / penalty function.

3. The intelligent lane-changing method according to claim 2, characterized in that, The reward and penalty functions include lane change reward and penalty, distance reward and penalty, comfort reward and penalty, and collision reward and penalty; The formula for lane-changing rewards and penalties is as follows: Where abs represents taking the absolute value; This indicates the lateral coordinate position of the center point of the lane-changing vehicle at time t. The lateral coordinate of the centerline of the target lane; The distance reward / penalty formula is as follows: in, As a distance reward; It is the distance to the vehicle in front; the maximum safe distance is defined as: ， in, This is the current vehicle speed; Reaction time; That is the speed of the vehicle in front; Basic safety threshold; The comfort reward / penalty formula is as follows: ; The collision reward / penalty formula is as follows: , in, The penalty coefficient is... As a collision penalty.

4. The intelligent lane-changing method according to claim 1, characterized in that, Training the reinforcement learning model includes: Initialize the actor and critic networks in the online net, and copy the network parameters of the actor and critic networks in the online net to the actor and critic networks in the target net; In the Online network, the actor network interacts with the environment st: the real actor obtains the state st and gets the action at, applies the action at to the environment, and the environment st returns the state st+1 and reward rt for the next time step. The data of the above process is represented by a quadruple (st, at, rt, st+1) and the quadruple is put into the experience pool. Update the critic network in the Online net: The critic network obtains the actual Q(st,at) value based on st and at in (st,at,rt,st+1), and lets Q=Q(S,a). The actor network in the target net obtains action a' from st+1 in (st,at,rt,st+1). At the same time, the critic network in the Target Net obtains Q(st+1,a') based on st+1 and a'. The target Q value is Q'=r+gamma×Q(st+1,a') value. Update the actor network in the online net; update the actors based on the update formula to maximize the Q-value output. Regularly update the actors in the target net and the critics in the target net; The actor network in the online net is cyclically sampled and updated.

5. An intelligent lane-changing system, characterized in that, include: The signal acquisition module is used to acquire vehicle driving data and vehicle environment information. The vehicle driving data includes the vehicle's own situation data and the vehicle's environment information includes the situation data of the vehicle in front of the vehicle in front of the vehicle and the situation information of the vehicle in front of the target lane on both sides of the target lane. The intelligent algorithm module is used to output control actions of acceleration and steering angle based on the pre-trained reinforcement learning model and the assigned parameters, wherein the vehicle driving data and the information of the vehicle's environment are assigned to the parameters; The motion correction module is used to correct the control actions that process the acceleration and steering angle. The action execution module is used to convert the corrected control actions into operation information, and the lane-changing vehicle performs intelligent lane changing based on the operation information; The reinforcement learning model predefines parameters, including a state space and an action space. The state space is defined as: defining the current state data of the lane-changing vehicle as C. t C t =[Speed ​​of the lane-changing vehicle, steering angle of the lane-changing vehicle, distance of the lane-changing vehicle from the center line of the target lane, speed of the vehicle in front, longitudinal distance between the front of the lane-changing vehicle and the rear of the vehicle in front]; Define the current information on the vehicles ahead and behind in the target lane as T. t T t =[Speed ​​of the vehicle in front in the target lane, longitudinal distance between the front of the vehicle changing lanes and the rear of the vehicle in front in the target lane, speed of the vehicle behind in the target lane, longitudinal distance between the rear of the vehicle changing lanes and the front of the vehicle behind in the target lane]; T in the target lane t For timing information, set 2 Historical status Set the time step to 2. Historical state refers to the state s at the current moment. t Settings and These two data points represent the current state s. t Defined as: in, This includes current data on the vehicle changing lanes and data on the vehicle in front. It is the temporal status of vehicles in the target lane. Two historical experiences are extracted to extract the situation of the vehicles in front and behind in the target lane. The motion space is defined as: the acceleration a of the lane-changing vehicle. v The interval range is [-t1m / s] 2 ,t1m / s 2 ], when a v The value of is in the interval [-t1m / s] 2 When [0], the lane-changing vehicle decelerates; a v The value of is in the interval [0, t1m / s] 2 At that time, the lane-changing vehicle accelerates; the rate of change of the steering angle of the lane-changing vehicle, α y The interval range is [-p o / s,p o / s], taking values ​​in the interval [-p o ` / s,0` represents a left turn operation, with values ​​in the range [0, p]. o / s] is a right turn operation; The motion space of the lane-changing vehicle is defined as a continuous motion space, and the speed transformation formula is: , Indicates the velocity at the next moment; The formula for transforming the direction angle is: , Indicates the direction angle at the next moment; The control actions for processing the acceleration and steering angle are corrected, including: speed constraints and steering angle constraints; When defining the velocity constraint, if the reinforcement learning model outputs... If The correction is made using the following formula: ； When defining the steering angle constraint, if the reinforcement learning model outputs... If The correction is made using the following formula: 。 6. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the intelligent lane-changing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the intelligent lane-changing method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the intelligent lane-changing method as described in any one of claims 1 to 4.

9. A smart car, characterized in that, Including the intelligent lane-changing system as described in claim 5.

Images