Vehicle lane-changing decision method, device and storage medium

Abstract

The present application relates to a vehicle lane change decision method, device and storage medium. The method includes: determining the first state data of the target vehicle and the second state data of the obstacles around the target vehicle, the obstacles around the target vehicle include other vehicles around the target vehicle and virtual obstacles used to represent road information; inputting the first state data and the second state data into a neural network model, and outputting a lane change decision result about the target vehicle, the lane change decision result includes causing the target vehicle to perform any of the lane change actions of changing lanes to the left, changing lanes to the right, and maintaining the current lane. According to the embodiment of the present application, the influence of the road topology can be normalized to normalize the road information and the obstacle information, which is conducive to improving the generalization ability and performance of the model, so that the model can always make accurate and effective lane change decisions in a complex actual road environment, while improving the vehicle's driving efficiency and fully ensuring safety.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a vehicle lane-changing decision method, device, and storage medium. Background Art

[0002] AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Autonomous driving is a mainstream application in the field of artificial intelligence. Autonomous driving technology relies on the collaboration of computer vision, radar, monitoring devices and global positioning systems to enable motor vehicles to achieve autonomous driving without the need for active human operation. Therefore, autonomous driving technology is receiving more and more attention.

[0003] In the field of autonomous driving, the design of lane-changing decisions for autonomous driving vehicles still faces huge challenges. The lateral lane-changing decision of a vehicle refers to the behavior of the vehicle to keep lanes, change lanes left, or change lanes right according to the current environmental status, which is a basic decision-making behavior in the autonomous driving scenario. The current vehicle lane-changing decision methods mainly include rule-based methods and reinforcement learning-based methods. Rule-based decision methods need to rely on a large amount of expert knowledge for manual design. Due to the coupling of a large number of parameters, such methods are difficult to make long-term optimal decisions in different dynamic traffic scenarios. The existing reinforcement learning-based lane-changing decision methods usually have good decision-making effects in simple scenarios, but in real dynamic traffic scenarios, the reinforcement learning lane-changing model is not stable enough and it is difficult to guarantee the optimal. At the same time, the existing reinforcement learning-based lane-changing methods have limited application scenarios, and it is difficult to efficiently solve the lane-changing decision problems of vehicles in dynamic traffic environments and complex road topologies. Therefore, a new type of vehicle lane-changing decision method is urgently needed. Summary of the invention

[0004] In view of this, a vehicle lane changing decision method, device and storage medium are proposed.

[0005] In a first aspect, an embodiment of the present application provides a vehicle lane change decision method. The method comprises:

[0006] Determining first state data of a target vehicle and second state data of obstacles around the target vehicle, wherein the obstacles around the target vehicle include other vehicles around the target vehicle and virtual obstacles for representing road information;

[0007] The first state data and the second state data are input into a neural network model, and a lane changing decision result about a target vehicle is output, wherein the lane changing decision result includes causing the target vehicle to execute any one of lane changing actions, namely, changing lanes to the left, changing lanes to the right, and maintaining the current lane.

[0008] According to the embodiment of the present application, by characterizing the road information with virtual obstacles and inputting the state data of the virtual obstacles into the neural network model, the influence of the road topology can be normalized and taken into account. By inputting the state data of other vehicles around the target vehicle into the model, the influence of other vehicles in a dynamic traffic environment can be taken into account. The method of the present application performs a complete environmental characterization through the second state data, normalizes the road information and the obstacle information, and is conducive to improving the generalization ability of the model. Therefore, by inputting the first state data of the target vehicle and the second state data of the obstacle into the neural network model, the output lane change decision result of the target vehicle is more realistic and more accurate. This method can be applied to vehicle-mounted terminal equipment, so that the lane change decision result can be provided in real time on the vehicle side, and effective lane change decisions can always be made in a complex actual road environment, while improving the driving efficiency of the vehicle, fully ensuring safety.

[0009] According to the first aspect, in a first possible implementation of the vehicle lane change decision method, the road information includes road topology information, lane speed limit information and restricted lane change information, and the virtual obstacle includes a first virtual obstacle set based on the road topology information, a second virtual obstacle set based on the lane speed limit information and a third virtual obstacle set based on the restricted lane change information.

[0010] According to the embodiments of the present application, different virtual obstacles can be set based on different types of road information, so that different road conditions can be flexibly represented by virtual obstacles to achieve different driving restriction purposes. Based on this, virtual obstacles can also be used to accurately represent complex road scenes, which greatly broadens the application scenarios of the model and improves the accuracy of lane change decisions.

[0011] According to a first possible implementation manner of the first aspect, in a second possible implementation manner of the vehicle lane change decision method, the speed of the first virtual obstacle is 0, and it is set at the position where the lane ends or merges; the speed of the second virtual obstacle is the speed limit of the lane with a speed limit, and it is set at a position at a predetermined distance from the target vehicle on the lane with a speed limit; the speed of the third virtual obstacle is the speed of the target vehicle, and it is set at a position parallel to the target vehicle on the lane with restricted lane entry.

[0012] According to the embodiments of the present application, by setting the speed and position of the corresponding virtual obstacles according to different road conditions, the solution can be flexibly applied to different road scenes, and these virtual obstacles and other vehicles around the target vehicle can be represented in a normalized manner. Therefore, when making lane change decisions later, more scientific decision results can be obtained for various road scenes.

[0013] According to the second possible implementation manner of the first aspect, in a third possible implementation manner of the vehicle lane change decision method, the predetermined distance is a maximum distance perceivable by a sensor of the target vehicle or human eyes.

[0014] According to the embodiments of the present application, the actual vehicle driving process can be simulated more realistically, so that the lane speed limit can be more scientifically considered during the automatic driving process to achieve a better lane changing effect.

[0015] According to the first aspect or the first or second or third possible implementation of the first aspect, in a fourth possible implementation of the vehicle lane change decision method, the obstacles around the target vehicle include obstacles around the target vehicle in the lane where the target vehicle is located, the left lane of the lane where the target vehicle is located, and the right lane.

[0016] According to an embodiment of the present application, by selecting obstacles in the lane where the target vehicle is located, the left lane of the lane, and the right lane, the obstacles around the target vehicle can be considered efficiently and as comprehensively as possible in the process of making lane changing decisions, so as to obtain a more accurate lane changing decision result.

[0017] According to the first aspect or the first or second or third or fourth possible implementation of the first aspect, in a fifth possible implementation of the vehicle lane change decision method, the second state data includes third state data of obstacles in the lane where the target vehicle is located and in the left lane of the lane where the target vehicle is located, and fourth state data of obstacles in the lane where the target vehicle is located and in the right lane of the lane where the target vehicle is located, the first state data and the second state data are input into the neural network model, and the lane change decision result about the target vehicle is output, including:

[0018] The neural network model determines a first set of evaluation values ​​based on the first state data and the third state data, the first set of evaluation values ​​including an evaluation value of an action of changing lanes to the left and an evaluation value of an action of maintaining a current lane;

[0019] The neural network model determines a second set of evaluation values ​​based on the first state data and the fourth state data, the second set of evaluation values ​​including an evaluation value of the action of changing lanes to the right and an evaluation value of the action of maintaining the current lane;

[0020] The neural network model outputs a lane-changing decision result based on the first set of evaluation values ​​and the second set of evaluation values.

[0021] According to an embodiment of the present application, by dividing the second state data into third state data of obstacles in the lane where the target vehicle is located and in the lane to the left of the target vehicle's lane, and fourth state data of obstacles in the lane where the target vehicle is located and in the lane to the right of the target vehicle's lane, and determining two corresponding groups of evaluation values ​​based on the first state data and the third state data, and the first state data and the fourth state data, respectively, it is possible to symmetrically evaluate the actions of changing lanes to the left and changing lanes to the right. This construction method of a symmetrical input shared parameter network can significantly reduce the overall scale of the neural network model, remove redundant structures in the model, and optimize the decision-making process of the model.

[0022] According to a fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the vehicle lane change decision method, the neural network model determines a first set of evaluation values ​​based on the first state data and the third state data, including:

[0023] The neural network model extracts features from the third state data to determine a first feature vector;

[0024] The neural network model determines a first set of evaluation values ​​based on the first state data and the first eigenvector;

[0025] The neural network model determines a second set of evaluation values ​​based on the first state data and the fourth state data, including:

[0026] The neural network model performs feature extraction on the fourth state data to determine a second feature vector;

[0027] The neural network model determines a second set of evaluation values ​​based on the first state data and the second eigenvector.

[0028] According to the embodiment of the present application, by processing the vehicle status data and the obstacle status data separately and only performing feature extraction on the obstacle status data, the lane changing behavior can be better evaluated while reducing the amount of calculation, thereby improving the performance of the model.

[0029] According to the first aspect or the first or second or third or fourth or fifth or sixth possible implementation of the first aspect, in a seventh possible implementation of the vehicle lane change decision method, the neural network model is a trained neural network model, and the method further includes:

[0030] Based on the data generated by the vehicle during the lane changing process, a training data set is determined, where the training data set includes state data at each moment in the lane changing process, a lane changing action, a reward value of the lane changing process, and state data at the next moment of the moment, where the state data includes the state data of the vehicle and the state data of obstacles around the vehicle;

[0031] The initial neural network model is trained using the training data set to obtain a trained neural network model.

[0032] According to an embodiment of the present application, by utilizing the data generated during the lane changing process to determine the training data set, a large amount of vehicle straight-moving data that is not helpful for training is removed. During the training process, training can be performed based on higher-quality valid data, thereby improving the training efficiency of the model and making the performance of the trained model better.

[0033] According to the seventh possible implementation of the first aspect, in an eighth possible implementation of the vehicle lane changing decision method, the lane changing process includes the vehicle lane changing process when the vehicle's lane changing strategy is successfully executed and the lane keeping process when the vehicle's lane changing strategy is not executed.

[0034] According to the embodiments of the present application, training data can be collected in a more targeted manner, so that the collected data is of higher quality and various lane change situations are taken into account, thereby improving training efficiency and obtaining better training results.

[0035] According to the seventh or eighth possible implementation of the first aspect, in a ninth possible implementation of the vehicle lane changing decision method, when the vehicle's lane changing strategy is successfully executed, the reward value of the lane changing process is determined based on the distance and speed difference between the vehicle and the vehicle in front before the lane changing, and the distance and speed difference between the vehicle and the vehicle in front after the lane changing; when the vehicle's lane changing strategy is not executed, the reward value of the lane changing process is a negative value.

[0036] According to the embodiment of the present application, by designing a calculation method for the reward value using the distance and speed difference at the beginning and end of the lane change, the reward value can be made into a posteriori value, reducing the amount of calculation, and can better evaluate the macro-benefits of the lane changing decision made by the vehicle in a complex dynamic traffic environment, reducing the impact of the micro-dynamic changes of the real-time traffic scene on the lane changing decision, so that the model can make effective and stable decisions in real dynamic scenes, thereby improving the performance of the model.

[0037] According to the first aspect or the first or second or third or fourth or fifth or sixth seventh or eighth or ninth possible implementation of the first aspect, in a tenth possible implementation of the vehicle lane changing decision method, the state data includes any one or more of position, speed, and acceleration.

[0038] According to the embodiment of the present application, data of different dimensions can be taken into consideration for lane changing decisions, making the decision results more accurate.

[0039] In a second aspect, an embodiment of the present application provides a vehicle lane change decision device. The device comprises:

[0040] A first determination module is used to determine first state data of a target vehicle and second state data of obstacles around the target vehicle, where the obstacles around the target vehicle include other vehicles around the target vehicle and virtual obstacles used to represent road information;

[0041] The output module is used to input the first state data and the second state data into the neural network model, and output the lane changing decision result about the target vehicle, and the lane changing decision result includes making the target vehicle execute any one of the lane changing actions of changing lanes to the left, changing lanes to the right, and maintaining the current lane.

[0042] According to the second aspect, in a first possible implementation of the vehicle lane change decision device, the road information includes road topology information, lane speed limit information and restricted lane change information, and the virtual obstacle includes a first virtual obstacle set based on the road topology information, a second virtual obstacle set based on the lane speed limit information and a third virtual obstacle set based on the restricted lane change information.

[0043] According to a first possible implementation manner of the second aspect, in a second possible implementation manner of the vehicle lane changing decision device, the speed of the first virtual obstacle is 0, and it is set at the position where the lane ends or merges; the speed of the second virtual obstacle is the speed limit of the lane with a speed limit, and it is set at a position at a predetermined distance from the target vehicle on the lane with a speed limit; the speed of the third virtual obstacle is the speed of the target vehicle, and it is set at a position parallel to the target vehicle on the lane with restricted entry.

[0044] According to a second possible implementation manner of the second aspect, in a third possible implementation manner of the vehicle lane change decision device, the predetermined distance is a maximum distance perceivable by a sensor of the target vehicle or human eyes.

[0045] According to the second aspect or the first or second or third possible implementation of the second aspect, in a fourth possible implementation of the vehicle lane change decision device, the obstacles around the target vehicle include obstacles around the target vehicle in the lane where the target vehicle is located, the left lane of the lane where the target vehicle is located, and the right lane.

[0046] According to the second aspect or the first or second or third or fourth possible implementation of the second aspect, in a fifth possible implementation of the vehicle lane change decision device, the second state data includes third state data of obstacles in the lane where the target vehicle is located and in the left lane of the lane where the target vehicle is located, and fourth state data of obstacles in the lane where the target vehicle is located and in the right lane of the lane where the target vehicle is located, and the output module is used to:

[0047] The neural network model determines a first set of evaluation values ​​based on the first state data and the third state data, the first set of evaluation values ​​including an evaluation value of an action of changing lanes to the left and an evaluation value of an action of maintaining a current lane;

[0048] The neural network model determines a second set of evaluation values ​​based on the first state data and the fourth state data, the second set of evaluation values ​​including an evaluation value of the action of changing lanes to the right and an evaluation value of the action of maintaining the current lane;

[0049] The neural network model outputs a lane-changing decision result based on the first set of evaluation values ​​and the second set of evaluation values.

[0050] According to a fifth possible implementation manner of the second aspect, in a sixth possible implementation manner of the vehicle lane change decision device, the neural network model determines a first set of evaluation values ​​based on the first state data and the third state data, including:

[0051] The neural network model extracts features from the third state data to determine a first feature vector;

[0052] The neural network model determines a first set of evaluation values ​​based on the first state data and the first eigenvector;

[0053] The neural network model determines a second set of evaluation values ​​based on the first state data and the fourth state data, including:

[0054] The neural network model performs feature extraction on the fourth state data to determine a second feature vector;

[0055] The neural network model determines a second set of evaluation values ​​based on the first state data and the second eigenvector.

[0056] According to the second aspect or the first or second or third or fourth or fifth or sixth possible implementation of the second aspect, in a seventh possible implementation of the vehicle lane change decision device, the neural network model is a trained neural network model, and the device further includes:

[0057] a second determination module, for determining a training data set based on data generated by the vehicle during a lane changing process, the training data set including state data at each moment in the lane changing process, a lane changing action, a reward value of the lane changing process, and state data at a next moment of the moment, the state data including state data of the vehicle and state data of obstacles around the vehicle;

[0058] The third determination module is used to train the initial neural network model using the training data set to obtain a trained neural network model.

[0059] According to the seventh possible implementation of the second aspect, in an eighth possible implementation of the vehicle lane changing decision device, the lane changing process includes a vehicle lane changing process when the vehicle's lane changing strategy is successfully executed and a lane keeping process when the vehicle's lane changing strategy is not executed.

[0060] According to the seventh or eighth possible implementation method of the second aspect, in a ninth possible implementation method of the vehicle lane changing decision device, when the vehicle's lane changing strategy is successfully executed, the reward value of the lane changing process is determined based on the distance and speed difference between the vehicle and the vehicle in front before the lane changing, and the distance and speed difference between the vehicle and the vehicle in front after the lane changing; when the vehicle's lane changing strategy is not executed, the reward value of the lane changing process is a negative value.

[0061] According to the second aspect or the first or second or third or fourth or fifth or sixth seventh or eighth or ninth possible implementation of the second aspect, in a tenth possible implementation of the vehicle lane changing decision device, the status data includes any one or more of position, speed, and acceleration.

[0062] In a third aspect, an embodiment of the present application provides a vehicle lane changing decision device, comprising: a processor and a memory; the memory is used to store programs; the processor is used to execute the programs stored in the memory, so that the device implements the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect.

[0063] In a fourth aspect, an embodiment of the present application provides a terminal device, which can execute the vehicle lane changing decision method of the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect.

[0064] In the fifth aspect, an embodiment of the present application provides a computer-readable storage medium having program instructions stored thereon, characterized in that when the program instructions are executed by a computer, the computer implements the vehicle lane changing decision method of the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect.

[0065] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes program instructions. When the program instructions are executed by a computer, the computer implements the vehicle lane changing decision method of the above-mentioned first aspect or one or more of the multiple possible implementation methods of the first aspect.

[0066] These and other aspects of the present application will become more apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1 A schematic diagram showing a system architecture according to an embodiment of the present application.

[0069] Figure 2 A schematic diagram showing an application scenario according to an embodiment of the present application.

[0070] Figure 3 A flow chart of a vehicle lane changing decision method according to an embodiment of the present application is shown.

[0071] Figure 4 (a) is a schematic diagram showing a first obstacle according to an embodiment of the present application.

[0072] Figure 4 (b) shows a schematic diagram of a second obstacle according to an embodiment of the present application.

[0073] Figure 4 (c) shows a schematic diagram of a third obstacle according to an embodiment of the present application.

[0074] Figure 5 A flow chart of a vehicle lane changing decision method according to an embodiment of the present application is shown.

[0075] Figure 6 A schematic diagram showing the structure of a neural network model according to an embodiment of the present application.

[0076] Figure 7 A flow chart of a vehicle lane changing decision method according to an embodiment of the present application is shown.

[0077] Figure 8 (a) is a schematic diagram showing a vehicle lane changing process according to an embodiment of the present application.

[0078] Figure 8 (b) is a schematic diagram showing a vehicle lane changing process according to an embodiment of the present application.

[0079] Fig. 9 A schematic diagram showing a method for determining a reward value according to an embodiment of the present application.

[0080] Fig.10 A structural diagram of a vehicle lane-changing decision device according to an embodiment of the present application is shown.

[0081] Fig.11 A structural diagram of an electronic device 1100 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0082] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0083] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0084] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0085] AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Autonomous driving is a mainstream application in the field of artificial intelligence. Autonomous driving technology relies on the collaboration of computer vision, radar, monitoring devices and global positioning systems to enable motor vehicles to achieve autonomous driving without human active operation. Therefore, autonomous driving technology has received more and more attention. In the field of autonomous driving, the lane change decision design of autonomous driving vehicles still faces huge challenges. The lateral lane change decision of a vehicle refers to the behavior of the vehicle to keep lanes, change lanes left or right according to the current environmental state, which is a basic decision-making behavior in the autonomous driving scenario. The current vehicle lane change decision methods mainly include rule-based methods and reinforcement learning-based methods. Rule-based decision methods need to rely on a large amount of expert knowledge for manual design. Due to the coupling of a large number of parameters, such methods are difficult to make long-term optimal decisions in different dynamic traffic scenarios. The existing lane change decision methods based on reinforcement learning usually have good decision-making effects in simple scenarios, but in real dynamic traffic scenarios, the reinforcement learning lane change model is not stable enough and it is difficult to guarantee the optimal. At the same time, the existing lane-changing methods based on reinforcement learning have limited application scenarios and are difficult to efficiently solve the lane-changing decision problem of vehicles in dynamic traffic environments and complex road topologies. Therefore, a new vehicle lane-changing decision method is urgently needed.

[0086] In order to solve the above technical problems, the present application provides a vehicle lane change decision method. The vehicle lane change decision method of the embodiment of the present application uses virtual obstacles to represent road information, and inputs the state data of the virtual obstacles into the neural network model, which can normalize the influence of road topology. By inputting the state data of other vehicles around the target vehicle into the model, the influence of other vehicles in a dynamic traffic environment can be considered. The method of the present application uses virtual obstacles and the state data of other vehicles to perform a complete environmental representation, normalizes the road information and the obstacle information, and is conducive to improving the generalization ability of the model. Therefore, the model can be used to output the lane change decision result of the target vehicle based on the state data of the target vehicle and the state data of the obstacle. The result is more realistic and more accurate. The method can be applied to vehicle-mounted terminal equipment, so that the lane change decision result can be provided in real time on the vehicle side, and effective lane change decisions can always be made in a complex actual road environment, while improving the driving efficiency of the vehicle, fully ensuring safety.

[0087] Figure 1 A schematic diagram of a system architecture according to an embodiment of the present application is shown. The vehicle lane change decision system of the embodiment of the present application can be deployed in an automatic driving system of a vehicle, such as Figure 1 As shown in the figure, the vehicle's automatic driving system may include a decision-making planning layer, a perception layer, and a control layer. The perception layer can be used to perceive the environment and the vehicle's own state and collect relevant information; the decision-making planning layer can be used to make decisions and judgments by obtaining information from the perception layer, formulate corresponding control strategies, and make driving decisions instead of humans; the control layer can be used to control the vehicle instead of humans after the decision-making planning layer makes a decision, and feed back the control command to the underlying module to perform basic actions such as speed adjustment, distance keeping, lane changing, overtaking, etc.

[0088] Among them, the decision planning layer may include a target navigation system, a behavior decision system and a motion planning system. The target navigation system can be used to determine the target trajectory to the destination during the automatic driving process. The behavior decision system can be used to make decisions on the behavior of the vehicle based on the information obtained from the perception layer. The vehicle lane change decision system of the embodiment of the present application can be deployed in the behavior decision system. In the scenario of the lane change decision in the automatic driving of the embodiment of the present application, the vehicle lane change decision system can be used to determine the lane change decision result based on the state data of the vehicle and the state data of the obstacles around the vehicle. The motion planning system can plan the motion trajectory from the current lane to the target lane (for example, when changing lanes to the left, the target lane is the left lane of the current lane) based on the lane change decision result, so as to be used for the control layer to perform corresponding lane change actions, such as changing lanes to the left, changing lanes to the right, and keeping the current lane.

[0089] Figure 2A schematic diagram of an application scenario according to an embodiment of the present application is shown. The vehicle lane change decision system of the embodiment of the present application can be used in a scenario of vehicle automatic driving, such as Figure 2 As shown, when an autonomous vehicle (i.e., the ego, such as the ego in the figure) is driving on the road, there may be road scenes such as lane merging, splitting, solid lines, and different speed limits in each lane within the driving range. As shown in the figure, the road dividing line may include a dotted line indicating a changeable lane and a solid line indicating an immutable lane, and the speed limit of each lane may be different. For example, the maximum speed limits of the lanes from left to right in the figure are 120km / h, 100km / h, 100km / h, and 80km / h, respectively. At the same time, there may be an indefinite number of other vehicles within the driving range of the ego, and these vehicles may make autonomous lane changes at any time. For example, the vehicle in the lane to the right of the ego may change lanes to the left at any time to overtake the ego. In this scenario, the car computer on the ego can use the method of the embodiment of the present application to represent various road information such as lane merging, splitting, solid lines, and different speed limits in each lane as virtual obstacles. In different roads and traffic scenarios, by obtaining the state data of the vehicle and the state information of surrounding obstacles (including other vehicles around the vehicle and the above-mentioned virtual obstacles), it inputs them into the neural network model to output the corresponding lane changing decision, so as to control the ego to perform effective lane changing actions.

[0090] The vehicle lane changing decision system of the embodiment of the present application can be deployed on the vehicle's computer, for example, it can be implemented by an on-board computing unit, and the computing power of the on-board computing unit can, for example, ensure an update frequency of 10 Hz for the algorithm.

[0091] The training process of the neural network model can also be implemented on a cloud server. After the trained neural network model is obtained, the trained model is deployed to the vehicle side. The cloud server can be a physical device or a virtual device, such as a virtual machine, a container, etc., with a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function, and the wireless connection function refers to the ability to connect to other servers or vehicle sides through wireless connection methods such as Wi-Fi and Bluetooth. The server of the present application may also have the function of communicating through a wired connection. For example, the server of the present application can be located in the cloud, communicate with the vehicle side, and send the trained neural network model to the vehicle side.

[0092] It should be noted that the method of the embodiment of the present application can also be used in other scenarios of longitudinal or lateral control of the vehicle in vehicle automatic driving or assisted driving, and the present application does not limit this.

[0093] The following Figure 3-Figure 9, the vehicle lane changing decision method of the embodiment of the present application is introduced in detail. The vehicle lane changing decision method may include a training phase and a reasoning phase. Figure 3-Figure 6 , the reasoning stage of the vehicle lane changing decision method of the embodiment of the present application is introduced.

[0094] Figure 3 FIG. 1 is a flow chart showing a vehicle lane change decision method according to an embodiment of the present application. The method can be used in a vehicle terminal device, such as Figure 3 As shown, the method includes:

[0095] Step S301, determining first state data of a target vehicle and second state data of obstacles around the target vehicle.

[0096] The target vehicle may be an autonomous vehicle, such as the ego vehicle mentioned above.

[0097] The obstacles around the target vehicle may include other vehicles around the target vehicle and virtual obstacles for representing road information. Other vehicles around the target vehicle may include vehicles around the target vehicle in the target lane, the left lane of the target vehicle lane, and the right lane.

[0098] The first state data and the second state data may include any one or more of position, speed, and acceleration. For example, the position in the first state data may be the position of the target vehicle in the world coordinate system, and the position in the second state data may represent the position of the obstacle in the world coordinate system, or the relative position between the obstacle and the target vehicle. The speed and acceleration in the state data may represent the longitudinal speed and acceleration.

[0099] In this way, data from different dimensions can be taken into account when making lane-changing decisions, making the decision results more accurate.

[0100] Considering various special road environments during vehicle driving, the present application represents the road information during vehicle driving as a virtual obstacle, which can be a virtual vehicle, so that the road information and other vehicles can be represented in a normalized manner, so that they can be processed as obstacle input models together, thereby simplifying the structure of the model and effectively representing various complex road scenes. See below.

[0101] Optionally, the road information may include road topology information, lane speed limit information, and lane change restriction information.

[0102] Among them, road topology information can represent the connection relationship between roads, for example, it can indicate the end position of the lane, the position where the lane merges, etc.; lane speed limit information can include the maximum speed limit of the lane and the minimum speed limit of the lane; lane change restriction information can be determined by road traffic markings, for example, it can include lane dividing line information, lane road surface marking information, etc., wherein the lane dividing line information may include dotted lines indicating changeable lanes, solid lines indicating non-changeable lanes, and lane lines indicating changeable lanes within a predetermined time period (such as tidal lane lines); lane road surface marking information may include lane restriction or prohibited entry indication information, such as bus lane markings, etc.

[0103] Due to the different natures of the above three types of road information, different virtual obstacles can be set for these three types of road information respectively. On this basis, the virtual obstacles can include a first virtual obstacle set based on road topology information, a second virtual obstacle set based on lane speed limit information, and a third virtual obstacle set based on lane change restriction information.

[0104] According to the embodiments of the present application, different virtual obstacles can be set based on different types of road information, so that different road conditions can be flexibly represented by virtual obstacles to achieve different driving restriction purposes. Based on this, virtual obstacles can also be used to accurately represent complex road scenes, which greatly broadens the application scenarios of the model and improves the accuracy of lane change decisions.

[0105] When setting virtual obstacles, optionally, for the road topology information, since the lane starting position or fork position does not need to be restricted, the first obstacle can be set at the lane end position or merging position based on the road topology information to restrict the target vehicle from exceeding the road topology boundary. The merging position of the lane can be found in Figure 2 As shown in FIG. 1 , when two lanes are about to merge, the merge position may be the end position of the original lane to be merged into the other lane. The speed of the first virtual obstacle may be 0, and the first virtual obstacle may be set at the end or merge position of the lane.

[0106] See also Figure 4 (a) is a schematic diagram showing a first obstacle according to an embodiment of the present application. As shown in the figure, the left lane road of the lane where the target vehicle (ego) is located ends, and one (or more) stationary virtual cars can be set at the end position of the lane, as shown in the dotted box in the figure, and the length and width of the virtual car can be set as needed.

[0107] In response to the lane speed limit information, a dynamic virtual obstacle can be set on the corresponding lane to restrict the target vehicle from traveling within the speed limit. The speed of the second virtual obstacle can be the speed limit of the lane with the speed limit, and it is set at a predetermined distance from the target vehicle on the lane with the speed limit. When the speed limit of the lane is the maximum speed limit of the lane, the second virtual obstacle can be set in front of the target vehicle on the lane with the speed limit; when the speed limit of the lane is the minimum speed limit of the lane, the second virtual obstacle can also be set behind the target vehicle on the lane with the speed limit.

[0108] The predetermined distance may be related to a distance perceivable by a sensor or human eye. Optionally, the predetermined distance may be a maximum distance perceivable by a sensor or human eye of the target vehicle (e.g., 150 m). Thus, the actual vehicle driving process may be simulated more realistically, so that the lane speed limit may be more scientifically considered during the autonomous driving process, thereby achieving a better lane changing effect.

[0109] See also Figure 4 (b) shows a schematic diagram of a second obstacle according to an embodiment of the present application. As shown in the figure, the maximum speed limits of the three lanes are 120km / h, 100km / h and 80km / h respectively. Therefore, one (or more) virtual cars with speeds consistent with the speed limits can be set at positions at a predetermined distance (such as 150m) from the target vehicle (ego) on the three lanes. As shown in the dotted box in the figure, a virtual car with a speed of 120km / h can be set on a lane with a speed limit of 120km / h, a virtual car with a speed of 100km / h can be set on a lane with a speed limit of 100km / h, and a virtual car with a speed of 80km / h can be set on a lane with a speed limit of 80km / h. The length and width of the above virtual cars can be set as needed.

[0110] For restricted lane change information, for example, when a lane is solid, it indicates that the vehicle is restricted from changing into the adjacent lane on that side, or, for example, when the adjacent lane is indicated as a tidal lane, it indicates that the vehicle is restricted from changing into the tidal lane within a predetermined time period. Therefore, a virtual obstacle with the same speed as the target vehicle can be placed on the restricted lane to replace the restricted lane change information, so as to achieve the effect of restricting the vehicle from changing lanes. Among them, the speed of the third virtual obstacle can be the speed of the target vehicle, and it is set at a position parallel to the target vehicle on the restricted lane. For example, one or more virtual obstacles with the same speed as the target vehicle can be set at a position parallel to the target vehicle.

[0111] See also Figure 4(c) is a schematic diagram of a third obstacle according to an embodiment of the present application. As shown in the figure, the lane separators between the lane where the target vehicle (ego) is located and other lanes are all solid lines, which can indicate that the target vehicle cannot change lanes to the left or right. Since the speed of the target vehicle is 60km / h at this time, two virtual vehicles with the same speed as the target vehicle can be added to the left and right lanes of the target vehicle's lane parallel to the target vehicle. The length and width of the virtual vehicles can be set as needed.

[0112] According to the embodiments of the present application, by setting the speed and position of the corresponding virtual obstacles according to different road conditions, the solution can be flexibly applied to different road scenes, and these virtual obstacles and other vehicles around the target vehicle can be represented in a normalized manner. Therefore, when making lane change decisions later, more scientific decision results can be obtained for various road scenes.

[0113] The above three virtual obstacles are only used as an example. In fact, virtual obstacles may include more or fewer types and may be set according to the needs of different road environment types. The number of virtual obstacles to be set may also be determined as needed, and this application does not impose any restrictions on this.

[0114] After representing the road information as virtual obstacles and setting up the virtual obstacles on each lane, the obstacles around the target vehicle (including other vehicles around the target vehicle and virtual obstacles) can be sorted from near to far based on the distance to the target vehicle on each lane, and then a predetermined number of obstacles are selected on each lane to use the state data of these obstacles as the second state data to be input into the neural network model.

[0115] Optionally, the obstacles around the target vehicle may include obstacles around the target vehicle in the lane where the target vehicle is located, the left lane of the lane where the target vehicle is located, and the right lane.

[0116] For example, one or more obstacles may be selected in the lane where the target vehicle is located, the left lane of the lane where the target vehicle is located, and the right lane, for example, two obstacles in front of the target vehicle and one obstacle behind the target vehicle, that is, three obstacles may be selected in each lane, for a total of nine obstacles. Their state data may be obtained as the second state data, and the second state data of the nine obstacles and the first state data of the target vehicle may be combined into a vector as the data to be input into the neural network model.

[0117] If there are not enough obstacles on the lane, virtual obstacles can be added to fill the gaps in the lane to ensure the stability of the model input dimension and avoid adverse effects on the model's decision-making process. For example, if three obstacles are selected on each lane, if there is only one obstacle on a lane, a virtual obstacle with a speed equal to the maximum speed limit of the road can be set at the front of the lane, or a stationary virtual obstacle can be set at the back of the lane.

[0118] According to an embodiment of the present application, by selecting obstacles in the lane where the target vehicle is located, the left lane of the lane, and the right lane, the obstacles around the target vehicle can be considered efficiently and as comprehensively as possible in the process of making lane changing decisions, so as to obtain a more accurate lane changing decision result.

[0119] Thus, the first state data of the target vehicle and the second state data of the surrounding obstacles can be obtained. The first state data and the second state data may include the same dimensions, such as speed and position, to facilitate the model to process the data.

[0120] Step S302: input the first state data and the second state data into a neural network model, and output a lane change decision result regarding the target vehicle.

[0121] The lane change decision result may include causing the target vehicle to execute any of the lane change actions of changing lanes to the left, changing lanes to the right, and maintaining the current lane. For example, if the lane change decision result is to change lanes to the left, the subsequent vehicle computer may control the target vehicle to execute the lane change action of changing lanes to the left based on the lane change decision result.

[0122] The neural network model may be a neural network model based on reinforcement learning, and the process of the neural network model processing the first state data and the second state data will be introduced in detail later.

[0123] According to the embodiment of the present application, by characterizing the road information with virtual obstacles and inputting the state data of the virtual obstacles into the neural network model, the influence of the road topology can be normalized and taken into account. By inputting the state data of other vehicles around the target vehicle into the model, the influence of other vehicles in a dynamic traffic environment can be taken into account. The method of the present application performs a complete environmental characterization through the second state data, normalizes the road information and the obstacle information, and is conducive to improving the generalization ability of the model. Therefore, by inputting the first state data of the target vehicle and the second state data of the obstacle into the neural network model, the output lane change decision result of the target vehicle is more realistic and more accurate. This method can be applied to vehicle-mounted terminal equipment, so that the lane change decision result can be provided in real time on the vehicle side, and effective lane change decisions can always be made in a complex actual road environment, while improving the driving efficiency of the vehicle, fully ensuring safety.

[0124] Since changing lanes to the left and to the right are symmetrical during vehicle driving, the symmetry of the lane changing problem is taken into consideration in the present application, so that changing lanes to the left and to the right share an internal network structure for evaluation, thereby reducing the size of the neural network model, removing redundant structures in the model, and optimizing the decision-making process of the model.

[0125] Optionally, the second state data may include third state data of obstacles on the lane where the target vehicle is located and on the left lane of the target lane, and fourth state data of obstacles on the lane where the target vehicle is located and on the right lane of the target lane. Therefore, after the second state data is input into the neural network model, it can be divided into two groups of data. The first group of data may include the first state data and the third state data, and the second group of data may include the first state data and the fourth state data. The neural network model can be used to process the first group of data and the second group of data respectively, and then the processing results of the two groups are combined to obtain the final evaluation values ​​corresponding to the three lanes. This process can be seen in Figure 5 , shows a flow chart of a vehicle lane change decision method according to an embodiment of the present application. Figure 5 As shown, step S302 may include:

[0126] Step S501: The neural network model determines a first set of evaluation values ​​based on the first state data and the third state data.

[0127] The first set of evaluation values ​​includes the evaluation value of the action of changing lanes to the left and the evaluation value of the action of keeping the current lane. These two evaluation values ​​can be the values ​​of the value function Q in reinforcement learning. The higher the evaluation value, the higher the benefit of taking the corresponding action, that is, the more the neural network model tends to output the decision to perform the corresponding action.

[0128] That is to say, the neural network model can determine the evaluation value of the action of changing lanes to the left and the evaluation value of the action of maintaining the current lane based on the state data of the target vehicle and the state data of the obstacle on the left lane of the lane where the target vehicle is located (i.e., the first set of data mentioned above), so as to judge whether the target vehicle should be controlled to change lanes to the left.

[0129] Optionally, in the process of determining the first set of evaluation values, the state data of the target vehicle and the state data of the obstacle may be separated and processed separately to reduce the amount of calculation during feature extraction. Referring to the following, step S501 may include:

[0130] The neural network model extracts features from the third state data to determine a first feature vector; the neural network model determines a first set of evaluation values ​​based on the first state data and the first feature vector.

[0131] The third state data may be encoded using a network structure such as a convolutional neural network (CNN) to extract features, so as to determine a first feature vector corresponding to obstacles in the lane where the target vehicle is located and in the left lane.

[0132] Next, the first state data and the first feature vector may be concatenated to form a concatenated vector, and the concatenated vector may be processed using a network structure such as a multi-layer perceptron (MLP) to determine an evaluation value of the action of changing lanes to the left and an evaluation value of the action of maintaining the current lane.

[0133] Similarly, for the evaluation of the right lane change action, the neural network model handles it symmetrically, as shown below.

[0134] Step S502: The neural network model determines a second set of evaluation values ​​based on the first state data and the fourth state data.

[0135] The second set of evaluation values ​​includes the evaluation value of the action of changing lanes to the right and the evaluation value of the action of keeping the current lane. These two evaluation values ​​can be the values ​​of the value function Q in reinforcement learning. The higher the evaluation value, the higher the benefit of taking the corresponding action, that is, the more the neural network model tends to output the decision to perform the corresponding action.

[0136] That is to say, the neural network model can symmetrically determine the evaluation value of the action of changing lanes to the right and the evaluation value of the action of maintaining the current lane based on the state data of the target vehicle and the state data of the obstacle on the right lane of the lane where the target vehicle is located (i.e., the second set of data mentioned above), so as to judge whether the target vehicle should be controlled to change lanes to the right.

[0137] Optionally, in the process of determining the second set of evaluation values, similarly, the state data of the target vehicle and the state data of the obstacle may be separated and processed separately to reduce the amount of calculation during feature extraction. See below, step S502 may include:

[0138] The neural network model extracts features from the fourth state data to determine a second feature vector; the neural network model determines a second set of evaluation values ​​based on the first state data and the second feature vector.

[0139] Among them, the fourth state data can be encoded using a network structure such as CNN to extract features to determine the second feature vector corresponding to the obstacles in the lane where the target vehicle is located and in the right lane.

[0140] Next, the first state data and the second feature vector may be concatenated to form a concatenated vector, and the concatenated vector may be processed using a network structure such as an MLP to determine an evaluation value of the action of changing lanes to the right and an evaluation value of the action of maintaining the current lane.

[0141] Step S503: The neural network model outputs a lane change decision result based on the first set of evaluation values ​​and the second set of evaluation values.

[0142] Among them, since the first set of evaluation values ​​includes the evaluation value of the action of changing lanes to the left and the evaluation value of the action of keeping the current lane, the second set of evaluation values ​​includes the evaluation value of the action of changing lanes to the right and the evaluation value of the action of keeping the current lane. The larger of the two evaluation values ​​of the action of keeping the current lane in the two sets of evaluation values, as well as the evaluation value of the action of changing lanes to the left and the evaluation value of the action of changing lanes to the right can be selected to form a final set of evaluation values, representing the evaluation results of the neural network model for these three actions respectively. The action corresponding to the maximum value of the three evaluation values ​​can be selected as the final output lane change decision result.

[0143] According to an embodiment of the present application, by dividing the second state data into third state data of obstacles in the lane where the target vehicle is located and in the lane to the left of the target vehicle's lane, and fourth state data of obstacles in the lane where the target vehicle is located and in the lane to the right of the target vehicle's lane, and determining two corresponding groups of evaluation values ​​based on the first state data and the third state data, and the first state data and the fourth state data, respectively, it is possible to symmetrically evaluate the actions of changing lanes to the left and changing lanes to the right. This construction method of a symmetrical input shared parameter network can significantly reduce the overall scale of the neural network model, remove redundant structures in the model, and optimize the decision-making process of the model.

[0144] Figure 6 A schematic diagram showing the structure of a neural network model according to an embodiment of the present application is shown. Figure 6As shown, left observation can represent the first state data and the third state data (i.e., the first set of data mentioned above), right observation can represent the first state data and the fourth state data (i.e., the second set of data mentioned above), and for each set of data, the state data of the target vehicle (such as ego info in the figure) and the state data of the obstacle (such as objects info in the figure) can be split. After segmentation, objects info can be feature extracted using a CNN encoder (such as CNN encoder in the figure), and the corresponding feature vector can be concatenated with ego info to obtain a concatenated vector. The concatenated vector can be used to obtain a corresponding evaluation value group using MLP. For the first set of data, the evaluation value of the action of changing lanes to the left (such as Q(left) in the figure) and the evaluation value of the action of keeping the current lane (which can be called Q(keep)_1) can be obtained; for the second set of data, the evaluation value of the action of changing lanes to the right (such as Q(right) in the figure) and the evaluation value of the action of keeping the current lane (which can be called Q(keep)_2) can be obtained. The larger value of Q(keep)_1 and Q(keep)_2 can be selected as the evaluation value of the action of keeping the current lane (such as Q(keep) in the figure). The action (action) corresponding to the largest value (argmax) among Q(left), Q(right), and Q(keep) is selected as the lane change decision result of the data. For example, when the Q(left) value is the largest, the lane change decision result may be to execute the action of changing lanes to the left.

[0145] Optionally, the neural network model may be a trained neural network model. Before the inference stage, the neural network model may be trained on a cloud server to obtain a trained neural network model. Figure 7-Figure 9 , the training phase of the vehicle lane changing decision method of an embodiment of the present application is introduced.

[0146] Figure 7 A flow chart of a vehicle lane change decision method according to an embodiment of the present application is shown. The method can be used in a cloud server, such as Figure 7 As shown, the method also includes:

[0147] Step S701, determining a training data set based on data generated by the vehicle during a lane change process.

[0148] The vehicles can be vehicles in real road environments or simulated vehicles. In other words, the source of the training data set can be the driving data of real vehicles or the simulated driving data of vehicles in the simulator. Since the vehicles in the simulator can adopt a random lane change strategy, the collected data can include reasonable lane changes (with benefits) and unreasonable lane changes (without benefits), which can enrich the data in the training data set. The lane change decisions taken in the driving data of real vehicles are generally profitable.

[0149] The lane changing process may be a process in which a vehicle changes lanes (successfully) or attempts to change lanes (unsuccessfully) during driving. Optionally, the lane changing process may include a vehicle lane changing process when a lane changing strategy of the vehicle is successfully executed and a lane keeping process when a lane changing strategy of the vehicle is not executed.

[0150] Figure 8 (a) and Figure 8 (b) is a schematic diagram showing a vehicle lane changing process according to an embodiment of the present application. The vehicle lane changing process when the vehicle lane changing strategy is successfully executed may be a process from the vehicle starting to generate a lateral deviation until the vehicle finally changes into the target lane, that is, Figure 8 (a) Lane-changing stage. The vehicle's lane-changing strategy is not executed because the vehicle's lane-changing decision cannot be executed by the lower layer due to safety factors, that is, the lane-changing decision cannot be actually executed because it is being monitored by the lower layer. In this case, lane keeping will continue. At this time, the lane-keeping process can be the process from the vehicle's lane-changing behavior being monitored by the lower layer to the end of the monitoring, that is, Figure 8 (b) The lane keeping stage.

[0151] In this way, training data can be collected in a more targeted manner, so that the collected data is of higher quality and various lane change situations are taken into account, thereby improving training efficiency and obtaining better training results.

[0152] The training data set may include state data at each moment in the lane changing process, lane changing actions, reward values ​​of the lane changing process, and state data at the next moment of the lane changing process.

[0153] The state data may include state data of the vehicle and state data of obstacles around the vehicle, and the state data may include position, speed, acceleration, etc. The lane changing action may be indicated by a left lane change / right lane change mark during a lane changing process.

[0154] In order to better evaluate the lane-changing behavior in complex scenarios, the reward value of the lane-changing process in this application can be determined without considering the specific situation at each moment in the lane-changing process. The reward value of the entire lane-changing process is calculated based on the state of the vehicle and an obstacle in front (which can be another vehicle or a virtual obstacle) at the initial moment and the end moment of the lane-changing process. In other words, the reward value is a posteriori value, so it can well evaluate the vehicle lane-changing behavior in complex scenarios.

[0155] The following is an introduction to one method of determining the reward value. Optionally, when the vehicle's lane changing strategy is successfully executed, the reward value of the lane changing process can be determined based on the distance and speed difference between the vehicle and the vehicle in front before the lane changing, and the distance and speed difference between the vehicle and the vehicle in front after the lane changing.

[0156] That is, when the vehicle's lane change strategy is successfully executed, the reward value can be calculated based on the speed gain and position gain before and after the lane change. One calculation method can be found in formula (1):

[0157] R=R d (d'-d)+R v (v'-v) Formula (1)

[0158] Among them, R can represent the reward value of the lane changing process. Fig. 9 , showing a schematic diagram of a method for determining a reward value according to an embodiment of the present application. Fig. 9 As shown, d can represent the distance between the vehicle and the obstacle in front at the initial moment of lane change, and d' can represent the distance between the vehicle and the obstacle in front at the end moment of lane change; v can represent the speed difference between the vehicle and the obstacle in front at the initial moment of lane change, and v' can represent the speed difference between the vehicle and the obstacle in front at the end moment of lane change.

[0159] R d (·) and R v (·) can represent the reward function for position gain and speed gain respectively. The reward function can be designed to reward a fixed score when the distance / speed difference reaches a predetermined value. For example, when the distance before and after the lane change reaches more than 30m, R d The value is 50. When the speed difference reaches 3m / s or more, R v The value of R is 50, otherwise d and R v The value of is 0.

[0160] Since the vehicle's lane-changing strategy is not executed generally due to safety considerations, that is, the corresponding lane-changing action violates the safety constraint. Therefore, when the vehicle's lane-changing strategy is not executed, the reward value of the lane-changing process can be a negative value, such as -50.

[0161] The data generated at each moment in the lane changing process can be saved as experience data to obtain a corresponding experience library. Therefore, when training the neural network model, a certain amount of experience data corresponding to the lane changing process can be randomly extracted from the experience library to form a training data set to train the model.

[0162] Step S702, using the training data set to train the initial neural network model to obtain a trained neural network model.

[0163] The structure of the initial neural network model can be found in the above Figure 6 The training data can be input into the neural network model according to Figure 5 The corresponding evaluation value Q is obtained in the manner shown. After obtaining the evaluation value Q, the loss function value can be calculated using the method shown in the following formula (2):

[0164] L(θ)=R+γ·max a’ ·Q(s',a')-Q(s) Formula (2)

[0165] Among them, L(θ) can represent the calculated loss function value, and θ can represent the parameters of the neural network model. R can represent the reward value of the lane changing process. γ can represent the decay rate, and its value range can be 0-1. a' can represent the lane changing action at the next moment. Since the neural network model has a symmetrical structure, the evaluation process a' for changing lanes to the left can be changing lanes to the left or keeping the current lane, and the evaluation process a' for changing lanes to the right can be changing lanes to the right or keeping the current lane. s can represent the state data at the current moment, s' can represent the state data at the next moment, and Q(·) can represent the corresponding evaluation value.

[0166] The loss function value can be calculated according to the above method, and the parameters of the neural network model can be iteratively updated using a deep double Q network (DDQN) and other methods until the model converges to obtain a trained neural network model. By using the evaluation value of the lane change process when calculating the loss function value, the trained neural network model can make effective and stable decisions in real dynamic scenes.

[0167] Therefore, the trained neural network model can be applied in the above-mentioned reasoning stage to make lane-changing decisions.

[0168] According to an embodiment of the present application, by utilizing the data generated during the lane changing process to determine the training data set, a large amount of vehicle straight-moving data that is not helpful for training is removed. During the training process, training can be performed based on higher-quality valid data, thereby improving the training efficiency of the model and making the performance of the trained model better.

[0169] Fig.10 FIG. 1 shows a structural diagram of a vehicle lane change decision device according to an embodiment of the present application. The device can be used in a vehicle terminal device, such as Fig.10 As shown, the device may include:

[0170] A first determining module 1001 is used to determine first state data of a target vehicle and second state data of obstacles around the target vehicle, where the obstacles around the target vehicle include other vehicles around the target vehicle and virtual obstacles used to represent road information;

[0171] The output module 1002 is used to input the first state data and the second state data into the neural network model, and output the lane changing decision result about the target vehicle, and the lane changing decision result includes making the target vehicle execute any one of the lane changing actions of changing lanes to the left, changing lanes to the right, and maintaining the current lane.

[0172] According to the embodiment of the present application, by characterizing the road information with virtual obstacles and inputting the state data of the virtual obstacles into the neural network model, the influence of the road topology can be normalized and taken into account. By inputting the state data of other vehicles around the target vehicle into the model, the influence of other vehicles in a dynamic traffic environment can be taken into account. The method of the present application performs a complete environmental characterization through the second state data, normalizes the road information and the obstacle information, and is conducive to improving the generalization ability of the model. Therefore, by inputting the first state data of the target vehicle and the second state data of the obstacle into the neural network model, the output lane change decision result of the target vehicle is more realistic and more accurate. This method can be applied to vehicle-mounted terminal equipment, so that the lane change decision result can be provided in real time on the vehicle side, and effective lane change decisions can always be made in a complex actual road environment, while improving the driving efficiency of the vehicle, fully ensuring safety.

[0173] Optionally, the road information may include road topology information, lane speed limit information and lane change restriction information, and the virtual obstacle includes a first virtual obstacle set based on the road topology information, a second virtual obstacle set based on the lane speed limit information and a third virtual obstacle set based on the lane change restriction information.

[0174] According to the embodiments of the present application, different virtual obstacles can be set based on different types of road information, so that different road conditions can be flexibly represented by virtual obstacles to achieve different driving restriction purposes. Based on this, virtual obstacles can also be used to accurately represent complex road scenes, which greatly broadens the application scenarios of the model and improves the accuracy of lane change decisions.

[0175] Optionally, the speed of the first virtual obstacle is 0, and it is set at the position where the lane ends or merges; the speed of the second virtual obstacle is the speed limit of the lane with a speed limit, and it is set at a position at a predetermined distance from the target vehicle on the lane with a speed limit; the speed of the third virtual obstacle is the speed of the target vehicle, and it is set at a position parallel to the target vehicle on the lane with restricted entry.

[0176] According to the embodiments of the present application, by setting the speed and position of the corresponding virtual obstacles according to different road conditions, the solution can be flexibly applied to different road scenes, and these virtual obstacles and other vehicles around the target vehicle can be represented in a normalized manner. Therefore, when making lane change decisions later, more scientific decision results can be obtained for various road scenes.

[0177] Optionally, the predetermined distance is a maximum distance that can be perceived by a sensor of the target vehicle or human eyes.

[0178] According to the embodiments of the present application, the actual vehicle driving process can be simulated more realistically, so that the lane speed limit can be more scientifically considered during the automatic driving process to achieve a better lane changing effect.

[0179] Optionally, the obstacles around the target vehicle include obstacles around the target vehicle in the lane where the target vehicle is located, the left lane of the lane where the target vehicle is located, and the right lane.

[0180] According to an embodiment of the present application, by selecting obstacles in the lane where the target vehicle is located, the left lane of the lane, and the right lane, the obstacles around the target vehicle can be considered efficiently and as comprehensively as possible in the process of making lane changing decisions, so as to obtain a more accurate lane changing decision result.

[0181] Optionally, the second state data includes third state data of obstacles on the lane where the target vehicle is located and on the left lane of the lane where the target vehicle is located, and fourth state data of obstacles on the lane where the target vehicle is located and on the right lane of the lane where the target vehicle is located, and the output module is used to:

[0182] The neural network model determines a first set of evaluation values ​​based on the first state data and the third state data, the first set of evaluation values ​​including an evaluation value of an action of changing lanes to the left and an evaluation value of an action of maintaining a current lane;

[0183] The neural network model determines a second set of evaluation values ​​based on the first state data and the fourth state data, the second set of evaluation values ​​including an evaluation value of the action of changing lanes to the right and an evaluation value of the action of maintaining the current lane;

[0184] The neural network model outputs a lane-changing decision result based on the first set of evaluation values ​​and the second set of evaluation values.

[0185] According to an embodiment of the present application, by dividing the second state data into third state data of obstacles in the lane where the target vehicle is located and in the lane to the left of the target vehicle's lane, and fourth state data of obstacles in the lane where the target vehicle is located and in the lane to the right of the target vehicle's lane, and determining two corresponding groups of evaluation values ​​based on the first state data and the third state data, and the first state data and the fourth state data, respectively, it is possible to symmetrically evaluate the actions of changing lanes to the left and changing lanes to the right. This construction method of a symmetrical input shared parameter network can significantly reduce the overall scale of the neural network model, remove redundant structures in the model, and optimize the decision-making process of the model.

[0186] Optionally, the neural network model determines a first set of evaluation values ​​based on the first state data and the third state data, including:

[0187] The neural network model extracts features from the third state data to determine a first feature vector;

[0188] The neural network model determines a first set of evaluation values ​​based on the first state data and the first eigenvector;

[0189] The neural network model determines a second set of evaluation values ​​based on the first state data and the fourth state data, including:

[0190] The neural network model performs feature extraction on the fourth state data to determine a second feature vector;

[0191] The neural network model determines a second set of evaluation values ​​based on the first state data and the second eigenvector.

[0192] According to the embodiment of the present application, by processing the vehicle status data and the obstacle status data separately and only performing feature extraction on the obstacle status data, the lane changing behavior can be better evaluated while reducing the amount of calculation, thereby improving the performance of the model.

[0193] Optionally, the neural network model is a trained neural network model, and the device further includes:

[0194] a second determination module, for determining a training data set based on data generated by the vehicle during a lane changing process, the training data set including state data at each moment in the lane changing process, a lane changing action, a reward value of the lane changing process, and state data at a next moment of the moment, the state data including state data of the vehicle and state data of obstacles around the vehicle;

[0195] The third determination module is used to train the initial neural network model using the training data set to obtain a trained neural network model.

[0196] According to an embodiment of the present application, by utilizing the data generated during the lane changing process to determine the training data set, a large amount of vehicle straight-moving data that is not helpful for training is removed. During the training process, training can be performed based on higher-quality valid data, thereby improving the training efficiency of the model and making the performance of the trained model better.

[0197] Optionally, the lane changing process includes a vehicle lane changing process when the vehicle's lane changing strategy is successfully executed and a lane keeping process when the vehicle's lane changing strategy is not executed.

[0198] According to the embodiments of the present application, training data can be collected in a more targeted manner, so that the collected data is of higher quality and various lane change situations are taken into account, thereby improving training efficiency and obtaining better training results.

[0199] Optionally, when the vehicle's lane changing strategy is successfully executed, the reward value of the lane changing process is determined based on the distance and speed difference between the vehicle and the vehicle in front before the lane changing, and the distance and speed difference between the vehicle and the vehicle in front after the lane changing; when the vehicle's lane changing strategy is not executed, the reward value of the lane changing process is a negative value.

[0200] According to the embodiment of the present application, by designing a calculation method for the reward value using the distance and speed difference at the beginning and end of the lane change, the reward value can be made into a posteriori value, reducing the amount of calculation, and can better evaluate the macro-benefits of the lane changing decision made by the vehicle in a complex dynamic traffic environment, reducing the impact of the micro-dynamic changes of the real-time traffic scene on the lane changing decision, so that the model can make effective and stable decisions in real dynamic scenes, thereby improving the performance of the model.

[0201] Optionally, the state data includes any one or more of position, velocity, and acceleration.

[0202] According to the embodiment of the present application, data of different dimensions can be taken into consideration for lane changing decisions, making the decision results more accurate.

[0203] An embodiment of the present application provides a vehicle lane change decision device, comprising: a processor and a memory for storing processor executable instructions; wherein the processor is configured to implement the above-mentioned vehicle lane change decision method when executing the instructions.

[0204] An embodiment of the present application provides a terminal device, which can execute the above-mentioned vehicle lane changing decision method.

[0205] An embodiment of the present application provides a non-volatile computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions implement the above-mentioned vehicle lane change decision method when executed by a processor.

[0206] An embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned vehicle lane change decision method.

[0207] Fig.11 FIG. 1 shows a structural diagram of an electronic device 1100 according to an embodiment of the present application. Fig.11 As shown, the electronic device 1100 may be the vehicle terminal or the cloud server, and is used to execute the above Figure 3-Figure 9 The electronic device 1100 includes at least one processor 1801, at least one memory 1802, and at least one communication interface 1803. In addition, the electronic device may also include common components such as an antenna, which will not be described in detail here.

[0208] Combine the following Fig.11 Each component of the electronic device 1100 is introduced in detail.

[0209] Processor 1801 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program. Processor 1801 may include one or more processing units, for example: processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0210] The communication interface 1803 is used to communicate with other electronic devices or communication networks, such as Ethernet, radio access network (RAN), core network, wireless local area network (WLAN), etc. For example, when the electronic device 1100 is a vehicle-side, the communication interface 1803 can be used to communicate with a cloud server. For another example, when the electronic device 1100 is a cloud server, the communication interface 1803 can be used to communicate with the vehicle-side.

[0211] The memory 1802 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0212] The memory 1802 is used to store application code for executing the above solution, and the execution is controlled by the processor 1801. The processor 1801 is used to execute the application code stored in the memory 1802.

[0213] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing.

[0215] The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0216] The computer program instructions for performing the operation of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present application.

[0217] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0218] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0219] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0220] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, system, method and computer program product according to multiple embodiments of the present application. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can also be executed in reverse order sometimes, depending on the functions involved.

[0221] It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.

[0222] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0223] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A vehicle lane-changing decision method, characterized in that: The method comprises: Determining first state data of a target vehicle and second state data of obstacles around the target vehicle, wherein the obstacles around the target vehicle include other vehicles around the target vehicle and virtual obstacles for representing road information; Inputting the first state data and the second state data into a reinforcement learning model, and outputting a lane changing decision result about the target vehicle, wherein the lane changing decision result includes causing the target vehicle to execute any one of a lane changing action of changing lanes to the left, changing lanes to the right, and maintaining the current lane; The second state data includes third state data of obstacles in the lane where the target vehicle is located and in the lane to the left of the lane where the target vehicle is located, and fourth state data of obstacles in the lane where the target vehicle is located and in the lane to the right of the lane where the target vehicle is located. The first state data and the second state data are input into the reinforcement learning model, and the lane change decision result about the target vehicle is output, including: The reinforcement learning model determines a first set of evaluation values ​​based on the first state data and the third state data, wherein the first set of evaluation values ​​includes an evaluation value of an action of changing lanes to the left and an evaluation value of an action of maintaining a current lane; The reinforcement learning model determines a second set of evaluation values ​​based on the first state data and the fourth state data, wherein the second set of evaluation values ​​includes an evaluation value of an action of changing lanes to the right and an evaluation value of an action of maintaining a current lane; The reinforcement learning model outputs the lane changing decision result based on the first set of evaluation values ​​and the second set of evaluation values.

2. The method according to claim 1, characterized in that The road information includes road topology information, lane speed limit information and lane change restriction information, and the virtual obstacle includes a first virtual obstacle set based on the road topology information, a second virtual obstacle set based on the lane speed limit information and a third virtual obstacle set based on the lane change restriction information.

3. The method according to claim 2, characterized in that The speed of the first virtual obstacle is 0, and it is set at the position where the lane ends or merges; the speed of the second virtual obstacle is the speed limit of the lane with a speed limit, and it is set at a position at a predetermined distance from the target vehicle on the lane with a speed limit; the speed of the third virtual obstacle is the speed of the target vehicle, and it is set at a position parallel to the target vehicle on the lane with restricted entry.

4. The method according to claim 3, characterized in that The predetermined distance is the maximum distance that can be perceived by a sensor of the target vehicle or human eyes.

5. The method according to any one of claims 1 to 4, characterized in that: The obstacles around the target vehicle include obstacles around the target vehicle in the lane where the target vehicle is located, the left lane of the lane where the target vehicle is located, and the obstacles around the target vehicle in the right lane.

6. The method according to claim 1, characterized in that The reinforcement learning model determines a first set of evaluation values ​​based on the first state data and the third state data, including: The reinforcement learning model performs feature extraction on the third state data to determine a first feature vector; The reinforcement learning model determines the first set of evaluation values ​​based on the first state data and the first feature vector; The reinforcement learning model determines a second set of evaluation values ​​based on the first state data and the fourth state data, including: The reinforcement learning model performs feature extraction on the fourth state data to determine a second feature vector; The reinforcement learning model determines the second set of evaluation values ​​based on the first state data and the second feature vector.

7. The method according to any one of claims 1 to 4 and 6, characterized in that: The reinforcement learning model is a trained reinforcement learning model, and the method further includes: Based on data generated by the vehicle during the lane changing process, a training data set is determined, wherein the training data set includes state data at each moment in the lane changing process, a lane changing action, a reward value of the lane changing process, and state data at the next moment of the moment, wherein the state data includes state data of the vehicle and state data of obstacles around the vehicle; The initial reinforcement learning model is trained using the training data set to obtain a trained reinforcement learning model.

8. The method according to claim 7, characterized in that The lane changing process includes a vehicle lane changing process when the vehicle's lane changing strategy is successfully executed and a lane keeping process when the vehicle's lane changing strategy is not executed.

9. The method according to claim 7, characterized in that: When the vehicle's lane changing strategy is successfully executed, the reward value of the lane changing process is determined based on the distance and speed difference between the vehicle and the vehicle in front before the lane changing, and the distance and speed difference between the vehicle and the vehicle in front after the lane changing; when the vehicle's lane changing strategy is not executed, the reward value of the lane changing process is a negative value.

10. The method according to any one of claims 1-4, 6, 8, and 9, characterized in that: The state data includes any one or more of position, speed, and acceleration.

11. A vehicle lane-changing decision device, characterized in that: The device comprises: A first determination module is used to determine first state data of a target vehicle and second state data of obstacles around the target vehicle, wherein the obstacles around the target vehicle include other vehicles around the target vehicle and virtual obstacles used to represent road information; an output module, configured to input the first state data and the second state data into a reinforcement learning model, and output a lane-changing decision result about a target vehicle, wherein the lane-changing decision result includes causing the target vehicle to execute any one of a lane-changing action of changing lanes to the left, changing lanes to the right, and maintaining a current lane; The second state data includes third state data of obstacles on the lane where the target vehicle is located and on the left lane of the lane where the target vehicle is located, and fourth state data of obstacles on the lane where the target vehicle is located and on the right lane of the lane where the target vehicle is located. The output module is used to: The reinforcement learning model determines a first set of evaluation values ​​based on the first state data and the third state data, wherein the first set of evaluation values ​​includes an evaluation value of an action of changing lanes to the left and an evaluation value of an action of maintaining a current lane; The reinforcement learning model determines a second set of evaluation values ​​based on the first state data and the fourth state data, wherein the second set of evaluation values ​​includes an evaluation value of an action of changing lanes to the right and an evaluation value of an action of maintaining a current lane; The reinforcement learning model outputs the lane changing decision result based on the first set of evaluation values ​​and the second set of evaluation values.

12. The device according to claim 11, characterized in that The road information includes road topology information, lane speed limit information and lane change restriction information, and the virtual obstacle includes a first virtual obstacle set based on the road topology information, a second virtual obstacle set based on the lane speed limit information and a third virtual obstacle set based on the lane change restriction information.

13. The device according to claim 12, characterized in that The speed of the first virtual obstacle is 0, and it is set at the position where the lane ends or merges; the speed of the second virtual obstacle is the speed limit of the lane with a speed limit, and it is set at a position at a predetermined distance from the target vehicle on the lane with a speed limit; the speed of the third virtual obstacle is the speed of the target vehicle, and it is set at a position parallel to the target vehicle on the lane with restricted entry.

14. The device according to claim 13, characterized in that The predetermined distance is the maximum distance that can be perceived by a sensor of the target vehicle or human eyes.

15. The device according to any one of claims 11 to 14, characterized in that: The obstacles around the target vehicle include obstacles around the target vehicle in the lane where the target vehicle is located, the left lane of the lane where the target vehicle is located, and the obstacles around the target vehicle in the right lane.

16. The device according to claim 11, characterized in that The reinforcement learning model determines a first set of evaluation values ​​based on the first state data and the third state data, including: The reinforcement learning model performs feature extraction on the third state data to determine a first feature vector; The reinforcement learning model determines the first set of evaluation values ​​based on the first state data and the first feature vector; The reinforcement learning model determines a second set of evaluation values ​​based on the first state data and the fourth state data, including: The reinforcement learning model performs feature extraction on the fourth state data to determine a second feature vector; The reinforcement learning model determines the second set of evaluation values ​​based on the first state data and the second feature vector.

17. The device according to any one of claims 11 to 14 and 16, characterized in that: The reinforcement learning model is a trained reinforcement learning model, and the device further comprises: A second determination module is used to determine a training data set based on data generated by the vehicle during the lane changing process, wherein the training data set includes state data at each moment in the lane changing process, a lane changing action, a reward value of the lane changing process, and state data at a next moment of the moment, wherein the state data includes state data of the vehicle and state data of obstacles around the vehicle; The third determination module is used to train the initial reinforcement learning model using the training data set to obtain a trained reinforcement learning model.

18. The device according to claim 17, characterized in that The lane changing process includes a vehicle lane changing process when the vehicle's lane changing strategy is successfully executed and a lane keeping process when the vehicle's lane changing strategy is not executed.

19. The device according to claim 17, characterized in that When the vehicle's lane changing strategy is successfully executed, the reward value of the lane changing process is determined based on the distance and speed difference between the vehicle and the vehicle in front before the lane changing, and the distance and speed difference between the vehicle and the vehicle in front after the lane changing; when the vehicle's lane changing strategy is not executed, the reward value of the lane changing process is a negative value.

20. The device according to any one of claims 11-14, 16, 18, 19, characterized in that: The state data includes any one or more of position, speed, and acceleration.

21. A vehicle lane-changing decision device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method according to any one of claims 1-10 when executing the instructions.

22. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

23. A computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method according to any one of claims 1 to 10.

Images