Automatic lane changing method, device and storage medium

By combining local neighbor features and global statistical features to generate target action instructions and using feedback information to update the control strategy, the global optimality problem in lane-changing decisions of autonomous vehicles is solved, and more accurate lane-changing strategy optimization is achieved.

CN115578711BActive Publication Date: 2026-11-10HUAWEI TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202211030605.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-05-21
Publication Date
2026-11-10
Estimated Expiration
2039-05-21

AI Technical Summary

Technical Problem

In existing autonomous driving technologies, autonomous vehicles only consider local neighbor vehicle information when making lane-changing decisions, failing to comprehensively consider the overall macro-level traffic flow situation, resulting in generated strategy actions that are not globally optimal.

Method used

By calculating the local neighbor features and global statistical features of autonomous vehicles, and combining them with the current control strategy, target action instructions are generated. The control strategy is then updated through feedback information. Taking into account local and global obstacle information, the gradient descent method is used to optimize the control strategy.

Benefits of technology

It achieves a globally optimal lane-changing strategy, ensuring the accuracy and adaptability of target actions at each moment, and improving the quality of lane-changing decisions for autonomous vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115578711B_ABST
    Figure CN115578711B_ABST
Patent Text Reader

Abstract

This application relates to artificial intelligence and provides an automatic lane-changing method, apparatus, and storage medium. The method includes: calculating the local neighbor features and global statistical features of the autonomous vehicle at the current moment based on the driving information of the autonomous vehicle at the current moment and the motion information of obstacles in each lane within the perception range of the autonomous vehicle; further, obtaining a target action instruction based on the local neighbor features, global statistical features, and the current control strategy, and executing the target action according to the target action instruction. It is evident that by introducing global statistical features into the current control strategy to obtain the target action instruction based on the local neighbor features, not only is the information of local neighbor obstacles considered, but also the macroscopic situation of global statistical features. Therefore, the target action obtained by integrating local and all road obstacle information is the globally optimal strategy action.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of invention application No. 201910426248.7, filed on May 21, 2019, entitled "Automatic lane changing method, apparatus and storage medium". Technical Field

[0002] This application relates to the field of autonomous driving technology, and in particular to an automatic lane changing method, device and storage medium. Background Technology

[0003] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0004] Autonomous driving is a mainstream application in the field of artificial intelligence. Autonomous driving technology relies on the collaborative efforts of computer vision, radar, monitoring devices, and global positioning systems to enable motor vehicles to operate autonomously without human intervention. Autonomous vehicles use various computing systems to help transport passengers from one location to another. Some autonomous vehicles may require initial or continuous input from an operator (such as a navigator, driver, or passenger). Autonomous vehicles allow operators to switch from manual mode to autonomous driving mode or a mode in between. Because autonomous driving technology eliminates the need for human drivers, it theoretically can effectively avoid human driving errors, reduce traffic accidents, and improve highway transportation efficiency. Therefore, autonomous driving technology is receiving increasing attention. However, the design of intelligent lane-changing decisions for autonomous vehicles still faces significant challenges in the field of autonomous driving technology.

[0005] In related technologies, deep networks are used to simulate the action value function Q corresponding to the current state and action. The input of the action value function Q is a local relative semantic grid centered on the autonomous vehicle. By considering the speed and distance information of the nearest neighboring vehicles to the autonomous vehicle, as well as some road semantic information (such as whether the autonomous vehicle is in an acceleration lane or a left-turn lane), the action that maximizes the action value function Q is selected as the current decision action.

[0006] However, the related technologies only consider the local neighbor vehicle information of the vehicle itself, without considering the overall traffic flow situation. Therefore, the generated strategy action is not the globally optimal strategy action. Summary of the Invention

[0007] This application provides an automatic lane-changing method, apparatus, and storage medium, which solves the problem that the generated strategy actions in related technologies are not the globally optimal strategy actions.

[0008] In a first aspect, embodiments of this application provide an automatic lane-changing method, including:

[0009] Based on the current driving information of the autonomous vehicle and the motion information of obstacles in each lane within the perception range of the autonomous vehicle, the local neighbor features and global statistical features of the autonomous vehicle at the current moment are calculated. The local neighbor features are used to represent the motion state information of specific neighbor obstacles of the autonomous vehicle relative to the autonomous vehicle. The global statistical features are used to represent the sparsity and density of obstacles in each lane within the perception range.

[0010] The target action instruction is obtained based on the local neighbor characteristics, the global statistical characteristics and the current control policy. The target action instruction is used to instruct the autonomous vehicle to perform the target action, which includes at least two types: changing lanes or maintaining straight driving.

[0011] Perform the target action according to the target action instruction.

[0012] In the automatic lane-changing method provided in the first aspect, the local neighbor features and global statistical features of the autonomous vehicle at the current moment are calculated based on the driving information of the autonomous vehicle at the current moment and the movement information of obstacles in each lane within the perception range of the autonomous vehicle. Further, the target action instruction is obtained based on the local neighbor features, global statistical features, and the current control strategy, and the target action is executed according to the target action instruction. It can be seen that by introducing global statistical features into the current control strategy to obtain the target action instruction based on the local neighbor features, not only is the information of local neighbor obstacles (such as other vehicles) considered, but also the macroscopic situation of global statistical features (such as the overall traffic flow). Therefore, the target action obtained by integrating local and all road obstacle information is the globally optimal strategy action.

[0013] In one possible implementation, the method further includes:

[0014] Feedback information is obtained by executing the target action, and this feedback information is used to update the current control strategy. This feedback information includes the driving information of the autonomous vehicle after executing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the autonomous vehicle's perception range at the next moment. When the target action is a lane change, the feedback information also includes: the ratio of the time to execute the target action to the historical average time, and the changes in the density of the lane the autonomous vehicle is in after the lane change compared to the lane it was in before the lane change. The historical average time includes the average time the autonomous vehicle takes to execute similar actions within a preset historical time period.

[0015] The current control strategy is updated based on the feedback information to obtain the control strategy for the next moment.

[0016] In this implementation, feedback information is obtained by executing the target action, and the current control strategy is updated based on the feedback information to obtain the control strategy for the next moment. This allows the target action for the next moment to be accurately determined based on the control strategy for that moment. It is worth noting that the control strategy for moment t can be continuously updated based on the feedback information at moment t to obtain the control strategy for moment t+1. This ensures that the control strategy for generating the target action is continuously and adaptively updated and optimized, guaranteeing that there is an optimal control strategy for each moment and ensuring the accurate generation of the target action at each moment.

[0017] In one possible implementation, updating the current control policy based on the feedback information to obtain the control policy for the next moment includes:

[0018] Based on the feedback information, calculate the reward value corresponding to the target action, as well as the local neighbor features and global statistical features of the autonomous vehicle in the next moment;

[0019] Determine the quadruple information at the current moment; wherein, the quadruple information at the current moment corresponds to the vehicle condition at the current moment, including: the feature at the current moment, the target action, the reward value corresponding to the target action, and the feature at the next moment. The feature at the current moment includes the local neighbor features and global statistical features of the autonomous vehicle at the current moment, and the feature at the next moment includes the local neighbor features and global statistical features of the autonomous vehicle at the next moment.

[0020] The control strategy for the next moment is obtained by updating the current control strategy based on the quaternion information at the current moment.

[0021] In one possible implementation, when the target action is to maintain straight movement, the control policy for the next moment is updated based on the current four-tuple information, including:

[0022] Based on the quaternion information at the current moment, generate the target value corresponding to the quaternion information;

[0023] The parameter q in the first preset function containing the target value is iteratively updated using the gradient descent method;

[0024] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0025] In one possible implementation, when the target action is a lane change, the control policy for the next moment is updated based on the current four-tuple information, including:

[0026] Obtain the extended quadruple information for the current moment, which corresponds to the extended vehicle condition for the current moment. The extended vehicle condition for the current moment is obtained by processing the vehicle condition for the current moment using symmetry and monotonic rules. The symmetry rule refers to symmetrically transforming the positions of obstacles in all lanes on both sides of the lane where the autonomous vehicle is located, with the lane as the axis. The monotonic rule refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane and / or, the distance between the obstacles in front and behind the autonomous vehicle in the non-target lane changes to a range less than a preset distance.

[0027] Based on the current quadruple information and the extended quadruple information at the current moment, the current control policy is updated to obtain the control policy for the next moment.

[0028] In one possible implementation, the control policy for the next moment is obtained by updating the current control policy based on the current four-tuple information and the current extended four-tuple information, including:

[0029] Based on the quaternion information at the current moment and the ith quaternion information in the extended quaternion information at the current moment, generate the target value corresponding to the ith quaternion information; where i is a positive integer not greater than n, and n is the total number of quaternion information included in the quaternion information at the current moment and the extended quaternion information at the current moment.

[0030] The parameter q in the second preset function containing the target value corresponding to the i-th quadtuple is iteratively updated using the gradient descent method;

[0031] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0032] In one possible implementation, when the target action is to maintain straight movement, the control policy for the next moment is updated based on the current four-tuple information, including:

[0033] The control strategy for the next moment is obtained by updating the current control strategy based on the quaternion information at the current moment, the quaternion information at the historical moment, and the extended quaternion information at the historical moment.

[0034] The four-tuple information of this historical moment corresponds to the vehicle condition at this historical moment, including: the features of this historical moment, the target action of this historical moment, the reward value corresponding to the target action of this historical moment, and the features of the next moment of this historical moment. The features of this historical moment include the local neighbor features and global statistical features of the autonomous vehicle at this historical moment, and the features of the next moment of this historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of this historical moment. The extended four-tuple information of this historical moment corresponds to the extended vehicle condition of this historical moment, which is obtained by processing the vehicle condition of this historical moment with symmetric rules and monotonic rules.

[0035] In one possible implementation, the control policy for the next moment is obtained by updating the current control policy based on the current four-tuple information, the four-tuple information of the historical moment, and the extended four-tuple information of the historical moment, including:

[0036] Based on the j-th quadruple information in the current time, the historical time, and the extended quadruple information of the historical time, generate the target value corresponding to the j-th quadruple information; where j is a positive integer not greater than m, and m is the total number of quadruple information included in the current time, the historical time, and the extended quadruple information of the historical time.

[0037] The gradient descent method is used to iteratively update the parameter q in the third preset function that contains the target value corresponding to the j-th quadtuple information;

[0038] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0039] In one possible implementation, when the target action is a lane change, the control policy for the next moment is updated based on the current four-tuple information, including:

[0040] Obtain the extended quadruple information at the current moment; wherein, the extended quadruple information at the current moment corresponds to the extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment with symmetric rules and monotonic rules;

[0041] Based on the current four-tuple information, the current extended four-tuple information, the historical four-tuple information, and the historical extended four-tuple information, the current control strategy is updated to obtain the control strategy for the next moment. Among them, the historical four-tuple information corresponds to the vehicle condition at the historical moment, and the historical extended four-tuple information corresponds to the extended vehicle condition at the historical moment. The extended vehicle condition at the historical moment is obtained by processing the historical vehicle condition with symmetric rules and monotonic rules.

[0042] In one possible implementation, the control policy for the next moment is obtained by updating the current control policy based on the current four-tuple information, the current extended four-tuple information, the four-tuple information of the historical moment, and the extended four-tuple information of the historical moment, including:

[0043] Based on the k-th quaternion information from the current quaternion information, the extended quaternion information from the current quaternion information, the quaternion information from the historical quaternion information, and the extended quaternion information from the historical quaternion information, generate the target value corresponding to the k-th quaternion information; where k is a positive integer not greater than p, and p is the total number of quaternion information included in the current quaternion information, the extended quaternion information from the current quaternion information, the quaternion information from the historical quaternion information, and the extended quaternion information from the historical quaternion information.

[0044] The gradient descent method is used to iteratively update the parameter q in the fourth preset function that contains the target value corresponding to the kth quadtuple information;

[0045] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0046] In one possible implementation, when the target action is to maintain straight-line movement, the reward value corresponding to the target action is calculated based on the feedback information, along with the local neighbor features and global statistical features of the autonomous vehicle at the next moment, including:

[0047] The reward value is calculated based on the driving information of the autonomous vehicle after it performs the target action.

[0048] Based on the autonomous vehicle's driving information at the next moment and the movement information of obstacles in each lane within the autonomous vehicle's perception range at the next moment, calculate the autonomous vehicle's local neighbor features and global statistical features at the next moment.

[0049] In one possible implementation, when the target action is a lane change, the reward value corresponding to the target action is calculated based on the feedback information, as well as the local neighbor features and global statistical features of the autonomous vehicle at the next moment, including:

[0050] The reward value is calculated based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after changing lanes compared to the lane before changing lanes.

[0051] Based on the autonomous vehicle's driving information at the next moment and the movement information of obstacles in each lane within the autonomous vehicle's perception range at the next moment, calculate the autonomous vehicle's local neighbor features and global statistical features at the next moment.

[0052] In one possible implementation, the specific neighboring obstacles of the autonomous vehicle include at least one of the following: obstacles in front and behind the autonomous vehicle in the lane where the autonomous vehicle is located, obstacles in front and behind the autonomous vehicle in the adjacent left lane of the lane where the autonomous vehicle is located, and obstacles in front and behind the autonomous vehicle in the adjacent right lane of the lane where the autonomous vehicle is located.

[0053] Wherein, when the autonomous vehicle is in the left lane, the motion state information of the obstacles in front of and behind the autonomous vehicle in the adjacent left lane relative to the autonomous vehicle is a default value; and / or,

[0054] When the autonomous vehicle is in the right lane, the motion state information of the obstacles in front and behind the autonomous vehicle in the adjacent right lane is the default value relative to the autonomous vehicle.

[0055] In one possible implementation, the global traffic flow statistics of the autonomous vehicle at the current moment include at least one of the following: the average speed and average interval of all obstacles in each lane within the perception range.

[0056] Secondly, embodiments of this application provide an automatic lane-changing device, comprising:

[0057] The calculation module is used to calculate the local neighbor features and global statistical features of the autonomous vehicle at the current moment based on the driving information of the autonomous vehicle at the current moment and the motion information of obstacles in each lane within the perception range of the autonomous vehicle. The local neighbor features are used to represent the motion state information of specific neighbor obstacles of the autonomous vehicle relative to the autonomous vehicle. The global statistical features are used to represent the sparsity and density of obstacles in each lane within the perception range.

[0058] The acquisition module is used to acquire a target action instruction based on the local neighbor features, the global statistical features and the current control policy. The target action instruction is used to instruct the autonomous vehicle to perform a target action, which includes at least two types: lane changing or maintaining straight driving.

[0059] The execution module is used to execute the target action according to the target action instruction.

[0060] In one possible implementation, the device further includes:

[0061] A feedback module is used to obtain feedback information by executing the target action, and the feedback information is used to update the current control strategy. The feedback information includes the driving information of the autonomous vehicle after executing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment. When the target action is a lane change, the feedback information also includes: the ratio of the time to execute the target action to the historical average time, and the changes in the density of the lane the autonomous vehicle is in after the lane change compared to the lane it was in before the lane change; wherein the historical average time includes the average time of the autonomous vehicle executing similar actions within a preset historical time period.

[0062] The update module is used to update the current control strategy based on the feedback information to obtain the control strategy for the next moment.

[0063] In one possible implementation, the update module includes:

[0064] The calculation unit is used to calculate the reward value corresponding to the target action based on the feedback information, as well as the local neighbor features and global statistical features of the autonomous vehicle in the next moment.

[0065] The determining unit is used to determine the quadruple information at the current moment; wherein the quadruple information at the current moment corresponds to the vehicle condition at the current moment, including: the feature at the current moment, the target action, the reward value corresponding to the target action, and the feature at the next moment. The feature at the current moment includes the local neighbor features and global statistical features of the autonomous vehicle at the current moment, and the feature at the next moment includes the local neighbor features and global statistical features of the autonomous vehicle at the next moment.

[0066] The update unit is used to update the current control strategy based on the quadruple information at the current moment to obtain the control strategy for the next moment.

[0067] In one possible implementation, when the target action is to maintain a straight line, the update unit is specifically used for:

[0068] Based on the quaternion information at the current moment, generate the target value corresponding to the quaternion information;

[0069] The parameter q in the first preset function containing the target value is iteratively updated using the gradient descent method;

[0070] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0071] In one possible implementation, when the target action is a lane change, the update unit is specifically used for:

[0072] Obtain the extended quadruple information for the current moment, which corresponds to the extended vehicle condition for the current moment. The extended vehicle condition for the current moment is obtained by processing the vehicle condition for the current moment using symmetry and monotonic rules. The symmetry rule refers to symmetrically transforming the positions of obstacles in all lanes on both sides of the lane where the autonomous vehicle is located, with the lane as the axis. The monotonic rule refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane and / or, the distance between the obstacles in front and behind the autonomous vehicle in the non-target lane changes to a range less than a preset distance.

[0073] Based on the current quadruple information and the extended quadruple information at the current moment, the current control policy is updated to obtain the control policy for the next moment.

[0074] In one possible implementation, the update unit is specifically used for:

[0075] Based on the quaternion information at the current moment and the ith quaternion information in the extended quaternion information at the current moment, generate the target value corresponding to the ith quaternion information; where i is a positive integer not greater than n, and n is the total number of quaternion information included in the quaternion information at the current moment and the extended quaternion information at the current moment.

[0076] The parameter q in the second preset function containing the target value corresponding to the i-th quadtuple is iteratively updated using the gradient descent method;

[0077] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0078] In one possible implementation, when the target action is to maintain a straight line, the update unit is specifically used for:

[0079] The control strategy for the next moment is obtained by updating the current control strategy based on the quaternion information at the current moment, the quaternion information at the historical moment, and the extended quaternion information at the historical moment.

[0080] The four-tuple information of this historical moment corresponds to the vehicle condition at this historical moment, including: the features of this historical moment, the target action of this historical moment, the reward value corresponding to the target action of this historical moment, and the features of the next moment of this historical moment. The features of this historical moment include the local neighbor features and global statistical features of the autonomous vehicle at this historical moment, and the features of the next moment of this historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of this historical moment. The extended four-tuple information of this historical moment corresponds to the extended vehicle condition of this historical moment, which is obtained by processing the vehicle condition of this historical moment with symmetric rules and monotonic rules.

[0081] In one possible implementation, the update unit is specifically used for:

[0082] Based on the j-th quadruple information in the current time, the historical time, and the extended quadruple information of the historical time, generate the target value corresponding to the j-th quadruple information; where j is a positive integer not greater than m, and m is the total number of quadruple information included in the current time, the historical time, and the extended quadruple information of the historical time.

[0083] The gradient descent method is used to iteratively update the parameter q in the third preset function that contains the target value corresponding to the j-th quadtuple information;

[0084] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0085] In one possible implementation, when the target action is a lane change, the update unit is specifically used for:

[0086] Obtain the extended quadruple information at the current moment; wherein, the extended quadruple information at the current moment corresponds to the extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment with symmetric rules and monotonic rules;

[0087] Based on the current four-tuple information, the current extended four-tuple information, the historical four-tuple information, and the historical extended four-tuple information, the current control strategy is updated to obtain the control strategy for the next moment. Among them, the historical four-tuple information corresponds to the vehicle condition at the historical moment, and the historical extended four-tuple information corresponds to the extended vehicle condition at the historical moment. The extended vehicle condition at the historical moment is obtained by processing the historical vehicle condition with symmetric rules and monotonic rules.

[0088] In one possible implementation, the update unit is specifically used for:

[0089] Based on the k-th quaternion information from the current quaternion information, the extended quaternion information from the current quaternion information, the quaternion information from the historical quaternion information, and the extended quaternion information from the historical quaternion information, generate the target value corresponding to the k-th quaternion information; where k is a positive integer not greater than p, and p is the total number of quaternion information included in the current quaternion information, the extended quaternion information from the current quaternion information, the quaternion information from the historical quaternion information, and the extended quaternion information from the historical quaternion information.

[0090] The gradient descent method is used to iteratively update the parameter q in the fourth preset function that contains the target value corresponding to the kth quadtuple information;

[0091] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0092] In one possible implementation, when the target action is to maintain straight movement, the computational unit is specifically used for:

[0093] The reward value is calculated based on the driving information of the autonomous vehicle after it performs the target action.

[0094] Based on the autonomous vehicle's driving information at the next moment and the movement information of obstacles in each lane within the autonomous vehicle's perception range at the next moment, calculate the autonomous vehicle's local neighbor features and global statistical features at the next moment.

[0095] In one possible implementation, when the target action is a lane change, the computing unit is specifically used for:

[0096] The reward value is calculated based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after changing lanes compared to the lane before changing lanes.

[0097] Based on the autonomous vehicle's driving information at the next moment and the movement information of obstacles in each lane within the autonomous vehicle's perception range at the next moment, calculate the autonomous vehicle's local neighbor features and global statistical features at the next moment.

[0098] In one possible implementation, the specific neighboring obstacles of the autonomous vehicle include at least one of the following: obstacles in front and behind the autonomous vehicle in the lane where the autonomous vehicle is located, obstacles in front and behind the autonomous vehicle in the adjacent left lane of the lane where the autonomous vehicle is located, and obstacles in front and behind the autonomous vehicle in the adjacent right lane of the lane where the autonomous vehicle is located.

[0099] Wherein, when the autonomous vehicle is in the left lane, the motion state information of the obstacles in front of and behind the autonomous vehicle in the adjacent left lane relative to the autonomous vehicle is a default value; and / or,

[0100] When the autonomous vehicle is in the right lane, the motion state information of the obstacles in front and behind the autonomous vehicle in the adjacent right lane is the default value relative to the autonomous vehicle.

[0101] In one possible implementation, the global traffic flow statistics of the autonomous vehicle at the current moment include at least one of the following: the average speed and average interval of all obstacles in each lane within the perception range.

[0102] Thirdly, embodiments of this application provide an automatic lane-changing device, including: a processor and a memory;

[0103] This memory is used to store program instructions;

[0104] The processor is used to call and execute program instructions stored in the memory. When the processor executes the program instructions stored in the memory, the automatic lane-changing device is used to execute the method described in any implementation of the first aspect above.

[0105] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any implementation of the first aspect.

[0106] Fifthly, embodiments of this application provide a program that, when executed by a processor, is used to perform the method described in any implementation of the first aspect described above.

[0107] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in any implementation of the first aspect described above.

[0108] Seventhly, embodiments of this application provide a method for training a control strategy, comprising:

[0109] Step A: Obtain a preset number of four-tuple information for historical moments. The four-tuple information for each historical moment corresponds to the vehicle condition at that historical moment, including: the features of that historical moment, the target action of the autonomous vehicle at that historical moment, the reward value corresponding to the target action at that historical moment, and the features of the next moment of that historical moment. The features of that historical moment include the local neighbor features and global statistical features of the autonomous vehicle at that historical moment. The features of the next moment of that historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of that historical moment.

[0110] Step B: Update the current control policy based on the quadruple information of at least one first historical moment, the extended quadruple information of the at least one first historical moment, and the quadruple information of at least one second historical moment to obtain the control policy for the next moment.

[0111] Wherein, the number of times steps A and B are executed repeatedly reaches a preset number, or steps A and B are executed repeatedly until the updated control strategy meets the preset conditions and then stops; the control strategy finally obtained by repeatedly executing steps A and B is used by the automatic lane changing device to obtain the target action instruction when executing the automatic lane changing method.

[0112] Wherein, the at least one first historical moment's quadruple information is the quadruple information of the historical moment whose target action is lane change among the quadruple information of the preset number of historical moments; the at least one second historical moment's quadruple information is the quadruple information of other historical moments besides the at least one first historical moment's quadruple information among the quadruple information of the preset number of historical moments; any extended quadruple information of the first historical moment corresponds to the extended vehicle condition of the first historical moment, and the extended vehicle condition of the first historical moment is obtained by processing the vehicle condition of the first historical moment with symmetry rules and monotonic rules.

[0113] The training method for the control strategy provided in the seventh aspect involves acquiring quadruple information from a preset number of historical time moments. Further, based on quadruple information from at least one first historical time moment, at least one extended quadruple information from the first historical time moment, and at least one quadruple information from the second historical time moment, the current control strategy is updated to obtain the control strategy for the next time moment. It is evident that by updating the current control strategy based on the extended quadruple information from the first historical time moment within the preset number of historical time moments' quadruple information, a more accurate control strategy can be obtained, enabling precise determination of the corresponding target action.

[0114] In one possible implementation, updating the current control policy to obtain the control policy for the next moment based on at least one first historical moment's quadruple information, the at least one first historical moment's extended quadruple information, and at least one second historical moment's quadruple information includes:

[0115] Based on the quadruple information of at least one first historical moment, the extended quadruple information of at least one first historical moment, and the first quadruple information in the quadruple information of at least one second historical moment, a target value corresponding to the first quadruple information is generated; wherein, l is a positive integer not greater than q, and q is the total number of quadruple information included in the quadruple information of at least one first historical moment, the extended quadruple information of at least one first historical moment, and the quadruple information of at least one second historical moment;

[0116] The gradient descent method is used to iteratively update the parameter q in the preset function that contains the target value corresponding to the information of the first quadruplet;

[0117] Replace the parameter q in the current control strategy with the updated parameter q to obtain the control strategy for the next time step.

[0118] In one possible implementation, before obtaining the quadruple information for a preset number of historical moments, the following steps are also included:

[0119] For each historical moment, based on the driving information of the autonomous vehicle and the movement information of obstacles in each lane within the perception range of the autonomous vehicle, the local neighbor features and global statistical features of the autonomous vehicle at that historical moment are calculated.

[0120] The target action indication for that historical moment is obtained based on the local neighbor characteristics, global statistical characteristics, and control policy of that historical moment. The target action indication is used to instruct the autonomous vehicle to perform the target action, which includes at least two types: changing lanes or maintaining straight driving.

[0121] Feedback information is obtained by executing the target action; the feedback information includes the driving information of the autonomous vehicle after executing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment; when the target action is a lane change, the feedback information also includes: the ratio of the time to execute the target action to the historical average time, and the change in the sparseness and density of the lane where the autonomous vehicle is located after the lane change compared to the lane before the lane change; wherein, the historical average time includes the average time of the autonomous vehicle executing similar actions within a preset historical time period;

[0122] The reward value corresponding to the target action is calculated based on the feedback information, as well as the local neighbor features and global traffic flow statistics features of the autonomous vehicle at the next moment of the historical moment.

[0123] Store the quadruple information for this historical moment.

[0124] In one possible implementation, when the target action is to maintain a straight line, the reward value corresponding to the target action is calculated based on the feedback information, including:

[0125] The reward value is calculated based on the driving information of the autonomous vehicle after it performs the target action.

[0126] In one possible implementation, when the target action is a lane change, the reward value corresponding to the target action is calculated based on the feedback information, including:

[0127] The reward value is calculated based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after changing lanes compared to the lane before changing lanes.

[0128] Eighthly, embodiments of this application provide a training apparatus for a control strategy, comprising:

[0129] The first acquisition module is used to perform step A: acquire a preset number of four-tuple information of historical moments, wherein the four-tuple information of the historical moment corresponds to the vehicle condition of the historical moment, including: the features of the historical moment, the target action of the autonomous vehicle at the historical moment, the reward value corresponding to the target action at the historical moment, and the features of the next moment of the historical moment. The features of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the historical moment, and the features of the next moment of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of the historical moment.

[0130] The update module is used to perform step B: update the current control strategy to obtain the control strategy for the next moment based on the quadruple information of at least one first historical moment, the extended quadruple information of the at least one first historical moment, and the quadruple information of at least one second historical moment.

[0131] Wherein, the number of times steps A and B are executed repeatedly reaches a preset number, or steps A and B are executed repeatedly until the updated control strategy meets the preset conditions and then stops; the control strategy finally obtained by repeatedly executing steps A and B is used by the automatic lane changing device to obtain the target action instruction when executing the automatic lane changing method.

[0132] Wherein, the at least one first historical moment's quadruple information is the quadruple information of the historical moment whose target action is lane change among the quadruple information of the preset number of historical moments; the at least one second historical moment's quadruple information is the quadruple information of other historical moments besides the at least one first historical moment's quadruple information among the quadruple information of the preset number of historical moments; any extended quadruple information of the first historical moment corresponds to the extended vehicle condition of the first historical moment, and the extended vehicle condition of the first historical moment is obtained by processing the vehicle condition of the first historical moment with symmetry rules and monotonic rules.

[0133] In one possible implementation, the update module includes:

[0134] The generation unit is configured to generate a target value corresponding to the first quadruple information based on the quadruple information of the at least one first historical moment, the extended quadruple information of the at least one first historical moment, and the first quadruple information of the quadruple information of the at least one second historical moment; wherein, l is a positive integer not greater than q, and q is the total number of quadruple information included in the quadruple information of the at least one first historical moment, the extended quadruple information of the at least one first historical moment, and the quadruple information of the at least one second historical moment;

[0135] The update unit is used to iteratively update the parameter q in the preset function containing the target value corresponding to the first quadruplet information using the gradient descent method;

[0136] The replacement unit is used to replace the parameter q in the current control strategy with the iteratively updated parameter q to obtain the control strategy for the next time step.

[0137] In one possible implementation, the device further includes:

[0138] The first calculation module is used to calculate the local neighbor features and global statistical features of the autonomous vehicle at each historical moment, based on the driving information of the autonomous vehicle and the movement information of obstacles in each lane within the perception range of the autonomous vehicle.

[0139] The second acquisition module is used to acquire the target action indication at the historical moment based on the local neighbor characteristics, global statistical characteristics and control strategy at the historical moment. The target action indication is used to instruct the autonomous vehicle to perform the target action, which includes at least two types: lane changing or keeping straight.

[0140] The feedback module is used to obtain feedback information by executing the target action; wherein, the feedback information includes the driving information of the autonomous vehicle after executing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment; when the target action is lane changing, the feedback information also includes: the ratio of the time to execute the target action to the historical average time, and the change in the sparseness and density of the lane where the autonomous vehicle is located after lane changing compared to the lane before lane changing; wherein, the historical average time includes the average time of the autonomous vehicle executing similar actions within a preset historical time period;

[0141] The second calculation module is used to calculate the reward value corresponding to the target action based on the feedback information, as well as the local neighbor features and global traffic flow statistics features of the autonomous vehicle at the next moment of the historical moment.

[0142] The storage module is used to store the quadruple information at this historical moment.

[0143] In one possible implementation, when the target action is to maintain straight movement, the second calculation module is specifically used for:

[0144] The reward value is calculated based on the driving information of the autonomous vehicle after it performs the target action.

[0145] In one possible implementation, when the target action is a lane change, the second calculation module is specifically used for:

[0146] The reward value is calculated based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after changing lanes compared to the lane before changing lanes.

[0147] Ninthly, embodiments of this application provide a training apparatus for a control strategy, comprising: a processor and a memory;

[0148] This memory is used to store program instructions;

[0149] The processor is used to call and execute program instructions stored in the memory. When the processor executes the program instructions stored in the memory, the training device for the control strategy is used to execute the method described in any implementation of the seventh aspect above.

[0150] In a tenth aspect, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any implementation of the seventh aspect above.

[0151] Eleventhly, embodiments of this application provide a program that, when executed by a processor, is used to perform the method described in any implementation of the seventh aspect above.

[0152] In a twelfth aspect, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in any implementation of the seventh aspect above.

[0153] In a thirteenth aspect, embodiments of this application provide a chip including a processor and a data interface. The processor reads instructions stored in a memory through the data interface and executes the methods described in any implementation of the first or seventh aspect.

[0154] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method described in any implementation of the first or seventh aspect described above.

[0155] In a fourteenth aspect, embodiments of this application provide an electronic device that includes the automatic lane-changing device described in any implementation of the second or third aspect above.

[0156] In a fifteenth aspect, embodiments of this application provide an electronic device that includes a training apparatus for the control strategy described in any implementation of the second or third aspect above. Attached Figure Description

[0157] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application;

[0158] Figure 2 This is a functional block diagram of the vehicle 100 provided in the embodiments of this application;

[0159] Figure 3 for Figure 2 A schematic diagram of the structure of a computer system in the image;

[0160] Figure 4 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application;

[0161] Figure 5 This is a schematic diagram of the operating environment provided for an embodiment of this application;

[0162] Figure 6 Symmetry principle illustration provided for embodiments of this application Figure 1 ;

[0163] Figure 7 Symmetry principle illustration provided for embodiments of this application Figure 2 ;

[0164] Figure 8 A schematic diagram illustrating the monotonicity principle provided for embodiments of this application;

[0165] Figure 9 A flowchart illustrating a training method for a control strategy provided in an embodiment of this application;

[0166] Figure 10 A flowchart illustrating a training method for a control strategy provided in another embodiment of this application;

[0167] Figure 11 A schematic flowchart of an automatic lane-changing method provided in an embodiment of this application;

[0168] Figure 12 A flowchart illustrating an automatic lane-changing method provided in another embodiment of this application;

[0169] Figure 13 This is a schematic diagram of training data provided in an embodiment of this application;

[0170] Figure 14 This is a schematic diagram of the structure of an automatic lane changing device provided in an embodiment of this application;

[0171] Figure 15 This is a schematic diagram of the structure of an automatic lane-changing device provided in another embodiment of this application;

[0172] Figure 16 A conceptual partial view of a computer program product provided for embodiments of this application;

[0173] Figure 17 A schematic diagram of the structure of a training device for a control strategy provided in an embodiment of this application;

[0174] Figure 18 A schematic diagram of the structure of a training device for a control strategy provided in another embodiment of this application. Detailed Implementation

[0175] First, the application scenarios and some terms involved in the embodiments of this application will be explained.

[0176] The automatic lane-changing method, apparatus, and storage medium provided in this application can be applied to lane-changing scenarios of autonomous vehicles. For example, the automatic lane-changing method, apparatus, and storage medium provided in this application can be applied to scenarios A and B, which are briefly described below.

[0177] A scene:

[0178] When autonomous vehicles are driving in multi-lane traffic, they need to send "go straight" or "change lanes" commands at appropriate times to increase their speed. For example, if there is a vehicle traveling at low speed ahead in the lane where the autonomous vehicle is located, the autonomous vehicle needs to go around that vehicle.

[0179] Scenario B:

[0180] When an autonomous vehicle is driving in a multi-lane road, if there is a ramp or intersection ahead of its lane, the vehicle needs to switch to the appropriate target lane before reaching the ramp or intersection to complete its mission. For example, if an autonomous vehicle is driving in the leftmost lane and there is an intersection 500 meters ahead, it needs to turn right at the intersection and switch to the rightmost lane before reaching the intersection to reach its destination.

[0181] Of course, the automatic lane changing method, apparatus and storage medium provided in the embodiments of this application can also be applied to other scenarios, and the embodiments of this application do not limit this.

[0182] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 1 As shown, the system architecture 1000 provided in this application embodiment may include a training device 1001 and an execution device 1002. The training device 1001 is used to train a control strategy according to the control strategy training method provided in this application embodiment; the execution device 1002 is used to determine a target action using the control strategy trained by the training device 1001 according to the automatic lane changing method provided in this application embodiment; of course, the execution device 1002 may also be used to train the control strategy in real time, or to train the control strategy at preset intervals.

[0183] In this embodiment of the application, the entity executing the training method for the control strategy can be the training device 1001 described above, or the control strategy training apparatus within the training device 1001. Exemplarily, the control strategy training apparatus provided in this embodiment can be implemented through software and / or hardware.

[0184] In this embodiment of the application, the executing entity for the automatic lane changing method can be the aforementioned execution device 1002, or the automatic lane changing device within the aforementioned execution device 1002. Exemplarily, the automatic lane changing device provided in this embodiment can be implemented through software and / or hardware.

[0185] For example, the training device 1001 provided in this application embodiment may include, but is not limited to, a model training platform device.

[0186] For example, the execution device 1002 provided in the embodiments of this application may include, but is not limited to, an autonomous vehicle or a control device in an autonomous vehicle.

[0187] Figure 2 This is a functional block diagram of a vehicle 100 provided in an embodiment of this application. In one embodiment, the vehicle 100 is configured in a fully or partially automated driving mode. For example, when the vehicle 100 is configured in a partially automated driving mode, the vehicle 100 can also determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of that other vehicle performing the possible behavior, and control the vehicle 100 based on the determined information. When the vehicle 100 is in the automated driving mode, the vehicle 100 can be set to operate without human interaction.

[0188] Vehicle 100 may include various subsystems, such as a mobility system 102, a sensing system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0189] The propulsion system 102 may include components that provide powered motion to the vehicle 100. In one embodiment, the propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121. The engine 118 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 118 converts the energy source 119 into mechanical energy.

[0190] Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 may also provide energy to other systems of vehicle 100.

[0191] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0192] The sensing system 104 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the sensing system 104 may include a positioning system 122 (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. The sensing system 104 may also include sensors for the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the autonomous vehicle 100.

[0193] The positioning system 122 can be used to estimate the geographic location of the vehicle 100. The IMU 124 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope.

[0194] Radar 126 can use radio signals to sense objects in the surrounding environment of vehicle 100. In some embodiments, in addition to sensing objects, radar 126 can also be used to sense the speed and / or direction of travel of objects.

[0195] The laser rangefinder 128 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0196] Camera 130 can be used to capture multiple images of the surrounding environment of vehicle 100. Camera 130 can be a still camera or a video camera.

[0197] The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 may include various elements, including a steering system 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

[0198] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one embodiment, it may be a steering wheel system.

[0199] Throttle 134 is used to control the operating speed of engine 118 and thus the speed of vehicle 100.

[0200] Braking unit 136 is used to control the deceleration of vehicle 100. Braking unit 136 can use friction to slow down wheel 121. In other embodiments, braking unit 136 can convert the kinetic energy of wheel 121 into electric current. Braking unit 136 may also take other forms to slow down the rotational speed of wheel 121 to control the speed of vehicle 100.

[0201] Computer vision system 140 is operable to process and analyze images captured by camera 130 to identify objects and / or features in the environment surrounding vehicle 100. The objects and / or features may include traffic signals, road boundaries, and obstacles. Computer vision system 140 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, computer vision system 140 may be used to map the environment, track objects, estimate object velocities, and so on.

[0202] The route control system 142 is used to determine the driving route of the vehicle 100. In some embodiments, the route control system 142 may combine data from sensor fusion algorithm 138, global positioning system (GPS) 122 and one or more predetermined maps to determine the driving route of the vehicle 100.

[0203] The obstacle avoidance system 144 is used to identify, assess, and avoid or otherwise traverse potential obstacles in the environment of the vehicle 100.

[0204] Of course, in one instance, the control system 106 may include additional or alternative components besides those shown and described. Alternatively, some of the components shown above may be reduced.

[0205] Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, an on-board computer 148, a microphone 150, and / or a speaker 152.

[0206] In some embodiments, peripheral device 108 provides a means for a user of vehicle 100 to interact with user interface 116. For example, on-board computer 148 may provide information to a user of vehicle 100. User interface 116 may also operate on-board computer 148 to receive user input. On-board computer 148 may be operated via a touchscreen. In other cases, peripheral device 108 may provide a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from a user of vehicle 100. Similarly, speaker 152 may output audio to a user of vehicle 100.

[0207] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as code division multiple access (CDMA), EVDO, Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 146 can communicate with a wireless local area network (WLAN) using wireless-fidelity (WiFi). In some embodiments, the wireless communication system 146 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are also possible. For example, the wireless communication system 146 may include one or more dedicated shortrange communications (DSRC) devices that can enable public and / or private data communication between vehicles and / or roadside stations.

[0208] Power source 110 can provide power to various components of vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, as is the case in some fully electric vehicles.

[0209] Some or all of the functions of vehicle 100 are controlled by computer system 112. Computer system 112 may include at least one processor 113, which executes instructions 115 stored in a non-transitory computer-readable medium such as data storage device 114. Computer system 112 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0210] Processor 113 can be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor can be a special-purpose device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Those skilled in the art will understand that the processor, computer, or memory can actually include multiple processors, computers, or memories that may or may not be stored in the same physical housing. For example, memory can be a hard disk drive or other storage media located in a housing different from that of a computer. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as the steering assembly and deceleration assembly, may each have their own processor that performs calculations only related to the component's specific function.

[0211] In the various aspects described herein, the processor may be located remotely from the vehicle and communicate wirelessly with the vehicle. In other aspects, some of the processes described herein are executed on a processor located within the vehicle, while others are executed by a remote processor, including taking the necessary steps to perform a single operation.

[0212] In some embodiments, the data storage device 114 may include instructions 115 (e.g., program logic) that can be executed by the processor 113 to perform various functions of the vehicle 100, including those described above. The data storage device 114 may also include additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the mobility system 102, sensing system 104, control system 106, and peripheral devices 108.

[0213] In addition to instruction 115, data storage device 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information can be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0214] User interface 116 is used to provide information to or receive information from users of vehicle 100. Optionally, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, on-board computer 148, microphone 150, and speaker 152.

[0215] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., driving system 102, sensing system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from control system 106 to control steering system 132 to avoid obstacles detected by sensing system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of vehicle 100 and its subsystems.

[0216] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, data storage device 114 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0217] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 2 This should not be construed as a limitation on the embodiments of this application.

[0218] Autonomous vehicles traveling on roads, such as vehicle 100 above, can identify objects in their surrounding environment to determine how to adjust their current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified obstacle can be considered independently, and the speed adjustment to be made by the autonomous vehicle (autonomous vehicle) can be determined based on the characteristics of each obstacle, such as its current speed, acceleration, and distance from the vehicle.

[0219] Optionally, the autonomous vehicle 100 or the computing device associated with the autonomous vehicle 100 (such as...) Figure 2The computer system 112, computer vision system 140, and data storage device 114 can predict the behavior of the identified obstacles based on the characteristics of the identified obstacles and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified obstacle depends on the behavior of the others, so all identified obstacles can also be considered together to predict the behavior of a single identified obstacle. The vehicle 100 can adjust its speed based on the predicted behavior of the identified obstacles. In other words, the autonomous vehicle can determine what state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the obstacles. In this process, other factors can also be considered in determining the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road, the curvature of the road, the proximity of static and dynamic objects, etc.

[0220] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of the vehicle 100 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from obstacles near the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).

[0221] The aforementioned vehicle 100 can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, and handcart, etc., and the embodiments of the present invention do not impose any particular limitation.

[0222] Figure 3 for Figure 2 A schematic diagram of the structure of computer system 112. (See diagram below.) Figure 3As shown, computer system 112 includes a processor 113, which is coupled to a system bus 105. The processor 113 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to the system bus 105. The system bus 105 is coupled to an input / output (I / O) bus via a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with various I / O devices, such as input devices 117 (e.g., keyboard, mouse, touchscreen), media trays 121a (e.g., CD-ROM, multimedia interfaces), transceivers 123 (which can send and / or receive radio communication signals), a camera 155 (which can capture still and moving digital video images), and an external USB port 125. Optionally, the interface connected to the I / O interface 115 can be a universal serial bus (USB) interface.

[0223] The processor 113 can be any conventional processor, including a Reduced Instruction Set Computing (“RISC”) processor, a Complex Instruction Set Computing (“CISC”) processor, or a combination thereof. Optionally, the processor can be a special-purpose device such as an Application-Specific Integrated Circuit (“ASIC”). Optionally, the processor 113 can be a neural network processor or a combination of a neural network processor and the conventional processors described above.

[0224] Alternatively, in the various embodiments described herein, the computer system may be located remotely from the autonomous vehicle and may communicate wirelessly with the autonomous vehicle. In other aspects, some processes described herein are executed on a processor located within the autonomous vehicle, while others are executed by a remote processor, including taking actions necessary to perform a single manipulation.

[0225] Computer system 112 can communicate with software deployment server 149 via network interface 129. Network interface 129 is a hardware network interface, such as a network interface card (NIC). Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a Virtual Private Network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network or a cellular network.

[0226] Hard disk drive interface 131 is coupled to system bus 105. Hard disk drive interface 131 is connected to hard disk drive 133. System memory 135 is coupled to system bus 105. Software running in system memory 135 may include operating system (OS) 137 and application programs 143 of computer system 112.

[0227] An operating system consists of a shell (139) and a kernel (141). The shell (139) is an interface between the user and the operating system kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: it waits for user input, interprets the user input for the operating system, and processes various operating system outputs.

[0228] The kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel 141 typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0229] Application 143 includes programs related to controlling autonomous driving, such as programs managing the interaction between the autonomous vehicle and obstacles on the road, programs controlling the autonomous vehicle's route or speed, and programs controlling the interaction between the autonomous vehicle and other autonomous vehicles on the road. Application 143 also exists on the system of software deployment server 149. In one embodiment, when application 143 needs to be executed, the computer system can download application 143 from deployment server 149.

[0230] Sensor 153 is associated with the computer system. Sensor 153 is used to detect the environment surrounding the computer system 112. For example, sensor 153 can detect animals, cars, obstacles, and pedestrian crossings. Furthermore, the sensor can also detect the environment around these objects, such as the environment around the animal (e.g., other animals nearby), weather conditions, and ambient light levels. Optionally, if the computer system 112 is located on an autonomous vehicle, the sensor can be a camera, infrared sensor, chemical detector, microphone, etc.

[0231] Figure 4 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application. Figure 4 As shown, the chip may include a neural network processor 30. This chip can be configured as follows: Figure 1The execution device 1002 shown is used to complete the automatic lane changing method provided in the application embodiment. This chip can also be set in, for example... Figure 1 The training device 1001 shown is used to complete the training method for the control strategy provided in the application embodiment.

[0232] The neural network processor 30 can be any processor suitable for large-scale XOR operations, such as an NPU, TPU, or GPU. Taking an NPU as an example: the NPU can be mounted as a coprocessor on the host CPU, which then assigns tasks to it. The core of the NPU is the arithmetic circuit 303, which is controlled by the controller 304 to retrieve matrix data from the memory (301 and 302) and perform multiplication and addition operations.

[0233] In some implementations, the arithmetic circuit 303 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0234] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 303 retrieves the weight data of matrix B from the weight memory 302 and caches it in each PE (Engineer Component) of the arithmetic circuit 303. The arithmetic circuit 303 retrieves the input data of matrix A from the input memory 301, performs matrix operations based on the input data of matrix A and the weight data of matrix B, and stores the partial or final result of the obtained matrix in the accumulator 308.

[0235] Unified memory 306 is used to store input and output data. Weight data is directly transferred to weight memory 302 via direct memory access controller (DMAC) 305. Input data is also transferred to unified memory 306 via DMAC.

[0236] The bus interface unit (BIU) 310 is used for interaction between the DMAC and the instruction fetch unit 309; the bus interface unit 310 is also used by the instruction fetch unit 309 to fetch instructions from external memory; the bus interface unit 310 is also used by the memory access controller 305 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0237] The DMAC is mainly used to move input data from external memory DDR to unified memory 306, or to weight data to weight memory 302, or to input data to input memory 301.

[0238] The vector computation unit 307 comprises multiple processing units that, when necessary, further process the output of the computation circuit 303, such as vector multiplication, vector addition, exponential operations, logarithmic operations, and size comparisons. The vector computation unit 307 is primarily used for computations in non-convolutional layers or fully connected layers (FC) of neural networks, specifically handling calculations such as pooling and normalization. For example, the vector computation unit 307 can apply nonlinear functions to the output of the computation circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 307 generates normalized values, merged values, or both.

[0239] In some implementations, the vector computation unit 307 stores the processed vector in the unified memory 306. In some implementations, the vector processed by the vector computation unit 307 can be used as the activation input of the arithmetic circuit 303.

[0240] The instruction fetch buffer 309 connected to the controller 304 is used to store the instructions used by the controller 304;

[0241] Unified memory 306, input memory 301, weighted memory 302, and instruction fetch memory 309 are all on-chip memories. External memory is independent of this NPU hardware architecture.

[0242] Figure 5 This is a schematic diagram of the operating environment provided for embodiments of this application. For example... Figure 5 As shown, the cloud service center can receive information (such as data collected by vehicle sensors or other information) from autonomous vehicles 510 and 512 within its operating environment 500 via network 502 (such as a wireless communication network).

[0243] For example, the cloud service center 520 can receive driving information of the autonomous vehicle 510 at any time (such as driving speed and / or driving position) and driving information of other vehicles within the perception range of the autonomous vehicle 510 from the autonomous vehicle 510 via a network 502 (such as a wireless communication network).

[0244] Based on the received information, the cloud service center 520 can run its stored programs related to controlling autonomous driving, thereby controlling autonomous vehicles 510 and 512. These programs can include those managing the interaction between the autonomous vehicle and obstacles on the road, controlling the autonomous vehicle's route or speed, and controlling the interaction between the autonomous vehicle and other autonomous vehicles on the road.

[0245] Network 502 provides portions of the map to autonomous vehicles 510 and 512.

[0246] For example, multiple cloud service centers can receive, verify, combine, and / or send information reports. In some examples, information reports and / or sensor data can also be sent between autonomous vehicles.

[0247] In some examples, the cloud service center 520 can send suggested solutions (e.g., informing the driver of an obstacle ahead and how to avoid it) to the autonomous vehicle (or autonomous car) based on possible driving situations within the environment. For example, the cloud service center 520 can assist the vehicle in determining how to proceed when facing a specific obstacle in the environment. The cloud service center 520 can send responses to the autonomous vehicle instructing it on how to proceed in a given scenario. For example, based on collected sensor data, the cloud service center can confirm the presence of a temporary stop sign ahead of the road and, based on sensor data from a "lane closure" sign and construction vehicles, determine that the lane is closed due to construction. Accordingly, the cloud service center 520 can send suggested operating modes for the autonomous vehicle to navigate the obstacle (e.g., instructing the vehicle to change lanes onto another road). The cloud service center 520 can observe the video stream within its operating environment and, after confirming that the autonomous vehicle can safely and successfully navigate the obstacle, the operating steps used by the autonomous vehicle can be added to the driving information map. Accordingly, this information can be sent to other vehicles in the area that may encounter the same obstacle, so that other vehicles can not only identify the closed lane but also know how to pass through it.

[0248] It should be noted that autonomous vehicles 510 and / or 512 can control their own driving during operation, or they may not require the control of the cloud service center 520.

[0249] The local neighbor features at any given time in the embodiments of this application are used to represent the motion state information (e.g., relative distance and relative speed) of specific neighbor obstacles relative to the autonomous vehicle.

[0250] For example, a specific neighboring obstacle may include, but is not limited to, at least one of the following: obstacles in front of or behind the autonomous vehicle in the lane in which the autonomous vehicle is located, obstacles in front of or behind the autonomous vehicle in the adjacent left lane of the lane in which the autonomous vehicle is located, and obstacles in front of or behind the autonomous vehicle in the adjacent right lane of the lane in which the autonomous vehicle is located.

[0251] For example, when the autonomous vehicle is in the left lane, the motion state information of the obstacles in front and behind the autonomous vehicle in the adjacent left lane of the autonomous vehicle's lane relative to the autonomous vehicle can be a default value; and / or, when the autonomous vehicle is in the right lane, the motion state information of the obstacles in front and behind the autonomous vehicle in the adjacent right lane of the autonomous vehicle's lane relative to the autonomous vehicle can be a default value.

[0252] The obstacles involved in the embodiments of this application can be dynamically moving obstacles or static obstacles. For example, obstacles can include, but are not limited to, at least one of the following: autonomous vehicles, non-autonomous motor vehicles, people, and objects. For example, when a specific neighboring obstacle is a static obstacle, the relative distance of the specific neighboring obstacle to the autonomous vehicle can be the distance between the neighboring obstacle and the autonomous vehicle, and the relative speed of the specific neighboring obstacle to the autonomous vehicle can be the moving speed of the autonomous vehicle.

[0253] For example, the local neighbor features of an autonomous vehicle at any given time may include, but are not limited to, the relative speed of obstacles in front of the autonomous vehicle in its lane relative to the autonomous vehicle. and relative distance The relative speed of an obstacle adjacent to the autonomous vehicle in its lane relative to the autonomous vehicle. and relative distance The relative speed of the obstacle in front of the autonomous vehicle in the adjacent left lane of the autonomous vehicle's lane relative to the autonomous vehicle. and relative distance The relative speed of an obstacle adjacent to the autonomous vehicle in the left lane of the lane in which the autonomous vehicle is located, relative to the autonomous vehicle. and relative distance The relative speed of the obstacle in front of the autonomous vehicle in the adjacent right lane relative to the autonomous vehicle. and relative distance The relative speed of an obstacle adjacent to the autonomous vehicle in the right lane to the autonomous vehicle's lane relative to the autonomous vehicle. and relative distance

[0254] Optionally, the local neighbor features of the autonomous vehicle at any given time may also include: the position information flag between the navigation target lane and the lane where the autonomous vehicle is located, and the distance dist2goal between the autonomous vehicle and the next intersection along the driving direction; where flag∈{0, -1, 1}, flag equal to 0 means that the autonomous vehicle is in the navigation target lane, flag equal to -1 means that the navigation target lane is to the left of the lane where the autonomous vehicle is located, and flag equal to 1 means that the navigation target lane is to the right of the lane where the autonomous vehicle is located.

[0255] The global statistical features at any given time in the embodiments of this application are used to represent the sparseness and density of obstacles in each lane within the perception range (i.e., the range that the sensors of the autonomous vehicle can detect, such as the range within a preset interval from the autonomous vehicle).

[0256] For example, the global statistical features at any given time may include, but are not limited to, at least one of the following: the average speed and average spacing of all obstacles in each lane within the perception range. For instance, if the average spacing of all obstacles in a lane is less than a preset spacing, it indicates that the obstacles in that lane are relatively dense; if the average spacing of all obstacles in a lane is greater than or equal to the preset spacing, it indicates that the obstacles in that lane are relatively sparse. For instance, the global statistical features at any given time may include, but are not limited to, the average gap between two adjacent obstacles in all lanes to the left of the lane where the autonomous vehicle is located. L The average gap between two adjacent obstacles in the lane where the autonomous vehicle is located. M The average gap between two adjacent obstacles in all lanes to the right of the lane in which the autonomous vehicle is located. R The average speed V of obstacles in all lanes to the left of the lane in which the autonomous vehicle is located. L The average speed V of obstacles in the lane where the autonomous vehicle is located M The average speed V of obstacles in all lanes to the right of the lane in which the autonomous vehicle is located. R .

[0257] Optionally, the local neighbor features and global statistical features of the autonomous vehicle at any given time in the embodiments of this application can be discretized features, which can satisfy the requirements of small discrete granularity in low-speed dense scenes and large discrete granularity in high-speed sparse scenes, for example:

[0258] 1) When the speed V of the autonomous vehicle ego ≤V threshold(e.g., 20 km / h), then the precision of the local relative distance feature is 0.01, and the precision of the local relative velocity feature is 0.05. For example, if a local relative distance feature is 0.1123, then it becomes 0.11 after discretization; if a local relative velocity feature is 0.276, then it becomes 0.25 after discretization.

[0259] 2) When the speed V of the autonomous vehicle ego >V threshold The accuracy of the local relative distance feature is 0.05, and the accuracy of the local relative velocity feature is 0.1.

[0260] 3) The accuracy of the average interval feature is uniformly set to 0.01, and the accuracy of the average velocity feature is uniformly set to 0.01.

[0261] The target action indication involved in this application embodiment is used to instruct an autonomous vehicle to perform a target action. Exemplarily, the target action may include at least the following two categories: lane changing or maintaining straight driving, wherein lane changing may include: changing lanes to the left adjacent lane or changing lanes to the right adjacent lane.

[0262] In this application embodiment, the quadruple information (s, a, r, s′) at any given time corresponds to the vehicle condition at that time and may include: the feature s at that time, the target action a of the autonomous vehicle at that time, the reward value r corresponding to the target action at that time, and the feature s′ of the next time step; wherein, the feature s at that time may include: the local neighbor features s of the autonomous vehicle at that time. l and global statistical features s g The features s′ of the next moment can include: the local neighbor features s of the autonomous vehicle at that next moment. l ′ and global statistical features s g ′.

[0263] Figure 6 Symmetry principle illustration provided for embodiments of this application Figure 1 , Figure 7 Symmetry principle illustration provided for embodiments of this application Figure 2 ,like Figure 6 and Figure 7 As shown, the symmetry rule involved in this application embodiment refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane where the autonomous vehicle is located as the axis.

[0264] Figure 8 A schematic diagram of the monotonicity principle provided for embodiments of this application, such as... Figure 8As shown, the monotonicity principle involved in this application embodiment refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane when the autonomous vehicle changes lanes, and / or, changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lanes by less than a preset distance range. For example, if the next vehicle in the target lane is A, the next vehicle in front is D, the next vehicle in front is B, and the next vehicle in the non-target lane is C, then the monotonicity principle may include, but is not limited to, the following operations:

[0265] Operation 1: Vehicle A moves backward a preset distance of 1 or decreases its speed by a preset value of 1;

[0266] Operation 2: Vehicle D moves forward a preset distance of 2 or increases its speed by a preset value of 2;

[0267] Operation 3: Vehicle B moves forward or backward a preset distance of 3, or increases or decreases its speed by a preset value of 3;

[0268] Operation 4: Vehicle C moves forward or backward a preset distance of 4, or increases or decreases its speed by a preset value of 4.

[0269] In the embodiments of this application, the extended quadruplet information at any time corresponds to the extended vehicle condition at that time, which is obtained by processing the vehicle condition at that time with symmetry rules and monotonic rules.

[0270] For example, the extended quadruple information at any given time may include: the symmetric quadruple information at that time (s e a e ,r,s e ′) and monotonic quadruple information (s m a m ,r,sx′); where, at this moment, the symmetric quadruple information (s e a e ,r,s e The quadruple information (s, a, r, s′) at that moment is constructed according to the symmetry rule, where s′ is the quadruple information at that moment. e It is a symmetrical feature of s, a e It is the symmetrical action of a, s e ′ is a symmetric feature of s′; the monotonic quadruple information at this moment (s m a m ,r,s m The quadruple information (s, a, r, s′) at that moment is constructed according to the monotonicity rule, where s is the quadruple information at that moment. m It is a monotonic characteristic of s, a m It is a monotonic action of a (for example, a) m It can be equal to a), s m′ is a monotonic feature of s′.

[0271] The following parts in the embodiments of this application respectively address the symmetric quadruple information (s) at any given time. e a e ,r,s e ′) and monotonic quadruple information (s m a m ,r,s m The construction method of ′) will be introduced.

[0272] For example, 1) Suppose that at any given time, the quadruple information (s, a, r, s′) contains s. l It is equal to the following formula:

[0273]

[0274] Then, based on the principle of symmetry, the symmetric quadruple information (s) at that moment is determined. e a e ,r,s e s in ′) el It is equal to the following formula:

[0275]

[0276] 2) Assume that at any given time, the quadruple information (s, a, r, s′) contains s. g It is equal to the following formula:

[0277] s g =(gap L gap M gap R V L V M V R )

[0278] Then, based on the principle of symmetry, the symmetric quadruple information (s) at that moment is determined. e a e ,r,s e s in ′) eg It is equal to the following formula:

[0279] s eg =(gap R gap M gap V V R V M V L )

[0280] Therefore, according to s el and s eg Construct s e .

[0281] 3) Determine the symmetric quadruple information (s) at this moment according to the symmetry principle. e a e ,r,s e a in ′) e It is equal to the following formula:

[0282]

[0283] In this context, a = 0 means to stay straight, a = 1 means to change lanes to the left adjacent lane, and a = 2 means to change lanes to the right adjacent lane.

[0284] It should be noted that the symmetric quadruple information (s) at this moment e a e ,r,s e s in ′) e The construction method of ′ is the same as s e The construction method is similar, and will not be described in detail here.

[0285] For example, suppose the normalized relative distance features of vehicles A, B, C, and D are d... A d B d C d D The relative velocity characteristics, after normalization, are v A v B v C v D , and Δ d Δ represents the relative distance accuracy at the current speed. v Given the relative speed accuracy corresponding to the current vehicle angle, then (d) A v A d B v B d C v C d D v D The value of ) will be changed by the monotonic principle to the value in the following set containing 2*2*3*3*3*3*2*2 elements:

[0286] s ml ∈{d A -Δ d d A}×{v A -Δ v v A}×{d B -Δ d d B d B +Δ d}

[0287] ×{v B -Δ v v v v B +Δ v}×{d C -Δ d d C d C +Δ d}×{v C -Δ v v v v C +Δ v}×{d D -Δ d d D}×{v D -Δ v v D Therefore, a predetermined number (e.g., 10) groups are randomly selected from the above set to constitute the monotonic quadruplet information (s) at that moment. m a m ,r,s m s in ′) ml .

[0288] Optionally, assume that s in the quadruple information (s, a, r, s′) at any time... g Equal to the monotonic quadruple information (s) at that moment m a m ,r,s m s in ′) mg Then according to s ml and s mg Construct s m .

[0289] It should be noted that the monotonic quadruple information (s) at this moment m a m ,r,s m s in ′) m The construction method of ′ is the same as s m The construction method is similar, and will not be described in detail here.

[0290] It should be noted that when the training device 1001 executes the training method of the control strategy provided in this application embodiment, the information involved, such as the autonomous vehicle, obstacles, and lanes, can be the simulated road information in the training device 1001, or historical data occurring on the actual road. When the execution device 1002 executes the automatic lane changing method provided in this application embodiment, the information involved, such as the autonomous vehicle, obstacles, and lanes, is the actual real-time road information.

[0291] The method provided in this application is described below from the perspectives of both the training side (or control policy model) and the application side:

[0292] The control strategy training method provided in this application involves computer processing and can be applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (such as the preset number of historical time-based quadruple information in this application embodiment) to ultimately obtain a trained control strategy. Furthermore, the automatic lane-changing method provided in this application embodiment can utilize the trained control strategy, inputting input data (such as local neighbor features and global statistical features in this application embodiment) into the trained control strategy to obtain output data (such as the target action indication in this application embodiment). Of course, the automatic lane-changing method can also update the control strategy in real time or at preset intervals. It should be noted that the control strategy training method and application method provided in this application embodiment are inventions based on the same concept and can be understood as two parts of a system or two stages of an overall process: such as a control strategy training stage and a control strategy application stage.

[0293] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0294] The training method for the control strategy provided in the embodiments of this application will be described in detail below.

[0295] Figure 9 This is a flowchart illustrating a training method for a control strategy provided in an embodiment of this application. The method in this embodiment can specifically be as follows: Figure 1 The training device 1001 shown is used for execution. Figure 9 As shown, the method provided in this application embodiment may include:

[0296] Step S901: Obtain the quadruple information of a preset number of historical moments.

[0297] In this step, a preset number of four-tuple information for historical moments are obtained from the database. The four-tuple information for any historical moment corresponds to the vehicle condition at that historical moment, and may include, but is not limited to: the characteristics of that historical moment, the target action of the autonomous vehicle at that historical moment (i.e., the target action determined at that historical moment according to the corresponding control strategy), the reward value corresponding to the target action at that historical moment, and the characteristics of the next moment of that historical moment.

[0298] For example, the features of this historical moment may include, but are not limited to: the local neighbor features and global statistical features of the autonomous vehicle at that historical moment.

[0299] For example, the features of the next moment in the historical moment may include, but are not limited to: the local neighbor features and global statistical features of the autonomous vehicle at that next moment.

[0300] The local neighbor features of the autonomous vehicle at any given time in the embodiments of this application are used to represent the motion state information (e.g., relative distance and relative speed) of a specific neighbor obstacle relative to the autonomous vehicle at that time.

[0301] For example, specific neighboring obstacles may include, but are not limited to: obstacles in front of and behind the autonomous vehicle in the lane where the autonomous vehicle is located at that moment, obstacles in front of and behind the autonomous vehicle in the adjacent left lane of the lane where the autonomous vehicle is located at that moment, and obstacles in front of and behind the autonomous vehicle in the adjacent right lane of the lane where the autonomous vehicle is located at that moment.

[0302] The global statistical features of the autonomous vehicle at any time in the embodiments of this application are used to represent the sparsity and density of obstacles in each lane within the perception range of the autonomous vehicle at that time, such as the average driving speed and average interval of all obstacles in each lane at that time.

[0303] For example, the global statistical characteristics at any given time may include, but are not limited to: the average interval between two adjacent obstacles in all lanes to the left of the lane where the autonomous vehicle is located at that time, the average interval between two adjacent obstacles in the lane where the autonomous vehicle is located at that time, the average interval between two adjacent obstacles in all lanes to the right of the lane where the autonomous vehicle is located at that time, the average speed of obstacles in all lanes to the left of the lane where the autonomous vehicle is located at that time, the average speed of obstacles in the lane where the autonomous vehicle is located at that time, and the average speed of obstacles in all lanes to the right of the lane where the autonomous vehicle is located at that time.

[0304] Step S902: Update the current control policy to obtain the control policy for the next moment based on at least one quadruple information of the first historical moment, at least one extended quadruple information of the first historical moment, and at least one quadruple information of the second historical moment.

[0305] For example, at least one first historical moment's quadruple information is the quadruple information of the historical moment corresponding to the target action of the autonomous vehicle at the aforementioned preset number of historical moment quadruple information, where the target action is lane change.

[0306] For example, at least one second historical moment's quadruple information is the quadruple information of other historical moments among the aforementioned preset number of historical moment quadruple information, excluding at least one first historical moment's quadruple information, that is, the quadruple information of the historical moment corresponding to the target action of the autonomous vehicle at the preset number of historical moment quadruple information being maintained.

[0307] For example, assuming that the quadruple information of a preset number of historical moments may include: quadruple information of historical moment ① (where the target action of the autonomous vehicle at historical moment ① is to change lanes), quadruple information of historical moment ② (where the target action of the autonomous vehicle at historical moment ② is to keep going straight), quadruple information of historical moment ③ (where the target action of the autonomous vehicle at historical moment ③ is to change lanes), and quadruple information of historical moment ④ (where the target action of the autonomous vehicle at historical moment ④ is to keep going straight), then at least one first historical moment quadruple information may include: quadruple information of historical moment ① and quadruple information of historical moment ③, and at least one second historical moment quadruple information may include: quadruple information of historical moment ② and quadruple information of historical moment ④.

[0308] In the embodiments of this application, the extended quadruple information of any first historical moment corresponds to the extended vehicle condition of the first historical moment, which is obtained by processing the vehicle condition of the first historical moment with symmetric rules and monotonic rules.

[0309] The symmetry rule involved in this application embodiment refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane where the autonomous vehicle is located as the axis.

[0310] The monotonicity principle involved in this application embodiment refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane where the autonomous vehicle changes lanes, and / or changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lanes by less than a preset distance range.

[0311] For example, the extended quadruple information at any first historical moment may include: symmetric quadruple information and monotonic quadruple information at that first historical moment. For instance, the symmetric quadruple information at that first historical moment may be obtained by applying the symmetry principle to the quadruple information at that first historical moment, and the monotonic quadruple information at that first historical moment may be obtained by applying the monotonic principle to the quadruple information at that first historical moment.

[0312] Specifically, the construction method of the symmetric quadruplet information and monotonic quadruplet information at the first historical moment can refer to the construction method of "symmetric quadruplet information and monotonic quadruplet information at any time" mentioned above in this application, and will not be repeated here.

[0313] In this step, the parameters in the current control strategy are updated based on at least one first historical moment's quadruple information, at least one first historical moment's extended quadruple information, and at least one second historical moment's quadruple information to obtain the control strategy for the next moment (used to determine the target action for the next moment).

[0314] For example, a target value corresponding to the l-th quaternion information is generated based on the quaternion information of at least one first historical moment, the extended quaternion information of at least one first historical moment, and the l-th quaternion information of at least one second historical moment; further, the parameter q in the preset function containing the target value corresponding to the l-th quaternion information is iteratively updated using the gradient descent method; further, the parameter q in the current control strategy is replaced by the iteratively updated parameter q to obtain the control strategy at the next moment.

[0315] In this embodiment, based on at least one quadruple information from a first historical moment, at least one extended quadruple information from a first historical moment, and the l-th quadruple information from at least one quadruple information from a second historical moment (s... l a l r l s l The target value y corresponding to the l-th quadruple can be generated using the following formula. l Where l is a positive integer not greater than q, and q is the total number of quadruples included in at least one first historical moment, at least one extended quadruple from the first historical moment, and at least one quadruple from the second historical moment.

[0316] For example,

[0317] Wherein, the final state refers to the autonomous vehicle automatically completing the preset maximum distance or the autonomous vehicle's driving being manually intervened; g represents the preset forgetting factor, g∈(0,1); Q(s l ′, a l q) represents the action value function; Represents traversing a l Make Q(s) l ′, a l ,q) takes the maximum value; s l ′ represents the feature of the next time step in the information of the l-th quadruple.

[0318] Of course, based on the quaternion information of at least one first historical moment, the extended quaternion information of at least one first historical moment, and the quaternion information of at least one second historical moment, the target value corresponding to the quaternion information can also be generated through other variations or equivalent formulas of the above formula. This application does not impose any restrictions on this.

[0319] Furthermore, the gradient descent method is used to calculate the target value y corresponding to the information of the l-th quadruple. l Preset functions The parameter q in the equation is updated iteratively; where Q(s) l a l ,q) is the action value function corresponding to the l-th quadruple information, s l a represents the feature of the previous time step in the information of the l-th quadruple. l This represents the target action in the previous time step within the l-th quadruple information.

[0320] Furthermore, the updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next moment, so as to determine the target action for the next moment.

[0321] Of course, based on at least one first historical moment's quadruple information, at least one first historical moment's extended quadruple information, and at least one second historical moment's quadruple information, the parameters in the current control strategy can be updated in other ways to obtain the control strategy for the next moment. This application embodiment does not impose any restrictions on this.

[0322] In this embodiment, the training device 1001 can repeatedly execute steps S901-S902 a preset number of times, or repeatedly execute steps S901-S902 multiple times until the updated control strategy meets the preset conditions. The control strategy finally obtained by the training device 1001 can be used by the execution device 1002 when executing the automatic lane changing method.

[0323] For example, when the training device 1001 executes step S902 for the first time, the current control strategy of this application embodiment may be a preset initial control strategy; when the training device 1001 does not execute step S902 for the first time, the current control strategy of this application embodiment may be the control strategy obtained after the training device 1001 last executed step S902.

[0324] In this embodiment, a preset number of historical time-stamped quadruple information are obtained. Further, the control strategy for the next time-stamp is updated based on at least one first historical time-stamped quadruple information, at least one extended quadruple information from the first historical time-stamped quadruple information, and at least one second historical time-stamped quadruple information. It is evident that by updating the current control strategy based on the preset number of historical time-stamped quadruple information and then further updating it according to the extended quadruple information from the first historical time-stamped quadruple information, a more accurate control strategy can be obtained, enabling precise determination of the corresponding target action.

[0325] Figure 10 This is a flowchart illustrating a training method for a control strategy provided in another embodiment of this application. Based on the above embodiments, this application describes the method for generating "four-tuple information of historical moments." For example... Figure 10 As shown, before step S901 above, the procedure further includes:

[0326] S1001. For each historical moment, based on the driving information of the autonomous vehicle and the movement information of obstacles in each lane within the perception range of the autonomous vehicle, calculate the local neighbor features and global statistical features of the autonomous vehicle at that historical moment.

[0327] It should be noted that when the obstacle is a vehicle or other mobile terminal, the motion information of the obstacle is driving information; when the obstacle is a person, animal or stationary object, the motion information of the obstacle may include relevant information such as speed and position.

[0328] In this step, for each historical moment, based on the autonomous vehicle's driving information (such as driving speed and / or driving position) and the motion information of obstacles in each lane within the autonomous vehicle's perception range (i.e., the range that the autonomous vehicle's sensors can detect, such as the range within a preset interval from the autonomous vehicle), the local neighbor features and global statistical features of the autonomous vehicle at that historical moment are calculated.

[0329] The local neighbor features of the autonomous vehicle at any historical moment in the embodiments of this application are used to represent the motion state information (e.g., relative distance and relative speed) of specific neighbor vehicles (such as obstacles in front and behind the autonomous vehicle in the lane where the autonomous vehicle is located, obstacles in front and behind the autonomous vehicle in the adjacent left lane of the lane where the autonomous vehicle is located, and obstacles in front and behind the autonomous vehicle in the adjacent right lane of the lane where the autonomous vehicle is located) relative to the autonomous vehicle.

[0330] For example, the local neighbor features s of an autonomous vehicle at any given time l This may include, but is not limited to: the relative speed of obstacles in front of the autonomous vehicle in its lane relative to the autonomous vehicle. and relative distance The relative speed of an obstacle adjacent to the autonomous vehicle in its lane relative to the autonomous vehicle. and relative distance The relative speed of the obstacle in front of the autonomous vehicle in the adjacent left lane of the autonomous vehicle's lane relative to the autonomous vehicle. and relative distance The relative speed of an obstacle adjacent to the autonomous vehicle in the left lane of the lane in which the autonomous vehicle is located, relative to the autonomous vehicle. and relative distance The relative speed of the obstacle in front of the autonomous vehicle in the adjacent right lane to the autonomous vehicle's lane relative to the autonomous vehicle. and relative distance The relative speed of an obstacle adjacent to the autonomous vehicle in the right lane to the autonomous vehicle's lane relative to the autonomous vehicle. and relative distance

[0331] The global statistical features of the autonomous vehicle at any historical moment in the embodiments of this application are used to represent the sparsity and density of obstacles in each lane within the perception range of the autonomous vehicle, such as the average driving speed and average interval of all obstacles in each lane at that historical moment.

[0332] For example, the global traffic flow statistics of autonomous vehicles at any given time s g This may include, but is not limited to: the average gap between obstacles in front of and behind in all lanes to the left of the lane in which the autonomous vehicle is located. L The average gap between two adjacent obstacles in the lane where the autonomous vehicle is located. M The average gap between two adjacent obstacles in all lanes to the right of the lane in which the autonomous vehicle is located. R The average speed V of obstacles in all lanes to the left of the lane in which the autonomous vehicle is located. L The average speed V of obstacles in the lane where the autonomous vehicle is located M The average speed V of obstacles in all lanes to the right of the lane in which the autonomous vehicle is located. R .

[0333] S1002. Obtain the target action instruction for the historical moment based on the local neighbor characteristics, global statistical characteristics, and control strategy of the historical moment.

[0334] In this step, by inputting the local neighbor features and global statistical features of the historical moment into the control policy of the historical moment, the target action instruction of the historical moment can be obtained (used to instruct the autonomous vehicle to perform the target action).

[0335] For example, the control policy at any given time (e.g., the control policy at this historical moment) can be expressed as:

[0336]

[0337] Where s represents the local neighbor features and global statistical features at that moment; a′∈(0,1,2), a′ equal to 0 means to keep going straight, a′ equal to 1 means to change lanes to the left adjacent lane, and a′ equal to 2 means to change lanes to the right adjacent lane.

[0338] In this embodiment, for any historical moment, based on the local neighbor features and global statistical features, the action a′ that maximizes the action value function Q(s, a′, q) is selected as the target action a at that historical moment.

[0339] Of course, the control strategy for this historical moment can also adopt other variations or equivalent formulas of the above formula, and this application embodiment does not limit this.

[0340] For example, the target action includes at least two types: changing lanes or keeping straight, wherein changing lanes may include changing lanes to the left adjacent lane or changing lanes to the right adjacent lane.

[0341] S1003. Obtain feedback information by performing the target action.

[0342] For example, the feedback information may include, but is not limited to: driving information (such as driving speed or driving position) after the autonomous vehicle performs the target action, driving information of the autonomous vehicle at the next moment, and movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment; when the target action is lane changing, the feedback information may also include: the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after lane changing compared to the lane it was in before lane changing; wherein, the historical average time includes the average time of the autonomous vehicle performing the same type of action (such as lane changing) within a preset historical time period (e.g., a 500-time window).

[0343] For example, the change in the sparseness and density of the lane after the autonomous vehicle changes lanes compared to the lane before the lane change can be based on the driving information of the autonomous vehicle and other obstacles within its perception range before and after the lane change (e.g., the average gap between two adjacent obstacles in the lane before the lane change). cur The average speed V of the obstacles in the lane in which the autonomous vehicle was located before changing lanes. cur The average gap between two adjacent obstacles in the lane where an autonomous vehicle changes lanes. goal The average speed V of obstacles in the lane where the autonomous vehicle is located after changing lanes. goal ), and determined by the preset global classification model F0.

[0344] S1004. Calculate the reward value corresponding to the target action based on the feedback information, as well as the local neighbor features and global statistical features of the autonomous vehicle at the next moment of this historical moment.

[0345] In this step, based on the autonomous vehicle's driving information at the next moment and the movement information of obstacles in each lane within the autonomous vehicle's perception range at the next moment, the local neighbor features and global statistical features of the autonomous vehicle at the next historical moment can be calculated. The specific calculation method can refer to the above-described method for obtaining the local neighbor features and global statistical features of the autonomous vehicle at that historical moment, which will not be repeated in this embodiment.

[0346] The following sections of the embodiments of this application describe the possible implementation of "calculating the reward value corresponding to the target action based on feedback information".

[0347] One possible implementation is to calculate the reward value based on the driving information after the autonomous vehicle performs the target action, when the target action is to maintain straight-line driving.

[0348] For example, the reward value is calculated based on a preset function R(s″) and the driving information s″ (such as driving speed or driving position) after the autonomous vehicle performs the target action. For example, the preset function R(s″) = V ego ′, V ego ′ represents the speed of the autonomous vehicle after performing the target action; of course, the preset function R(s″) can also be equal to other functions that include the driving information of the autonomous vehicle after performing the target action, and this application embodiment does not limit this.

[0349] Another possible approach is to calculate the reward value based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane the autonomous vehicle is in after the lane change compared to the lane it was in before the lane change.

[0350] In this implementation, the time T for executing the target action is compared with the historical average time T. e The ratio determines the local return coefficient K. l Furthermore, the global reward coefficient K is determined based on the changes in the sparsity and density of the lane the autonomous vehicle is in after changing lanes compared to its original lane. g Where, when the lane where the autonomous vehicle is located after changing lanes is denser than the lane where the autonomous vehicle was located before changing lanes, K g >1; When the lane where the autonomous vehicle is in after changing lanes is less sparse than the lane where the autonomous vehicle was in before changing lanes, K g <1. Furthermore, based on the driving information of the autonomous vehicle after performing the target action and the local reward coefficient K... l and global return coefficient K g Calculate the return value.

[0351] For example, according to the formula c*K l *K g *R(s″) calculates the reward value; where c represents a preset discount factor (e.g., 0.3), and R(s″) represents a preset function containing driving information after the autonomous vehicle performs the target action; of course, the reward value can also be calculated through other equivalent or modified formulas of the above formula, and this application embodiment does not limit this.

[0352] If the local neighbor features of the autonomous vehicle at this historical moment can also include: the position information flag between the navigation target lane and the lane where the autonomous vehicle is located, and the distance dist2goal between the autonomous vehicle and the next intersection along the driving direction; where flag∈{0, -1, 1}, where flag equals 0 represents that the autonomous vehicle is on the navigation target lane, flag equals -1 represents that the navigation target lane is to the left of the lane where the autonomous vehicle is located, and flag equals 1 represents that the navigation target lane is to the right of the lane where the autonomous vehicle is located, then based on the time T for executing the target action and the historical average time T e The ratio determines the local return coefficient K. l Furthermore, the first global reward coefficient is determined based on the changes in the sparsity and density of the lane the autonomous vehicle is in after changing lanes compared to its original lane.

[0353] Furthermore, a second global reward coefficient is determined based on the positional information (flag) between the navigation target lane and the lane where the autonomous vehicle is located, and the target action. For example, the second global return coefficient is determined according to the following formula.

[0354]

[0355] Among them, gap cur The gap represents the average distance between two adjacent obstacles in the lane in which the autonomous vehicle is located before changing lanes. goal This represents the average distance between two adjacent obstacles in the lane where the autonomous vehicle is located after changing lanes, and 'a' represents the target action.

[0356] Of course, the second global return coefficient can also be calculated using other equivalent or modified formulas of the above formula. This application does not impose any limitations on this aspect.

[0357] Furthermore, based on the driving information of the autonomous vehicle after performing the target action and the local reward coefficient K... l First global return coefficient Second global return coefficient Calculate the return value.

[0358] For example, according to the formula Calculate the return value; where c represents a preset discount factor (e.g., 0.3), and R(s″) represents a preset function containing driving information after the autonomous vehicle performs the target action.

[0359] For example, preset functions V ego ' represents the speed of the autonomous vehicle after performing the target action; flag' represents the position information between the navigation target lane and the lane where the autonomous vehicle is located after performing the target action; flag' equal to 0 means the autonomous vehicle is on the navigation target lane after performing the target action; flag' equal to -1 means the navigation target lane is to the left of the lane where the autonomous vehicle is located after performing the target action; flag' equal to 1 means the navigation target lane is to the right of the lane where the autonomous vehicle is located after performing the target action; dist2goal' represents the distance between the autonomous vehicle and the next intersection along the driving direction after performing the target action; of course, the preset function R(s″) can also be equal to other functions that include the driving information of the autonomous vehicle after performing the target action, and this application embodiment does not impose any restrictions on this.

[0360] Of course, the return value can also be calculated using other equivalent or modified formulas of the above formula, and this application embodiment does not limit this.

[0361] Of course, the reward value corresponding to the target action can also be calculated in other ways based on the feedback information, and this application embodiment does not limit this.

[0362] S1005. Store the quadruple information for this historical moment.

[0363] In this step, the four-tuple information of this historical moment can be stored in the database to facilitate subsequent training of the control strategy. For example, the four-tuple information of this historical moment, corresponding to the vehicle condition at that historical moment, can include: the features of this historical moment, the target action of the autonomous vehicle at this historical moment, the reward value corresponding to the target action at this historical moment, and the features of the next moment. The features of this historical moment include the local neighbor features and global statistical features of the autonomous vehicle at this historical moment, and the features of the next moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment.

[0364] In this embodiment, for each historical moment, the target action instruction for that historical moment is obtained based on the local neighbor features, global statistical features, and control policy of the autonomous vehicle at that historical moment. Further, feedback information is obtained by executing the target action, and the reward value corresponding to the target action, as well as the local neighbor features and global statistical features of the autonomous vehicle at the next moment in that historical moment, are calculated based on the feedback information, and the four-tuple information for that historical moment is stored. It is evident that by further introducing global statistical features and the reward value corresponding to the target action on top of the local neighbor features, the training data used to train the control policy is more complete, thereby facilitating the training of a more accurate control policy.

[0365] Furthermore, based on the above embodiments, this application describes the generation method of the "preset global classification model F0".

[0366] For example, global classification model features are obtained; wherein, global classification model features may include, but are not limited to: motion information of autonomous vehicles and obstacles in each lane within the perception range of autonomous vehicles within a preset time period (e.g., 2,000,000 time windows) before and after each lane change of autonomous vehicles (e.g., the average gap between two adjacent obstacles in the lane in which the autonomous vehicle is located before each lane change). cur The average speed V of obstacles in the lane in which the autonomous vehicle is located before each lane change. cur The average gap between two adjacent obstacles in the lane after each lane change for an autonomous vehicle. goal The average speed V of the obstacles in the lane after each lane change by the autonomous vehicle goalFurthermore, based on the features of the global classification model, a logistic regression algorithm is used to generate a preset global classification model F0.

[0367] In this embodiment, different sparse and dense road scenarios and / or different speeds can be preset in the simulator. For example, a map with three lanes and a preset length (e.g., 4km) is constructed as a training map, and the arrangement of social vehicles (i.e., other vehicles besides autonomous vehicles) covers scenarios such as no-car scenario, sparse medium-speed scenario, sparse high-speed scenario, sparse low-speed scenario, uniform medium-speed scenario, uniform high-speed scenario, uniform low-speed scenario, dense medium-speed scenario, dense high-speed scenario, dense low-speed scenario, and dense ultra-low-speed scenario (where the average vehicle density of sparse scenario, uniform scenario, and dense scenario is 15 vehicles / 4000m, 40 vehicles / 4000m, and 100 vehicles / 4000m, respectively, and the average speed of social vehicles ranges from 5 km / h, 10 km / h, 20 km / h, 30 km / h, 40 km / h, 50 km / h, and 60 km / h, etc.).

[0368] Secondly, a training map is randomly loaded into the simulator, allowing the autonomous vehicle to drive in the simulated environment using a random strategy. This random strategy means the autonomous vehicle randomly selects a target action to execute from the decision space A (e.g., 0, 1, 2). It is assumed that each time the autonomous vehicle reaches the end of the training map, it will randomly switch to a new training map and a new social vehicle configuration scenario, continuing until a preset driving time period (e.g., 2,000,000 time windows) is reached.

[0369] During the simulation, global classification model features are acquired. These global classification model features may include, but are not limited to, the driving information of the autonomous vehicle and other vehicles within its perception range before and after each lane change by the autonomous vehicle within a preset time period (e.g., a 2,000,000 time window). cur The average speed V of vehicles in the lane before each lane change by the autonomous vehicle cur The average gap between two adjacent vehicles in the lane after each lane change by an autonomous vehicle. goal The average speed V of vehicles in the lane after each lane change by the autonomous vehicle goal ).

[0370] Furthermore, if the gap in the driving information before and after any lane change... cur <gap goalIf the lane change is 1, the corresponding label for the driving information before and after the lane change is 1 (meaning that the lane where the autonomous vehicle is located after the lane change is sparser than the lane where the autonomous vehicle was located before the lane change), otherwise it is 0 (meaning that the lane where the autonomous vehicle is located after the lane change is denser than the lane where the autonomous vehicle was located before the lane change).

[0371] Furthermore, the driving information and corresponding labels before and after each lane change are added as sample data to the training set D, and the sample data in the training set D and the logistic regression algorithm are used to learn and generate a preset global classification model F0 (the output of the model is the probability that the lane after the lane change is more sparse).

[0372] Of course, a preset global classification model F0 can also be generated through other methods, and this application embodiment does not impose any restrictions on this.

[0373] The automatic lane changing method provided in the embodiments of this application will be described in detail below.

[0374] Figure 11 This is a flowchart illustrating an automatic lane-changing method according to an embodiment of this application. The method in this embodiment can specifically be as follows: Figure 1 The execution device 1002 shown is used for execution. For example... Figure 11 As shown, the method provided in this application embodiment may include:

[0375] S1101. Based on the current driving information of the autonomous vehicle and the movement information of obstacles in each lane within the perception range of the autonomous vehicle, calculate the local neighbor features and global statistical features of the autonomous vehicle at the current moment.

[0376] It should be noted that when the obstacle is a vehicle or other mobile terminal, the motion information is driving information; when the obstacle is a person, animal or stationary object, the motion information may include relevant information such as speed.

[0377] In this step, based on the current driving information of the autonomous vehicle (such as driving speed and / or driving position) and the motion information of obstacles in each lane within the autonomous vehicle's perception range (i.e., the range that the autonomous vehicle's sensors can detect, such as the range within a preset interval from the autonomous vehicle), the local neighbor features and global statistical features of the autonomous vehicle at the current moment are calculated.

[0378] The local neighbor features of the autonomous vehicle at the current moment in the embodiments of this application are used to represent the motion state information (e.g., relative distance and relative speed) of specific neighbor obstacles (e.g., obstacles in front and behind the autonomous vehicle in the lane where the autonomous vehicle is located at the current moment, obstacles in front and behind the autonomous vehicle in the adjacent left lane of the lane where the autonomous vehicle is located at the current moment, obstacles in front and behind the autonomous vehicle in the adjacent right lane of the lane where the autonomous vehicle is located at the current moment) relative to the autonomous vehicle.

[0379] For example, the local neighbor features s of an autonomous vehicle at any given time l This may include, but is not limited to: the relative speed of the obstacle in front of the autonomous vehicle in its lane at that moment relative to the autonomous vehicle. and relative distance The relative speed of the obstacle behind the autonomous vehicle in the lane it is in at that moment, relative to the autonomous vehicle. and relative distance The relative speed of the obstacle in front of the autonomous vehicle in the adjacent left lane of the lane in which the autonomous vehicle is located at that moment, relative to the autonomous vehicle. and relative distance The relative speed of the obstacle behind the autonomous vehicle in the adjacent left lane of the lane in which the autonomous vehicle is located at that moment, relative to the autonomous vehicle. and relative distance The relative speed of the obstacle in front of the autonomous vehicle in the adjacent right lane of the lane in which the autonomous vehicle is located at that moment, relative to the autonomous vehicle. and relative distance The relative speed of the obstacle adjacent to the autonomous vehicle in the right lane next to the autonomous vehicle at that moment. and relative distance

[0380] In this application embodiment, the global statistical features of the autonomous vehicle at the current moment are used to represent the sparsity and density of obstacles in each lane within the perception range of the autonomous vehicle at the current moment, such as the average driving speed and average interval of all obstacles in each lane at the current moment.

[0381] For example, the global statistical characteristics s of an autonomous vehicle at any given time. g This may include, but is not limited to: the average gap between two adjacent obstacles in all lanes to the left of the lane in which the autonomous vehicle is located at that moment. LThe average gap between two adjacent obstacles in the lane where the autonomous vehicle is located at that moment. M The average gap between two adjacent obstacles in all lanes to the right of the autonomous vehicle at that moment. R The average speed V of obstacles in all lanes to the left of the lane in which the autonomous vehicle is located at that moment. L The average speed V of the autonomous vehicle relative to obstacles in its lane at that moment. M and the average speed V of obstacles in all lanes to the right of the lane in which the autonomous vehicle is located at that moment. R .

[0382] S1102. Obtain the target action instruction based on the characteristics of local neighboring vehicles, global statistical characteristics, and the current control strategy.

[0383] For example, when the execution device 1002 executes steps S1101-S1102 for the first time, the current control strategy of this application embodiment may be a preset control strategy, such as the control strategy finally obtained by the training device 1001 through the training method of the control strategy; when the execution device 1002 does not execute steps S1101-S1102 for the first time, the current control strategy of this application embodiment may be the control strategy obtained by the execution device 1002 after the previous update.

[0384] In this step, by inputting the local neighbor features and global statistical features of the autonomous vehicle at the current moment into the current control policy (i.e., the control policy at the current moment), the target action instruction at the current moment can be obtained (used to instruct the autonomous vehicle to perform the target action).

[0385] For example, the control policy at any given time (e.g., the current control policy) can be represented as:

[0386]

[0387] Where s represents the local neighbor features and global statistical features of the autonomous vehicle at that moment; a′∈(0,1,2), a′ equal to 0 means to keep going straight, a′ equal to 1 means to change lanes to the left adjacent lane, and a′ equal to 2 means to change lanes to the right adjacent lane.

[0388] In this embodiment, for any current moment, the action a′ that maximizes Q(s, a′, q) is selected as the target action a at the current moment, based on the local neighbor features and global statistical features of the autonomous vehicle.

[0389] For example, the target action includes at least two types: lane changing or maintaining straight driving, wherein lane changing includes: changing lanes to the left adjacent lane or changing lanes to the right adjacent lane.

[0390] Of course, the control strategy at the current moment can also adopt other variations or equivalent formulas of the above formula, and this application embodiment does not limit this.

[0391] S1103. Execute the target action according to the target action instruction.

[0392] For example, if the target action indicates a lane change, the autonomous vehicle performs the lane change action; if the target action indicates a straight-ahead movement, the autonomous vehicle performs the straight-ahead movement.

[0393] Specifically, the method of executing the target action according to the target action instruction can be found in related technologies, and is not limited in this embodiment.

[0394] In this embodiment, based on the current driving information of the autonomous vehicle and the movement information of obstacles in each lane within the perception range of the autonomous vehicle, the local neighbor features and global statistical features of the autonomous vehicle at the current moment are calculated. Further, a target action instruction is obtained based on the local neighbor features, global statistical features, and the current control strategy, and the target action is executed according to the target action instruction. It can be seen that by introducing global statistical features into the current control strategy to obtain the target action instruction based on the local neighbor features, not only is the information of local neighbor obstacles (such as other vehicles) considered, but also the macroscopic situation of global statistical features (such as the overall traffic flow). Therefore, the target action obtained by integrating local and all road obstacle information is the globally optimal strategy action.

[0395] Furthermore, based on the above embodiments, in this application embodiment, feedback information can be obtained by executing the target action, and the current control strategy can be updated according to the feedback information to obtain the control strategy for the next moment, so that the target action for the next moment can be accurately determined according to the control strategy for the next moment.

[0396] It should be noted that at each time t, the control strategy at time t can be updated based on the feedback information at time t to obtain the control strategy at time t+1. This ensures that the control strategy for generating the target action is constantly being updated and optimized in an adaptive manner, thereby guaranteeing that there is a corresponding optimal control strategy at each time and providing a guarantee for the accurate generation of the target action at each time.

[0397] In this embodiment, feedback information is obtained by executing the target action (used to update the current control strategy) so as to determine the reward value corresponding to the target action and the local neighbor features and global statistical features of the autonomous vehicle at the next moment, thereby updating the current control strategy.

[0398] For example, the feedback information may include, but is not limited to: driving information (such as driving speed or driving position) after the autonomous vehicle performs the target action, driving information of the autonomous vehicle at the next moment, and movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment; when the target action is lane changing, the feedback information may also include: the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after lane changing compared to the lane it was in before lane changing; wherein, the historical average time includes the average time of the autonomous vehicle performing the same type of action (such as lane changing) within a preset historical time period (e.g., a 500-time window).

[0399] For example, the change in the sparsity and density of the lane after the autonomous vehicle changes lanes compared to the lane before the lane change can be based on the autonomous vehicle and the motion information of obstacles in each lane within the autonomous vehicle's perception range before and after the lane change (e.g., the average gap between two adjacent obstacles in the lane before the lane change). cur The average speed V of the obstacles in the lane in which the autonomous vehicle was located before changing lanes. cur The average gap between two adjacent obstacles in the lane where an autonomous vehicle changes lanes. goal The average speed V of obstacles in the lane where the autonomous vehicle is located after changing lanes. goal ), and determined by the preset global classification model F0.

[0400] Figure 12 This is a flowchart illustrating an automatic lane-changing method according to another embodiment of this application. Based on the above embodiments, this application describes a feasible method for "updating the current control strategy based on feedback information to obtain the control strategy for the next moment." Figure 12 As shown, the method in this application embodiment may include:

[0401] S1201. Calculate the reward value corresponding to the target action based on the feedback information, as well as the local neighbor features and global statistical features of the autonomous vehicle at the next moment.

[0402] In this step, based on the autonomous vehicle's driving information at the next moment and the movement information of obstacles in each lane within the autonomous vehicle's perception range at the next moment, the local neighbor features and global statistical features of the autonomous vehicle at the current moment can be calculated. The specific calculation method can refer to the method for obtaining the autonomous vehicle's local neighbor features and global statistical features at that historical moment in step S1001 above, which will not be elaborated further in this embodiment.

[0403] The following sections of the embodiments of this application describe the possible implementation of "calculating the reward value corresponding to the target action based on feedback information".

[0404] One possible implementation is to calculate the reward value based on the driving information after the autonomous vehicle performs the target action, when the target action is to maintain straight-line driving.

[0405] For example, the reward value is calculated based on a preset function R(s″) and the driving information s″ (such as driving speed or distance) after the autonomous vehicle performs the target action. For example, the preset function R(s″) = V ego ′, V ego ′ represents the speed of the autonomous vehicle after performing the target action; of course, the preset function R(s″) can also be equal to other functions that include the driving information of the autonomous vehicle after performing the target action, and this application embodiment does not limit this.

[0406] Another possible approach is to calculate the reward value based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane the autonomous vehicle is in after the lane change compared to the lane it was in before the lane change.

[0407] In this implementation, the time T for executing the target action is compared with the historical average time T. e The ratio determines the local return coefficient K. l Furthermore, the global reward coefficient K is determined based on the changes in the sparsity and density of the lane the autonomous vehicle is in after changing lanes compared to its original lane. g Where, when the lane where the autonomous vehicle is located after changing lanes is denser than the lane where the autonomous vehicle was located before changing lanes, K g >1; When the lane where the autonomous vehicle is in after changing lanes is less sparse than the lane where the autonomous vehicle was in before changing lanes, K g <1. Furthermore, based on the driving information of the autonomous vehicle after performing the target action and the local reward coefficient K... l and global return coefficient K g Calculate the return value.

[0408] For example, according to the formula c*K l *K g *R(s″) calculates the reward value; where c represents a preset discount factor (e.g., 0.3), and R(s″) represents a preset function containing driving information after the autonomous vehicle performs the target action; of course, the reward value can also be calculated through other equivalent or modified formulas of the above formula, and this application embodiment does not limit this.

[0409] If the local neighbor features of the autonomous vehicle at the current moment can also include: the position information flag between the navigation target lane and the lane where the autonomous vehicle is located, and the distance dist2goal between the autonomous vehicle and the next intersection along the driving direction; where flag∈{0, -1, 1}, where flag equals 0 represents that the autonomous vehicle is on the navigation target lane, flag equals -1 represents that the navigation target lane is to the left of the lane where the autonomous vehicle is located, and flag equals 1 represents that the navigation target lane is to the right of the lane where the autonomous vehicle is located, then based on the time T of executing the target action and the historical average time T e The ratio determines the local return coefficient K. l Furthermore, the first global reward coefficient is determined based on the change in the sparsity of the lane the autonomous vehicle is in after changing lanes compared to its original lane.

[0410] Furthermore, a second global reward coefficient is determined based on the positional information (flag) between the navigation target lane and the lane where the autonomous vehicle is located, and the target action. For example, the second global return coefficient is determined according to the following formula.

[0411]

[0412] Among them, gap cur The gap represents the average distance between two adjacent obstacles in the lane in which the autonomous vehicle is located before changing lanes. goal This represents the average distance between two adjacent obstacles in the lane where the autonomous vehicle is located after changing lanes, and 'a' represents the target action.

[0413] Of course, the second global return coefficient can also be calculated using other equivalent or modified formulas of the above formula. This application does not impose any limitations on this aspect.

[0414] Furthermore, based on the driving information of the autonomous vehicle after performing the target action and the local reward coefficient K... l First global return coefficient Second global return coefficient Calculate the return value.

[0415] For example, according to the formula Calculate the return value; where c represents a preset discount factor (e.g., 0.3), and R(s″) represents a preset function containing driving information after the autonomous vehicle performs the target action.

[0416] For example, preset functions V ego' represents the speed of the autonomous vehicle after performing the target action; flag' represents the position information between the navigation target lane and the lane where the autonomous vehicle is located after performing the target action; flag' equal to 0 means the autonomous vehicle is on the navigation target lane after performing the target action; flag' equal to -1 means the navigation target lane is to the left of the lane where the autonomous vehicle is located after performing the target action; flag' equal to 1 means the navigation target lane is to the right of the lane where the autonomous vehicle is located after performing the target action; dist2goal' represents the distance between the autonomous vehicle and the next intersection along the driving direction after performing the target action; of course, the preset function R(s″) can also be equal to other functions that include the driving information of the autonomous vehicle after performing the target action, and this application embodiment does not impose any restrictions on this.

[0417] Of course, the return value can also be calculated using other equivalent or modified formulas of the above formula, and this application embodiment does not limit this.

[0418] Of course, the reward value corresponding to the target action can also be calculated in other ways based on the feedback information, and this application embodiment does not limit this.

[0419] Step S1202: Determine the quadruple information at the current moment.

[0420] In this step, the quadruple information for the current moment is determined based on the local neighbor features and global statistical features of the autonomous vehicle at the current moment, the target action at the current moment, the reward value corresponding to the target action calculated in step S1201 above, and the local neighbor features and global statistical features of the autonomous vehicle at the next moment.

[0421] For example, the quadruple information at the current moment corresponds to the vehicle condition at the current moment, and may include: the features at the current moment, the target action of the autonomous vehicle at the current moment, the reward value corresponding to the target action, and the features of the next moment. The features at the current moment include the local neighbor features and global statistical features of the autonomous vehicle at the current moment, and the features of the next moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment.

[0422] S1203. Update the current control strategy based on the current quaternion information to obtain the control strategy for the next moment.

[0423] One possible implementation is as follows: when the target action is to move straight, generate the target value corresponding to the four-tuple information at the current moment; further, use gradient descent to iteratively update the parameter q in the first preset function containing the target value; further, replace the parameter q in the current control strategy with the iteratively updated parameter q to obtain the control strategy for the next moment.

[0424] In this implementation, when the target action is straight, the target value y corresponding to the four-tuple information at the current moment can be generated using the following formula.

[0425] For example,

[0426] Where g represents the preset forgetting factor, g∈(0,1); Q(s′,a,q) represents the action value function; The expression represents traversing a to maximize Q(s′, a, q); s′ represents the feature of the next time step after the current time step.

[0427] Of course, based on the quadruple information at the current moment, the target value corresponding to the quadruple information can also be generated through other variations or equivalent formulas of the above formula, and this application embodiment does not impose any restrictions on this.

[0428] Furthermore, gradient descent is used to apply the first preset function (yQ(s, a, q)) containing the target value y. 2 The parameter q in the equation is updated iteratively; where Q(s, a, q) is the action value function corresponding to the quadruple information at the current time, s represents the local neighbor features and global features at the current time in the quadruple information at the current time, and a represents the target action at the current time in the quadruple information at the current time.

[0429] Furthermore, the updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next time step, so as to determine the target action for the next time step.

[0430] Another possible implementation is to obtain the extended quadruple information at the current moment when the target action is lane change; further, update the current control strategy based on the quadruple information at the current moment and the extended quadruple information at the current moment to obtain the control strategy for the next moment.

[0431] The extended quadruple information of the current moment involved in the embodiments of this application corresponds to the extended vehicle condition of the current moment, which is obtained by processing the vehicle condition of the current moment with symmetric rules and monotonic rules.

[0432] The symmetry rule involved in this application embodiment refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane where the autonomous vehicle is located as the axis.

[0433] The monotonicity principle involved in this application embodiment refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane where the autonomous vehicle changes lanes, and / or changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lanes by less than a preset distance range.

[0434] For example, the extended quadruple information at the current time may include: the symmetric quadruple information at the current time and the monotonic quadruple information. For instance, the symmetric quadruple information at the current time may be obtained by applying the symmetry principle to the quadruple information at the current time, and the monotonic quadruple information at the current time may be obtained by applying the monotonic principle to the quadruple information at the current time.

[0435] Specifically, the construction methods for the symmetric quadruples and monotonic quadruples at the current moment can refer to the construction methods for "symmetric quadruples and monotonic quadruples at any moment" mentioned above in this application, and will not be repeated here.

[0436] In this implementation, when the target action is a lane change, the extended quadruple information at the current moment is obtained, and the information is based on the current quadruple information and the i-th quadruple information (s) in the extended quadruple information at the current moment. i a i r i s i The target value y corresponding to the i-th quadruple can be generated using the following formula. i Where i is a positive integer not greater than n, and n is the total number of quadruples included in the current time quadruples information and the current time extended quadruples information.

[0437] For example,

[0438] Where g represents the preset forgetting factor, g∈(0,1); Q(s i ′, a i q) represents the action value function; Represents traversing a i Make Q(s) i ′, a i ,q) takes the maximum value; s i ′ represents the feature of the next time step in the information of the i-th quadruple.

[0439] Of course, based on the quaternion information at the current moment and the i-th quaternion information in the extended quaternion information at the current moment, the target value corresponding to the i-th quaternion information can also be generated through other variations or equivalent formulas of the above formula. This application does not impose any restrictions on this.

[0440] Furthermore, the gradient descent method is used to calculate the target value y corresponding to the information of the i-th quadruple. i The second preset function The parameter q in the equation is updated iteratively; where Q(s) i a i ,q) is the action value function corresponding to the i-th four-tuple information, s i a represents the feature of the previous time step in the information of the i-th quadruple. i This represents the target action in the previous time step within the information of the i-th quadruple.

[0441] Furthermore, the updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next time step, so as to determine the target action for the next time step.

[0442] Another possible implementation is to update the control strategy for the next moment based on the current control strategy when the target action is to maintain straight movement, according to the current four-tuple information, the historical four-tuple information, and the extended four-tuple information of the historical moment.

[0443] The four-tuple information of the historical moment in this application embodiment corresponds to the vehicle condition at the historical moment, which may include, but is not limited to: the characteristics of the historical moment, the target action of the autonomous vehicle at the historical moment (i.e., the target action determined at the historical moment according to the corresponding control strategy), the reward value corresponding to the target action at the historical moment, and the characteristics of the next moment of the historical moment.

[0444] For example, the features of this historical moment may include, but are not limited to: the local neighbor features and global statistical features of the autonomous vehicle at that historical moment.

[0445] For example, the features of the next moment in the historical moment may include, but are not limited to: the local neighbor features and global statistical features of the autonomous vehicle at that next moment.

[0446] The extended quadruple information of the historical moment involved in the embodiments of this application corresponds to the extended vehicle condition of the historical moment, which is obtained by processing the vehicle condition of the historical moment with symmetric rules and monotonic rules.

[0447] The symmetry rule involved in this application embodiment refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane where the autonomous vehicle is located as the axis.

[0448] The monotonicity principle involved in this application embodiment refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane where the autonomous vehicle changes lanes, and / or changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lanes by less than a preset distance range.

[0449] For example, the extended quadruple information of a historical moment can include: symmetric quadruple information and monotonic quadruple information of the historical moment. For instance, the symmetric quadruple information of a historical moment can be obtained by processing the quadruple information of the historical moment according to the symmetry principle, and the monotonic quadruple information of a historical moment can be obtained by processing the quadruple information of the historical moment according to the monotonic principle.

[0450] Specifically, the construction methods for symmetric quadruples and monotonic quadruples at historical moments can refer to the construction methods for "symmetric quadruples and monotonic quadruples at any moment" mentioned above in this application, and will not be repeated here.

[0451] In this implementation, when the target action is to maintain straight movement, the j-th quadruple information (s) from the current time quadruple information, the historical time quadruple information, and the historical time extended quadruple information is used. j a j r j s j The target value y corresponding to the j-th quadruple can be generated using the following formula. j ; where j is a positive integer not greater than m, and m is the total number of quadruples included in the current quadruple information, the historical quadruple information, and the extended quadruple information of the historical time.

[0452] For example,

[0453] Where g represents the preset forgetting factor, g∈(0,1); Q(s j ′, a j q) represents the action value function; Represents traversing a j Make Q(s) j ′, a j ,q) takes the maximum value; s j ′ represents the feature of the next time step in the j-th quadruple information.

[0454] Of course, based on the j-th quaternion information in the current quaternion information, the historical quaternion information, and the extended quaternion information of the historical quaternion information, the target value corresponding to the j-th quaternion information can also be generated through other variations or equivalent formulas of the above formula. This application does not impose any restrictions on this.

[0455] Furthermore, the gradient descent method is used to calculate the target value y containing the information of the j-th quadruple. j The third preset function The parameter q in the equation is updated iteratively; where Q(s) j aj Let q) be the action value function corresponding to the j-th four-tuple information, and s j a represents the feature of the previous time step in the j-th quadruple information. j This represents the target action in the previous time step within the j-th quadruple information.

[0456] Furthermore, the updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next moment, so as to determine the target action for the next moment.

[0457] Another possible implementation is to obtain the extended quadruple information at the current moment when the target action is lane change; further, based on the quadruple information at the current moment, the extended quadruple information at the current moment, the quadruple information at the historical moment, and the extended quadruple information at the historical moment, update the current control strategy to obtain the control strategy for the next moment.

[0458] The extended quadruple information of the current moment involved in the embodiments of this application corresponds to the extended vehicle condition of the current moment, which is obtained by processing the vehicle condition of the current moment with symmetric rules and monotonic rules.

[0459] The symmetry rule involved in this application embodiment refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane where the autonomous vehicle is located as the axis.

[0460] The monotonicity principle involved in this application embodiment refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane where the autonomous vehicle changes lanes, and / or changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lanes by less than a preset distance range.

[0461] For example, the extended quadruple information at the current time may include: the symmetric quadruple information at the current time and the monotonic quadruple information. For instance, the symmetric quadruple information at the current time may be obtained by applying the symmetry principle to the quadruple information at the current time, and the monotonic quadruple information at the current time may be obtained by applying the monotonic principle to the quadruple information at the current time.

[0462] Specifically, the construction methods for the symmetric quadruples and monotonic quadruples at the current moment can refer to the construction methods for "symmetric quadruples and monotonic quadruples at any moment" mentioned above in this application, and will not be repeated here.

[0463] The four-tuple information of the historical moment in this application embodiment corresponds to the vehicle condition at the historical moment, which may include, but is not limited to: the characteristics of the historical moment, the target action of the autonomous vehicle at the historical moment (i.e., the target action determined at the historical moment according to the corresponding control strategy), the reward value corresponding to the target action at the historical moment, and the characteristics of the next moment of the historical moment.

[0464] For example, the features of this historical moment may include, but are not limited to: the local neighbor features and global statistical features of the autonomous vehicle at that historical moment.

[0465] For example, the features of the next moment in the historical moment may include, but are not limited to: the local neighbor features and global statistical features of the autonomous vehicle at that next moment.

[0466] The extended quadruple information of the historical moment involved in the embodiments of this application corresponds to the extended vehicle condition of the historical moment, which is obtained by processing the vehicle condition of the historical moment with symmetric rules and monotonic rules.

[0467] For example, the extended quadruple information of a historical moment can include: symmetric quadruple information and monotonic quadruple information of the historical moment. For instance, the symmetric quadruple information of a historical moment can be obtained by processing the quadruple information of the historical moment according to the symmetry principle, and the monotonic quadruple information of a historical moment can be obtained by processing the quadruple information of the historical moment according to the monotonic principle.

[0468] Specifically, the construction methods for symmetric quadruples and monotonic quadruples at historical moments can refer to the construction methods for "symmetric quadruples and monotonic quadruples at any moment" mentioned above in this application, and will not be repeated here.

[0469] In this implementation, when the target action is a lane change, the extended quadruplet information at the current moment is obtained, and the k-th quadruplet information (s) from the current quadruplet information, the current extended quadruplet information, the quadruplet information at the historical moment, and the extended quadruplet information at the historical moment is used. k a k r k s k The target value y corresponding to the k-th quadruple can be generated using the following formula. k ; where k is a positive integer not greater than p, and p is the total number of quadruples included in the current quadruple information, the current extended quadruple information, the historical quadruple information, and the historical extended quadruple information.

[0470] For example,

[0471] Where g represents the preset forgetting factor, g∈(0,1); Q(sk ′, a k q) represents the action value function; Represents traversing a k Make Q(s) k ′, a k ,q) takes the maximum value; s k ′ represents the feature of the next time step in the information of the k-th quadruple.

[0472] Of course, based on the kth quaternion information in the current time quaternion information, the current time extended quaternion information, the historical time quaternion information, and the historical time extended quaternion information, the target value corresponding to the kth quaternion information can also be generated through other variations or equivalent formulas of the above formula. This application does not impose any restrictions on this.

[0473] Furthermore, the gradient descent method is used to calculate the target value y containing the information of the k-th quadruple. k The fourth preset function The parameter q in the equation is updated iteratively; where Q(s) k a k ,q) is the action value function corresponding to the k-th quadruple information, s k a represents the feature of the previous time step in the information of the k-th quadruple. k This represents the target action in the previous time step within the information of the k-th quadruple.

[0474] Furthermore, the updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next time step, so as to determine the target action for the next time step.

[0475] Of course, based on the quadruple information at the current moment, the control strategy at the next moment can also be updated in other ways to obtain the control strategy at the current moment, and this application embodiment does not limit this.

[0476] Figure 13 This is a schematic diagram of training data provided for an embodiment of this application. For example... Figure 13 As shown in the embodiments of this application, the training method for the control strategy obtained during the training process demonstrates the performance trend of the control strategy at different stages in four different traffic flows (e.g., sparse scenarios, normal scenarios, congested scenarios, and very congested scenarios). Figure 13The horizontal axis represents the number of iterations in the entire training process (unit: 10,000 times), and the vertical axis represents the time taken for the autonomous vehicle to travel the entire distance on a fixed-length lane (unit: seconds). The red curve represents the convergence trend of training using only local neighbor features as input (Scheme 1); the blue curve represents the trend of training after further incorporating global statistical features on top of local neighbor features (Scheme 2); and the green curve represents the convergence trend of training after introducing both local and global reward coefficients when calculating the reward value (Scheme 3). It can be seen that the performance of the control strategy after adopting global statistical features is significantly improved; when both local and global reward coefficients are introduced when calculating the reward value, not only is convergence accelerated, but the performance of the control strategy is also enhanced.

[0477] Table 1 is a schematic table of training data provided in the embodiments of this application. As shown in Table 1, comparisons are made with related solutions in sparse, normal, congested, and very congested scenarios. It can be seen that our solution has advantages over related solutions in both average speed and average lane change frequency. At the same time, we also statistically analyzed some lane-changing behaviors that seem unreasonable locally but are actually reasonable in the long run, which we call "soft lane changes." We found that our solution has a certain proportion of "soft lane changes," indicating that our model has greater long-term intelligence.

[0478] Table 1 is a schematic table of training data provided in the embodiments of this application.

[0479]

[0480] Figure 14 This is a schematic diagram of the structure of an automatic lane-changing device provided in an embodiment of this application. Figure 14 As shown, the automatic lane changing device 140 provided in this embodiment may include: a calculation module 1401, an acquisition module 1402, and an execution module 1403.

[0481] The calculation module 1401 is used to calculate the local neighbor features and global statistical features of the autonomous vehicle at the current moment based on the driving information of the autonomous vehicle at the current moment and the motion information of obstacles in each lane within the perception range of the autonomous vehicle. The local neighbor features are used to represent the motion state information of specific neighbor obstacles of the autonomous vehicle relative to the autonomous vehicle. The global statistical features are used to represent the sparsity and density of obstacles in each lane within the perception range.

[0482] The acquisition module 1402 is used to acquire a target action indication based on the local neighbor features, the global statistical features and the current control strategy. The target action indication is used to instruct the autonomous vehicle to perform a target action, which includes at least two types: lane changing or maintaining straight driving.

[0483] The execution module 1403 is used to execute the target action according to the target action instruction.

[0484] In one possible implementation, the device further includes:

[0485] A feedback module is used to obtain feedback information by executing the target action, and the feedback information is used to update the current control strategy. The feedback information includes the driving information of the autonomous vehicle after executing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment. When the target action is a lane change, the feedback information further includes: the ratio of the time to execute the target action to the historical average time, and the change in the density of the lane the autonomous vehicle is in after the lane change compared to the lane it was in before the lane change. The historical average time includes the average time the autonomous vehicle takes to execute similar actions within a preset historical time period.

[0486] The update module is used to update the current control strategy based on the feedback information to obtain the control strategy for the next moment.

[0487] In one possible implementation, the update module includes:

[0488] The calculation unit is used to calculate the reward value corresponding to the target action based on the feedback information, as well as the local neighbor features and global statistical features of the autonomous vehicle at the next moment;

[0489] A determining unit is used to determine the quadruple information at the current moment; wherein, the quadruple information at the current moment corresponds to the vehicle condition at the current moment, including: the features at the current moment, the target action, the reward value corresponding to the target action, and the features at the next moment, wherein the features at the current moment include the local neighbor features and global statistical features of the autonomous vehicle at the current moment, and the features at the next moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment;

[0490] The update unit is used to update the current control strategy based on the quadruple information at the current moment to obtain the control strategy at the next moment.

[0491] In one possible implementation, when the target action is to maintain a straight line, the update unit is specifically used for:

[0492] Based on the quadruple information at the current moment, generate the target value corresponding to the quadruple information;

[0493] The parameter q in the first preset function containing the target value is iteratively updated using the gradient descent method;

[0494] The updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next time step.

[0495] In one possible implementation, when the target action is a lane change, the update unit is specifically used for:

[0496] Obtain the extended quadruple information at the current moment, which corresponds to the extended vehicle condition at the current moment. The extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment using symmetry and monotonic rules. The symmetry rule refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane as the axis. The monotonic rule refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane of the lane change, and / or, changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lane to a preset distance range.

[0497] The control strategy for the next moment is obtained by updating the current control strategy based on the current quadruple information and the extended quadruple information at the current moment.

[0498] In one possible implementation, the update unit is specifically used for:

[0499] Based on the quaternion information at the current time and the ith quaternion information in the extended quaternion information at the current time, generate the target value corresponding to the ith quaternion information; where i is a positive integer not greater than n, and n is the total number of quaternion information included in the quaternion information at the current time and the extended quaternion information at the current time.

[0500] The parameter q in the second preset function containing the target value corresponding to the i-th quadtuple information is iteratively updated using the gradient descent method.

[0501] The updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next time step.

[0502] In one possible implementation, when the target action is to maintain a straight line, the update unit is specifically used for:

[0503] The control strategy for the next moment is obtained by updating the current control strategy based on the quadruple information at the current moment, the quadruple information at the historical moment, and the extended quadruple information at the historical moment.

[0504] The four-tuple information of the historical moment corresponds to the vehicle condition at the historical moment, including: the features of the historical moment, the target action of the historical moment, the reward value corresponding to the target action of the historical moment, and the features of the next moment of the historical moment. The features of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the historical moment, and the features of the next moment of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of the historical moment. The extended four-tuple information of the historical moment corresponds to the extended vehicle condition at the historical moment, which is obtained by processing the vehicle condition at the historical moment using symmetric rules and monotonic rules.

[0505] In one possible implementation, the update unit is specifically used for:

[0506] Based on the j-th quadruple information from the current time quadruple information, the historical time quadruple information, and the historical time extended quadruple information, generate the target value corresponding to the j-th quadruple information; where j is a positive integer not greater than m, and m is the total number of quadruple information included in the current time quadruple information, the historical time quadruple information, and the historical time extended quadruple information;

[0507] The parameter q in the third preset function containing the target value corresponding to the j-th quadtuple is iteratively updated using the gradient descent method.

[0508] The updated parameter q is used to replace the parameter q in the current control strategy to obtain the control strategy for the next time step.

[0509] In one possible implementation, when the target action is a lane change, the update unit is specifically used for:

[0510] Obtain the extended quadruple information at the current moment; wherein, the extended quadruple information at the current moment corresponds to the extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment with symmetric rules and monotonic rules;

[0511] Based on the current time quadruple information, the current time extended quadruple information, the historical time quadruple information, and the historical time extended quadruple information, the current control strategy is updated to obtain the control strategy for the next time. The historical time quadruple information corresponds to the vehicle condition at the historical time, and the historical time extended quadruple information corresponds to the historical time extended vehicle condition. The historical time extended vehicle condition is obtained by processing the historical time vehicle condition using symmetric and monotonic rules.

[0512] In one possible implementation, the update unit is specifically used for:

[0513] Based on the k-th quaternion information from the current quaternion information, the extended quaternion information of the current quaternion information, the quaternion information of the historical quaternion information, and the extended quaternion information of the historical quaternion information, generate the target value corresponding to the k-th quaternion information; where k is a positive integer not greater than p, and p is the total number of quaternion information included in the current quaternion information, the extended quaternion information of the current quaternion information, the quaternion information of the historical quaternion information, and the extended quaternion information of the historical quaternion information;

[0514] The parameter q in the fourth preset function, which contains the target value corresponding to the kth quadtuple information, is iteratively updated using the gradient descent method.

[0515] The updated parameter q is used to replace the parameter g in the current control strategy to obtain the control strategy for the next time step.

[0516] In one possible implementation, when the target action is to maintain straight movement, the computing unit is specifically used for:

[0517] The reward value is calculated based on the driving information of the autonomous vehicle after it performs the target action;

[0518] Based on the driving information of the autonomous vehicle at the next moment and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment, the local neighbor features and global statistical features of the autonomous vehicle at the next moment are calculated.

[0519] In one possible implementation, when the target action is a lane change, the computing unit is specifically used for:

[0520] The reward value is calculated based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after changing lanes compared to the lane before changing lanes.

[0521] Based on the driving information of the autonomous vehicle at the next moment and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment, the local neighbor features and global statistical features of the autonomous vehicle at the next moment are calculated.

[0522] In one possible implementation, the specific neighboring obstacles of the autonomous vehicle include at least one of the following: obstacles in front of and behind the autonomous vehicle in the lane where the autonomous vehicle is located, obstacles in front of and behind the autonomous vehicle in the adjacent left lane of the lane where the autonomous vehicle is located, and obstacles in front of and behind the autonomous vehicle in the adjacent right lane of the lane where the autonomous vehicle is located.

[0523] Wherein, when the autonomous vehicle is in the left lane, the motion state information of the obstacles in front of and behind the autonomous vehicle in the adjacent left lane of the lane in which the autonomous vehicle is located, relative to the autonomous vehicle, is a default value; and / or,

[0524] When the autonomous vehicle is in the right lane, the motion state information of the obstacles in front and behind the autonomous vehicle in the adjacent right lane is the default value relative to the autonomous vehicle.

[0525] In one possible implementation, the global traffic flow statistics of the autonomous vehicle at the current moment include at least one of the following: the average speed and average interval of all obstacles in each lane within the perception range.

[0526] The automatic lane changing device 140 provided in this application embodiment can be used to execute the technical solutions in the above-described automatic lane changing method embodiments of this application. Its implementation principle and technical effects are similar, and will not be repeated here.

[0527] Figure 15 This is a schematic diagram of an automatic lane-changing device provided in another embodiment of this application. Figure 15 As shown, the automatic lane changing device 150 provided in this embodiment may include a processor 1501 and a memory 1502.

[0528] The memory 1502 is used to store program instructions;

[0529] The processor 1501 is used to call and execute the program instructions stored in the memory 1502. When the processor 1501 executes the program instructions stored in the memory 1502, the automatic lane changing device is used to execute the technical solution in the above-described automatic lane changing method embodiment of this application. Its implementation principle and technical effect are similar, and will not be described again here.

[0530] Understandable Figure 15 This illustration only shows a simplified design of the automatic lane changing device. In other embodiments, the automatic lane changing device may also include any number of processors, memory and / or communication units, etc., which is not limited in this application embodiment.

[0531] This application also provides a computer-readable storage medium storing instructions. When the instructions are executed on a computer, the computer performs the technical solution described in the above-described automatic lane-changing method embodiments of this application. The implementation principle and technical effects are similar and will not be repeated here.

[0532] This application also provides a program that, when executed by a processor, performs the technical solutions described in the above-described automatic lane-changing method embodiments of this application. The implementation principle and technical effects are similar and will not be repeated here.

[0533] This application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to execute the technical solution in the above-described automatic lane changing method embodiments of this application. Its implementation principle and technical effects are similar, and will not be repeated here.

[0534] In some embodiments, the disclosed method may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art. Figure 16 This is a conceptual partial view of a computer program product provided in an embodiment of this application. Figure 16 A conceptual partial view of an example computer program product arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer program product includes a computer program for executing computer processes on a computing device. In one embodiment, the example computer program product 600 is provided using a signal carrying medium 601. The signal carrying medium 601 may include one or more program instructions 602, which, when executed by one or more processors, can implement the technical solutions in the above-described embodiments of the automatic lane-changing method of this application. The implementation principles and technical effects are similar and will not be repeated here.

[0535] In some examples, signal-bearing medium 601 may comprise computer-readable medium 603, such as, but not limited to, hard disk drives, CDs, digital video optical discs (DVDs), digital magnetic tapes, memory, read-only memory (ROM), or random access memory (RAM), etc. In some embodiments, signal-bearing medium 601 may comprise computer-recordable medium 604, such as, but not limited to, memory, read / write (R / W) CDs, R / W DVDs, etc. In some embodiments, signal-bearing medium 601 may comprise communication medium 605, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, signal-bearing medium 601 may be transmitted by wireless communication medium 605 (e.g., wireless communication media conforming to the IEEE 802.11 standard or other transmission protocols). One or more program instructions 602 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to program instructions 602 transmitted to the computing device via one or more of computer-readable media 603, computer-recordable media 604, and / or communication media 605. It should be understood that the arrangements described herein are merely illustrative. Therefore, those skilled in the art will understand that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements may be omitted depending on the desired result. Furthermore, many of the described elements are functional entities that can be implemented as discrete or distributed components, or in any suitable combination and location in conjunction with other components.

[0536] Figure 17 This is a schematic diagram of the structure of a training device for a control strategy provided in one embodiment of this application. Figure 17 As shown, the control strategy training device 170 provided in this embodiment may include: a first acquisition module 1701 and an update module 1702.

[0537] The first acquisition module 1701 is used to perform step A: acquire a preset number of four-tuple information of historical moments, wherein the four-tuple information of the historical moments corresponds to the vehicle condition of the historical moments, including: the features of the historical moments, the target action of the autonomous vehicle at the historical moments, the reward value corresponding to the target action at the historical moments, and the features of the next moment of the historical moments. The features of the historical moments include the local neighbor features and global statistical features of the autonomous vehicle at the historical moments, and the features of the next moment of the historical moments include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of the historical moments.

[0538] The update module 1702 is used to perform step B: update the current control strategy to obtain the control strategy for the next moment based on the quadruple information of at least one first historical moment, the extended quadruple information of the at least one first historical moment, and the quadruple information of at least one second historical moment.

[0539] Wherein, the number of times steps A and B are executed repeatedly reaches a preset number, or steps A and B are executed repeatedly until the updated control strategy meets the preset conditions and then stops; the control strategy finally obtained by repeatedly executing steps A and B is used by the automatic lane changing device to obtain the target action indication when executing the automatic lane changing method.

[0540] Wherein, the at least one first historical moment's quadruple information is the quadruple information of the historical moment whose target action is lane change among the quadruple information of the preset number of historical moments; the at least one second historical moment's quadruple information is the quadruple information of other historical moments besides the at least one first historical moment's quadruple information among the quadruple information of the preset number of historical moments; any extended quadruple information of the first historical moment corresponds to the extended vehicle condition of the first historical moment, and the extended vehicle condition of the first historical moment is obtained by processing the vehicle condition of the first historical moment with symmetry rules and monotonic rules.

[0541] In one possible implementation, the update module 1702 includes:

[0542] A generation unit is configured to generate a target value corresponding to the l-th quadruple information based on the quadruple information of the at least one first historical moment, the extended quadruple information of the at least one first historical moment, and the l-th quadruple information of the at least one second historical moment; wherein, l is a positive integer not greater than q, and q is the total number of quadruple information included in the quadruple information of the at least one first historical moment, the extended quadruple information of the at least one first historical moment, and the quadruple information of the at least one second historical moment;

[0543] The update unit is used to iteratively update the parameter q in the preset function containing the target value corresponding to the l-th quadtuple information using the gradient descent method;

[0544] The replacement unit is used to replace the parameter q in the current control strategy with the iteratively updated parameter g to obtain the control strategy at the next moment.

[0545] In one possible implementation, the device further includes:

[0546] The first calculation module is used to calculate the local neighbor features and global statistical features of the autonomous vehicle at each historical moment, based on the driving information of the autonomous vehicle and the movement information of obstacles in each lane within the perception range of the autonomous vehicle.

[0547] The second acquisition module is used to acquire the target action indication at the historical moment based on the local neighbor features, global statistical features and the control strategy at the historical moment. The target action indication is used to instruct the autonomous vehicle to perform a target action, which includes at least two types: lane changing or maintaining straight driving.

[0548] A feedback module is used to obtain feedback information by executing the target action; wherein, the feedback information includes the driving information of the autonomous vehicle after executing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment; when the target action is a lane change, the feedback information further includes: the ratio of the time to execute the target action to the historical average time, and the change in the sparseness and density of the lane where the autonomous vehicle is located after the lane change compared to the lane before the lane change; wherein, the historical average time includes the average time of the autonomous vehicle executing the same type of action within a preset historical time period;

[0549] The second calculation module is used to calculate the reward value corresponding to the target action based on the feedback information, as well as the local neighbor features and global traffic flow statistics features of the autonomous vehicle at the next time step of the historical time.

[0550] The storage module is used to store the quadruple information of the historical moment.

[0551] In one possible implementation, when the target action is to maintain straight movement, the second calculation module is specifically used for:

[0552] The reward value is calculated based on the driving information of the autonomous vehicle after it performs the target action.

[0553] In one possible implementation, when the target action is a lane change, the second calculation module is specifically used for:

[0554] The reward value is calculated based on the driving information of the autonomous vehicle after performing the target action, the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after changing lanes compared to the lane before changing lanes.

[0555] The control strategy training device 170 provided in this application embodiment can be used to execute the technical solution in the control strategy training method embodiment of the above application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0556] Figure 18 A schematic diagram of the structure of a training device for a control strategy provided in another embodiment of this application. (See diagram below.) Figure 18 As shown, the control strategy training device 180 provided in this embodiment may include: a processor 1801 and a memory 1802;

[0557] The memory 1802 is used to store program instructions;

[0558] The processor 1801 is used to call and execute the program instructions stored in the memory 1802. When the processor 1801 executes the program instructions stored in the memory 1802, the control strategy training device is used to execute the technical solution in the above-mentioned control strategy training method embodiment of this application. Its implementation principle and technical effect are similar, and will not be described again here.

[0559] Understandable Figure 18 This illustration only shows a simplified design of the control policy training device. In other embodiments, the control policy training device may also include any number of processors, memory and / or communication units, etc., and this application does not impose any limitations on this embodiment.

[0560] This application also provides a computer-readable storage medium storing instructions. When the instructions are executed on a computer, the computer executes the technical solution in the training method embodiment of the control strategy described above. The implementation principle and technical effect are similar, and will not be repeated here.

[0561] This application also provides a program that, when executed by a processor, is used to perform the technical solution in the training method embodiment of the control strategy described above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0562] This application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to execute the technical solution in the training method embodiment of the control strategy described above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0563] For example, a conceptual partial view of the computer program product provided in the embodiments of this application can be referenced. Figure 16 As shown, it will not be elaborated further here.

[0564] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes the technical solutions in the above-described control strategy training method embodiment or the above-described automatic lane changing method embodiment. The implementation principle and technical effects are similar, and will not be repeated here.

[0565] Optionally, as one implementation, the chip may also include a memory storing instructions. The processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the technical solutions of the above-described control strategy training method embodiment or the above-described automatic lane changing method embodiment. The implementation principle and technical effects are similar, and will not be repeated here.

[0566] This application also provides an electronic device, which includes the automatic lane changing device provided in the above-described embodiments of the automatic lane changing device.

[0567] This application also provides an electronic device, which includes the control strategy training device provided in the above-described control strategy training device embodiment.

[0568] The processors involved in the embodiments of this application can be general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0569] The memory involved in the embodiments of this application can be non-volatile memory, such as hard disk drive (HDD) or solid-state drive (SSD), or it can be volatile memory, such as random-access memory (RAM). Memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0570] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0571] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0572] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0573] Those skilled in the art will understand that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0574] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

Claims

1. An automatic lane-changing method, characterized in that, include: Based on the current driving information of the autonomous vehicle and the movement information of obstacles in each lane within the perception range of the autonomous vehicle, calculate the local neighbor features and global statistical features of the autonomous vehicle at the current moment. The local neighbor features are used to represent the motion state information of specific neighbor obstacles of the autonomous vehicle relative to the autonomous vehicle; the global statistical features are used to represent the sparsity and density of obstacles in each lane within the perception range at the current moment. The global statistical features include at least one of the following: the average driving speed and average interval of all obstacles in each lane within the perception range at the current moment. Each lane includes: the lane where the autonomous vehicle is located, all lanes to the left of the lane where the autonomous vehicle is located, and all lanes to the right of the lane where the autonomous vehicle is located. The average driving speed represents the average driving speed of obstacles in the lane, and the average interval represents the average interval between two adjacent obstacles in the lane. Both the global statistical features and the local neighbor features are features after discretization processing, and the discrete granularity is related to the speed of the autonomous vehicle. The target action indication is obtained based on the local neighbor features, the global statistical features, and the current control strategy. The target action indication is used to instruct the autonomous vehicle to perform a target action, which includes at least two types: changing lanes or maintaining straight driving. Execute the target action according to the target action instruction.

2. The method according to claim 1, characterized in that, The method further includes: Feedback information is obtained by executing the target action, and the feedback information is used to update the current control strategy. The feedback information includes the driving information of the autonomous vehicle after performing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment. When the target action is a lane change, the feedback information also includes: the ratio of the time to perform the target action to the historical average time, and the changes in the sparseness and density of the lane where the autonomous vehicle is located after the lane change compared to the lane before the lane change. The historical average time includes the average time of the autonomous vehicle performing the same type of action within a preset historical time period. The current control strategy is updated based on the feedback information to obtain the control strategy for the next moment.

3. The method according to claim 2, characterized in that, The step of updating the current control strategy based on the feedback information to obtain the control strategy for the next moment includes: The reward value corresponding to the target action is calculated based on the feedback information, as well as the local neighbor features and global statistical features of the autonomous vehicle at the next moment; Determine the quadruple information at the current moment; wherein, the quadruple information at the current moment corresponds to the vehicle condition at the current moment, including: the feature at the current moment, the target action, the reward value corresponding to the target action, and the feature at the next moment, wherein the feature at the current moment includes the local neighbor features and global statistical features of the autonomous vehicle at the current moment, and the feature at the next moment includes the local neighbor features and global statistical features of the autonomous vehicle at the next moment; The control strategy for the next moment is obtained by updating the current control strategy based on the quadruple information at the current moment.

4. The method according to claim 3, characterized in that, When the target action is to maintain straight movement, updating the current control strategy based on the current four-tuple information to obtain the control strategy for the next moment includes: Based on the quadruple information at the current moment, generate the target value corresponding to the quadruple information; The parameters in the first preset function containing the target value are obtained using gradient descent. Perform iterative updates; The parameters after iteration and update Replace the parameters in the current control strategy The control strategy for the next moment is obtained.

5. The method according to claim 3, characterized in that, When the target action is a lane change, updating the current control strategy based on the current four-tuple information to obtain the control strategy for the next moment includes: Obtain the extended quadruple information at the current moment, which corresponds to the extended vehicle condition at the current moment. The extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment using symmetry and monotonic rules. The symmetry rule refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane as the axis. The monotonic rule refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane of the lane change, and / or, changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lane to a preset distance range. The control strategy for the next moment is obtained by updating the current control strategy based on the current quadruple information and the extended quadruple information at the current moment.

6. The method according to claim 5, characterized in that, The step of updating the current control policy to obtain the control policy for the next moment based on the current four-tuple information and the current extended four-tuple information includes: Based on the quaternion information at the current time and the ith quaternion information in the extended quaternion information at the current time, generate the target value corresponding to the ith quaternion information; where i is a positive integer not greater than n, and n is the total number of quaternion information included in the quaternion information at the current time and the extended quaternion information at the current time. The parameters in the second preset function containing the target value corresponding to the i-th quadtuple are obtained by using gradient descent. Perform iterative updates; The parameters after iteration and update Replace the parameters in the current control strategy The control strategy for the next moment is obtained.

7. The method according to claim 3, characterized in that, When the target action is to maintain straight movement, updating the current control strategy based on the current four-tuple information to obtain the control strategy for the next moment includes: The control strategy for the next moment is obtained by updating the current control strategy based on the quadruple information at the current moment, the quadruple information at the historical moment, and the extended quadruple information at the historical moment. The four-tuple information of the historical moment corresponds to the vehicle condition at the historical moment, including: the features of the historical moment, the target action of the historical moment, the reward value corresponding to the target action of the historical moment, and the features of the next moment of the historical moment. The features of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the historical moment, and the features of the next moment of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of the historical moment. The extended four-tuple information of the historical moment corresponds to the extended vehicle condition at the historical moment, which is obtained by processing the vehicle condition at the historical moment using symmetric rules and monotonic rules.

8. The method according to claim 7, characterized in that, The step of updating the current control policy based on the current four-tuple information, the four-tuple information of the historical time, and the extended four-tuple information of the historical time to obtain the control policy for the next time step includes: Based on the j-th quadruple information from the current time quadruple information, the historical time quadruple information, and the historical time extended quadruple information, generate the target value corresponding to the j-th quadruple information; where j is a positive integer not greater than m, and m is the total number of quadruple information included in the current time quadruple information, the historical time quadruple information, and the historical time extended quadruple information; The parameters in the third preset function containing the target value corresponding to the j-th quadtuple are obtained using the gradient descent method. Perform iterative updates; The parameters after iteration and update Replace the parameters in the current control strategy The control strategy for the next moment is obtained.

9. The method according to claim 3, characterized in that, When the target action is a lane change, updating the current control strategy based on the current four-tuple information to obtain the control strategy for the next moment includes: Obtain the extended quadruple information at the current moment; wherein, the extended quadruple information at the current moment corresponds to the extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment with symmetric rules and monotonic rules; Based on the current time quadruple information, the current time extended quadruple information, the historical time quadruple information, and the historical time extended quadruple information, the current control strategy is updated to obtain the control strategy for the next time. The historical time quadruple information corresponds to the vehicle condition at the historical time, and the historical time extended quadruple information corresponds to the historical time extended vehicle condition. The historical time extended vehicle condition is obtained by processing the historical time vehicle condition using symmetric and monotonic rules.

10. An automatic lane-changing device, characterized in that, include: The calculation module is used to calculate the local neighbor features and global statistical features of the autonomous vehicle at the current moment based on the driving information of the autonomous vehicle at the current moment and the movement information of obstacles in each lane within the perception range of the autonomous vehicle. The local neighbor features are used to represent the motion state information of specific neighbor obstacles of the autonomous vehicle relative to the autonomous vehicle; the global statistical features are used to represent the sparsity and density of obstacles in each lane within the perception range at the current moment. The global statistical features include at least one of the following: the average driving speed and average interval of all obstacles in each lane within the perception range at the current moment. Each lane includes: the lane where the autonomous vehicle is located, all lanes to the left of the lane where the autonomous vehicle is located, and all lanes to the right of the lane where the autonomous vehicle is located. The average driving speed represents the average driving speed of obstacles in the lane, and the average interval represents the average interval between two adjacent obstacles in the lane. Both the global statistical features and the local neighbor features are features after discretization processing, and the discrete granularity is related to the speed of the autonomous vehicle. The acquisition module is used to acquire a target action indication based on the local neighbor features, the global statistical features, and the current control strategy. The target action indication is used to instruct the autonomous vehicle to perform a target action, which includes at least two types: lane changing or maintaining straight driving. An execution module is used to execute the target action according to the target action instruction.

11. The apparatus according to claim 10, characterized in that, The device further includes: A feedback module is used to obtain feedback information by executing the target action, and the feedback information is used to update the current control strategy. The feedback information includes the driving information of the autonomous vehicle after executing the target action, the driving information of the autonomous vehicle at the next moment, and the movement information of obstacles in each lane within the perception range of the autonomous vehicle at the next moment. When the target action is a lane change, the feedback information further includes: the ratio of the time to execute the target action to the historical average time, and the change in the density of the lane the autonomous vehicle is in after the lane change compared to the lane it was in before the lane change. The historical average time includes the average time the autonomous vehicle takes to execute similar actions within a preset historical time period. The update module is used to update the current control strategy based on the feedback information to obtain the control strategy for the next moment.

12. The apparatus according to claim 11, characterized in that, The update module includes: The calculation unit is used to calculate the reward value corresponding to the target action based on the feedback information, as well as the local neighbor features and global statistical features of the autonomous vehicle at the next moment; A determining unit is used to determine the quadruple information at the current moment; wherein, the quadruple information at the current moment corresponds to the vehicle condition at the current moment, including: the features at the current moment, the target action, the reward value corresponding to the target action, and the features at the next moment, wherein the features at the current moment include the local neighbor features and global statistical features of the autonomous vehicle at the current moment, and the features at the next moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment; The update unit is used to update the current control strategy based on the quadruple information at the current moment to obtain the control strategy at the next moment.

13. The apparatus according to claim 12, characterized in that, When the target action is to maintain straight movement, the update unit is specifically used for: Based on the quadruple information at the current moment, generate the target value corresponding to the quadruple information; The parameters in the first preset function containing the target value are obtained using gradient descent. Perform iterative updates; The parameters after iteration and update Replace the parameters in the current control strategy The control strategy for the next moment is obtained.

14. The apparatus according to claim 12, characterized in that, When the target action is a lane change, the update unit is specifically used for: Obtain the extended quadruple information at the current moment, which corresponds to the extended vehicle condition at the current moment. The extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment using symmetry and monotonic rules. The symmetry rule refers to symmetrically transforming the positions of obstacles on all lanes to the left and right of the lane where the autonomous vehicle is located, with the lane as the axis. The monotonic rule refers to increasing the distance between the obstacles in front and behind the autonomous vehicle in the target lane of the lane change, and / or, changing the distance between the obstacles in front and behind the autonomous vehicle in the non-target lane to a preset distance range. The control strategy for the next moment is obtained by updating the current control strategy based on the current quadruple information and the extended quadruple information at the current moment.

15. The apparatus according to claim 14, characterized in that, The update unit is specifically used for: Based on the quaternion information at the current time and the ith quaternion information in the extended quaternion information at the current time, generate the target value corresponding to the ith quaternion information; where i is a positive integer not greater than n, and n is the total number of quaternion information included in the quaternion information at the current time and the extended quaternion information at the current time. The parameters in the second preset function containing the target value corresponding to the i-th quadtuple are obtained by using gradient descent. Perform iterative updates; The parameters after iteration and update Replace the parameters in the current control strategy The control strategy for the next moment is obtained.

16. The apparatus according to claim 12, characterized in that, When the target action is to maintain straight movement, the update unit is specifically used for: The control strategy for the next moment is obtained by updating the current control strategy based on the quadruple information at the current moment, the quadruple information at the historical moment, and the extended quadruple information at the historical moment. The four-tuple information of the historical moment corresponds to the vehicle condition at the historical moment, including: the features of the historical moment, the target action of the historical moment, the reward value corresponding to the target action of the historical moment, and the features of the next moment of the historical moment. The features of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the historical moment, and the features of the next moment of the historical moment include the local neighbor features and global statistical features of the autonomous vehicle at the next moment of the historical moment. The extended four-tuple information of the historical moment corresponds to the extended vehicle condition at the historical moment, which is obtained by processing the vehicle condition at the historical moment using symmetric rules and monotonic rules.

17. The apparatus according to claim 16, characterized in that, The update unit is specifically used for: Based on the j-th quadruple information from the current time quadruple information, the historical time quadruple information, and the historical time extended quadruple information, generate the target value corresponding to the j-th quadruple information; where j is a positive integer not greater than m, and m is the total number of quadruple information included in the current time quadruple information, the historical time quadruple information, and the historical time extended quadruple information; The parameters in the third preset function containing the target value corresponding to the j-th quadtuple are obtained using the gradient descent method. Perform iterative updates; The parameters after iteration and update Replace the parameters in the current control strategy The control strategy for the next moment is obtained.

18. The apparatus according to claim 12, characterized in that, When the target action is a lane change, the update unit is specifically used for: Obtain the extended quadruple information at the current moment; wherein, the extended quadruple information at the current moment corresponds to the extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment with symmetric rules and monotonic rules; Based on the current time quadruple information, the current time extended quadruple information, the historical time quadruple information, and the historical time extended quadruple information, the current control strategy is updated to obtain the control strategy for the next time. The historical time quadruple information corresponds to the vehicle condition at the historical time, and the historical time extended quadruple information corresponds to the historical time extended vehicle condition. The historical time extended vehicle condition is obtained by processing the historical time vehicle condition using symmetric and monotonic rules.

19. An automatic lane-changing device, characterized in that, include: Processor and memory; The memory is used to store program instructions; The processor is configured to call and execute program instructions stored in the memory. When the processor executes the program instructions stored in the memory, the automatic lane-changing device is configured to perform the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Vehicle networking based automatic driving control strategy generation method and device

    CN104391504A

  • Automatic driving control method and device, electronic equipment, program and medium

    CN108227710A

  • Lane decision method for automatic driving vehicles based on multi-objective decision matrix

    CN108583578A

  • Autonomous vehicle policy generation

    CN109649402A