Vehicle lane change decision control method, device, equipment and storage medium

Abstract

The application relates to a vehicle lane-changing decision control method, device, equipment and storage medium. The method comprises the following steps: acquiring current state information of a target vehicle, wherein the current state information comprises a motion state of the target vehicle and a motion state of other vehicles within a preset range of the target vehicle; determining a double-experience replay buffer according to the current state information, wherein the double-experience replay buffer is used for storing multiple groups of rule experiences and multiple groups of interaction experiences, the rule experiences are determined according to the current state information and a preset control rule, and the interaction experiences are determined according to the current state information and an initial decision model; training the initial decision model according to the double-experience replay buffer and a reinforcement learning algorithm to obtain a decision model; and determining a target control strategy according to the current state information and the decision model. The method can improve the efficiency and accuracy of control strategy determination.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a vehicle lane-changing decision control method and device, equipment and storage medium. BACKGROUND

[0002] In the field of automatic driving, intelligent decision technology can usually be realized based on deep learning data-driven methods. Reinforcement learning algorithm explores the dynamic characteristics of the environment autonomously, and the agent freely tries and errors in the simulation environment to collect experience data, and trains the deep neural network model through the experience data, so that it learns the driving strategy from the experience data.

[0003] However, when determining the vehicle driving strategy based on the traditional reinforcement learning algorithm, there are many problems such as low sample efficiency and slow strategy training speed. SUMMARY

[0004] Therefore, it is necessary to provide a vehicle lane-changing decision control method, device, equipment and storage medium capable of improving the efficiency and stability of vehicle control strategy determination.

[0005] In a first aspect, the present application provides a vehicle lane-changing decision control method, comprising:

[0006] obtaining current state information of a target vehicle, the current state information comprising the motion state of the target vehicle and the motion state of other vehicles within a preset range of the target vehicle;

[0007] determining a double experience replay buffer according to the current state information, the double experience replay buffer being used to store a plurality of rule experiences and a plurality of interaction experiences, the rule experiences being determined according to the current state information and a preset control rule, and the interaction experiences being determined according to the current state information and an initial decision model;

[0008] training the initial decision model according to the double experience replay buffer and a reinforcement learning algorithm to obtain a decision model;

[0009] determining a target control strategy according to the current state information and the decision model.

[0010] In a second aspect, the present application further provides a vehicle lane-changing decision control device, comprising:

[0011] an obtaining module configured to obtain current state information of a target vehicle, the current state information comprising the motion state of the target vehicle and the motion state of other vehicles within a preset range of the target vehicle;

[0012] The first determining module is configured to determine a double experience replay buffer according to the current state information, the double experience replay buffer being configured to store a plurality of rule experiences and a plurality of interaction experiences, the rule experiences being determined according to the current state information and a preset control rule, and the interaction experiences being determined according to the current state information and an initial decision model;

[0013] The training module is configured to train the initial decision model according to the double experience replay buffer and a reinforcement learning algorithm to obtain a decision model.

[0014] The second determining module is configured to determine a target control strategy according to the current state information and the decision model.

[0015] In a third aspect, the present application also provides a vehicle, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of the first aspect when executing the computer program.

[0016] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the method of any one of the first aspect when executed by a processor.

[0017] In a fifth aspect, the present application also provides a computer program product, which comprises a computer program, and the computer program implements the method of any one of the first aspect when executed by a processor.

[0018] The vehicle lane changing decision control method, device, equipment and storage medium described above, by obtaining the current state information of the target vehicle, the current state information comprising the motion state of the target vehicle and the motion state of other vehicles within a preset range of the target vehicle, determining a double experience replay buffer according to the current state information, the double experience replay buffer being configured to store a plurality of rule experiences and a plurality of interaction experiences, the rule experiences being determined according to the current state information and a preset control rule, and the interaction experiences being determined according to the current state information and an initial decision model, training the initial decision model according to the double experience replay buffer and a reinforcement learning algorithm to obtain a decision model, and determining a target control strategy according to the current state information and the decision model. By being based on a plurality of rule experiences and interaction experiences, meaningless exploration behavior can be reduced based on rule experiences during the training process, the stability and convergence efficiency of the training phase can be improved, and excessive dependence on rules can be avoided, the flexibility of the control strategy can be improved, and thus the efficiency and accuracy of the control strategy determination can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to make the technical solutions in the embodiments of the present application or the related art clearer, the accompanying drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other related drawings can be obtained without creative effort based on these drawings.

[0020] Figure 1 A flowchart of a vehicle lane-changing decision control method in an embodiment is shown in FIG. 1.

[0021] Figure 2 A flowchart of a step of determining a first control strategy in an embodiment is shown in FIG. 2.

[0022] Figure 3 A flowchart of a step of determining a first control strategy in another embodiment is shown in FIG. 3.

[0023] Figure 4 A flowchart of a step of training an initial decision model in an embodiment is shown in FIG. 4.

[0024] Figure 5 A flowchart of a step of determining training samples in an embodiment is shown in FIG. 5.

[0025] Figure 6 A flowchart of a vehicle lane-changing decision control method in another embodiment is shown in FIG. 6.

[0026] Figure 7 A block diagram of a vehicle lane-changing decision control device in an embodiment is shown in FIG. 7.

[0027] Figure 8 An internal structure diagram of a vehicle in an embodiment is shown in FIG. 8. DETAILED DESCRIPTION

[0028] In order to make the technical solutions in the embodiments of the present application or the related art clearer, the accompanying drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other related drawings can be obtained without creative effort based on these drawings.

[0029] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms.

[0030] In an exemplary embodiment, as shown in FIG. 1, a vehicle lane-changing decision control method is provided, which is applied to a vehicle as an example for illustration, and includes the following steps 101 to 104. Wherein: Figure 1

[0031] Step 101, obtaining current state information of a target vehicle. ​

[0032] The current state information includes a motion state of the target vehicle and motion states of other vehicles within a preset range of the target vehicle.

[0033] Optionally, the target vehicle can be any vehicle to be controlled in lane changing decision making. The motion state of the vehicle can be obtained by a sensor arranged on the vehicle, and the motion state of the vehicle is sent to surrounding vehicles through a communication network.

[0034] The other vehicles within the preset range of the target vehicle can include a front vehicle of a lane where the target vehicle is located, a parallel vehicle of a target lane, and a front vehicle of the target lane.

[0035] The motion state of the vehicle includes a global position (x, y), a heading angle , a longitudinal speed , and an acceleration , Therefore, the current state information of the target vehicle includes the motion states of the target vehicle and the other vehicles.

[0036] Exemplarily, the current state information of the target vehicle further includes a longitudinal distance between each of the other vehicles and the target vehicle, and a lateral offset between the target vehicle and the parallel vehicle in the global y direction, wherein the longitudinal distance and the lateral offset are calculated in a local coordinate system of the target vehicle. The current state information of the target vehicle can be represented in the following form:

[0037]

[0038] wherein, is the motion state of the front vehicle, is the motion state of the parallel vehicle of the target lane is the motion state of the front vehicle of the target lane, is the relative longitudinal distance, that is, the longitudinal distance between the front vehicle of the target lane and the target vehicle.

[0039] Step 102, determining a double experience replay buffer according to the current state information.

[0040] The double experience replay buffer is used to store a plurality of sets of rule experience and a plurality of sets of interaction experience. The rule experience is determined according to the current state information and a preset control rule, and the interaction experience is determined according to the current state information and an initial decision model.

[0041] Optionally, in the reinforcement learning algorithm, the experience replay buffer is also called an experience pool, includes a plurality of sets of experience, and can be used in reinforcement learning training. The experiences are time-series associated and have continuity. Each set of experience is recorded in the form of a five-tuple , wherein, represents state information, an action performed, i.e., a control policy of the vehicle, for reward, for state information of the next time, and done is a flag indicating whether the task is terminated.

[0042] wherein the control policy of the vehicle can be a two-dimensional continuous vector, denoted as wherein . The first dimension of the action space corresponds to a steering control signal, which will be scaled to the actual steering angle range of the vehicle in the simulation environment later. The second dimension represents a longitudinal control signal. When > 0, the action is interpreted as a throttle input, and the brake signal is zero; when < 0, the throttle signal is zero, and the brake intensity is .

[0043] Optionally, the experience reward can be determined according to a composite reward function combining both discrete reward and continuous reward components. At each time step, the total reward is calculated by the sum or weighted sum of the individual reward terms. The discrete reward is mainly used to constrain the basic behavior of the vehicle, such as avoiding collision, keeping within the legal lane, and maintaining forward driving; while the continuous reward is used to guide the fine adjustment of the vehicle in terms of spatial position, attitude control, and task completion quality.

[0044] Optionally, a plurality of rule experiences can be generated according to the current state information and preset control rules. The preset control rules can be the control policies that the target vehicle should perform under different state information, and the control rules can include lateral control rules and longitudinal control rules.

[0045] Step 103, training the initial decision model according to the double experience replay buffer and the reinforcement learning algorithm to obtain a decision model.

[0046] In one possible implementation, at the initial stage of training, the policy of the agent is not mature and is prone to produce unstable or even dangerous behavior. In order to ensure the controllability of the learning process, the training samples can be determined entirely relying on the rule experience at the initial stage of training, and the initial decision model is trained. With the advancement of training iterations, the interactive experience can be added to the training samples, so that the training can gradually transition from completely relying on the rule experience to relying on the agent's own experience, realizing the smooth switching from rule-driven to autonomous learning.

[0047] In another possible implementation, the rule experience and the interactive experience in the double experience replay buffer can be mixed, and then the training samples are determined randomly from the mixed experience to train the initial decision model.

[0048] Optionally, the reinforcement learning algorithm can be a Soft Actor-Critic (SAC) algorithm, or a Distributional Soft Actor-Critic with Timprovements (DSACT) algorithm, etc., and the embodiments of the present application do not limit this.

[0049] Exemplarily, taking the DSACT algorithm as an example, in the training process of the initial decision model, the algorithm improves from three aspects of target construction, value estimation and optimization robustness while maintaining the advantages of the maximum entropy framework: first, the variance caused by distribution sampling is reduced by “expected value replacing target”; second, “double distribution Q network” is used to suppress value overestimation and stabilize policy update; finally, “variance adaptive clipping and gradient scaling” is introduced to adaptively control the update step size at different uncertainty stages, so as to realize a smoother convergence process and stronger generalization ability.

[0050] In terms of target construction, since the SAC algorithm directly samples the target from the value distribution, it is easy to introduce high variance and amplify training noise. Therefore, the DSACT uses the expectation of the value distribution as the target value, specifically:

[0051]

[0052] wherein, is the immediate reward, is the discount factor, represents the distributed Q value expectation of the next state-action pair. This processing effectively reduces the variance at the target end without sacrificing the expression of distribution information, bringing more stable critic learning signals.

[0053] In terms of value estimation, in order to suppress the problem of over-aggressive policy caused by overestimation, the SACT trains two independent value distribution links , and selects the one with smaller mean value when constructing the target and updating the policy, which can be expressed as:

[0054]

[0055] This “distributed double clipping” extends the classic Clipped Double Q-learning to the distributed scenario, which not only retains the ability of distribution learning to describe return uncertainty, but also significantly reduces the overestimation risk through conservative target selection, thereby improving the numerical stability of the actor and critic collaborative update.

[0056] In terms of optimizing robustness, considering that the value distribution of different training stages and different state-action regions is not consistent in terms of dispersion degree, the standard deviation of the value distribution is used to adjust the gradient step size Adaptive adjustment of gradient step size. Specifically, define a gradient scaling factor:

[0057]

[0058] And in the parameter update:

[0059]

[0060] Wherein, is the parameter to be updated, is the corresponding loss function. The above mechanism is equivalent to automatically shrinking the step size in the region with high uncertainty (large ) and expanding the effective step size in the region with low uncertainty, thereby achieving the dual effect of "variance adaptive clipping + step size normalization", significantly reducing the probability of oscillation and divergence, and improving the convergence speed and cross-scene generalization performance.

[0061] Step 104, determining a target control strategy according to the current state information and the decision model.

[0062] Optionally, after the training of the initial decision model is completed, the current state information can be input into the trained decision model, and the target control strategy is determined according to the output of the decision model.

[0063] It can be understood that the initial decision model can include an initial policy network and an initial value network, and correspondingly, the trained decision model includes a policy network and a value network. The current state information is input into the policy network to obtain the target control strategy.

[0064] Optionally, the vehicle can directly execute the target control strategy, or can execute the target control strategy after safety verification, and the embodiments of the present application do not limit this.

[0065] The vehicle lane-changing decision control method determines a double experience replay buffer according to the current state information, the double experience replay buffer being used to store a plurality of sets of rule experience and a plurality of sets of interaction experience, the rule experience being determined according to the current state information and a preset control rule, and the interaction experience being determined according to the current state information and an initial decision model; the initial decision model is trained according to the double experience replay buffer and a reinforcement learning algorithm to obtain a decision model; and a target control strategy is determined according to the current state information and the decision model. Based on the plurality of sets of rule experience and the plurality of sets of interaction experience, meaningless exploration behavior can be reduced in the training process based on the rule experience, the stability and convergence efficiency of the training stage are improved, and excessive dependence on rules is avoided, so that the flexibility of the control strategy is improved, thereby improving the efficiency and accuracy of the control strategy determination.

[0066] In one exemplary embodiment, determining the double experience replay buffer according to the current state information comprises: iteratively performing a plurality of times an experience generation process until a preset condition is met, and determining the double experience replay buffer according to the results of the plurality of iterations.

[0067] The i-th experience generation process comprises:

[0068] A first control strategy is determined according to the state information and a preset control rule, the first control strategy is executed in a virtual environment, and a first reward and first next-time state information of the target vehicle are determined; wherein when i is equal to 1, the state information is the current state information, and when i is greater than 1, the state information is the first next-time state information determined in the (i-1)-th experience generation process.

[0069] A second control strategy is determined according to the state information and an initial decision model, the second control strategy is executed in a virtual environment, and a second reward and second next-time state information of the target vehicle are determined; wherein when i is equal to 1, the state information is the current state information, and when i is greater than 1, the state information is the second next-time state information determined in the (i-1)-th experience generation process.

[0070] The preset meeting condition can be that the number of iterations reaches a preset number of iterations.

[0071] Optionally, before the iteration starts, the experience pool is empty, the obtained experience is stored in the experience pool after each experience generation process is completed, and the double experience replay buffer is obtained after the number of iterations reaches the preset number of iterations.

[0072] In one experience generation process, the first control strategy can be taken as an action The first reward can be taken as a reward The first next-time state information can be taken as a next-time state , thereby obtaining a set of rule experiences. The second control policy is taken as an action , the second reward is taken as a reward , the second next time state information is taken as a next time state , thereby obtaining a set of interaction experiences.

[0073] Optionally, the position information of the target vehicle included in the next time state information can be determined according to a simplified dynamics model, according to the state information , the current position information of the target vehicle is determined, the longitudinal velocity and the heading angle are determined according to the control policy, and the position information of the target vehicle at the next time can be represented according to the following formula:

[0074]

[0075] For the surrounding other vehicles, a constant speed assumption model can be used, that is, it is considered that each surrounding vehicle maintains the current speed unchanged in the prediction time domain. Assuming that the current position of a surrounding vehicle is , the longitudinal and lateral velocity components are , the position prediction at the next time is:

[0076]

[0077] The form of the composite reward function in the embodiments of the present application is exemplarily described as follows:

[0078] The discrete rewards in the composite reward function can include R1 to R6, wherein R1 is a forward reward, used to encourage the vehicle to maintain a reasonable driving speed, R2 is a collision penalty, used to strongly punish the collision event, R3 is a boundary crossing penalty, used to ensure that the vehicle always remains within the road range, R4 is an illegal lane penalty, used to prevent reverse driving or driving into an illegal lane, R5 is a bypass reward, used to encourage the vehicle to perform correct lane changing operation, and R6 is a static steering penalty, used to punish the behavior of turning the steering wheel in the static state.

[0079] As shown in Table 1, the trigger conditions and corresponding reward values of each reward are shown, that is, after the control policy is executed, the corresponding discrete reward is determined according to the state information of the target vehicle.

[0080] Table 1

[0081]

[0082] Optionally, in addition to the discrete reward items listed in Table 1, a plurality of continuous reward items are designed to guide the vehicle to smoothly and accurately complete the lane changing to the target lane. In order to encourage the vehicle to gradually approach the center of the target lane, a proximity reward is introduced, which has the form of:

[0083]

[0084] wherein x denotes the longitudinal coordinate of the target vehicle current position, is the normalized distance to the target region. As the vehicle approaches the target region, the reward function value exponentially increases, which is equivalent to generating an "attractive" effect on the vehicle.

[0085] An alignment reward is introduced to keep the vehicle lateral position and heading angle aligned. The reward is composed of the product of two Gaussian functions:

[0086]

[0087] wherein, denotes the lateral deviation of the target vehicle relative to the center of the target lane (in meters), controls the lateral tolerance; denotes the deviation of the target vehicle heading angle relative to the target direction 180° (in degrees), controls the heading tolerance. When the target vehicle lateral position and heading angle are simultaneously aligned, the reward reaches the maximum value, thereby promoting the stability of the target vehicle pose during the lane change.

[0088] In an exemplary embodiment, the control strategy includes a steering control signal, as shown in Figure 2 The first control strategy is determined according to the state information and the preset control rule, including steps 201 to 202. Wherein:

[0089] Step 201, when the state information meets the lane change trigger condition, a fixed steering angle is taken as the steering control signal.

[0090] Optionally, the lane change trigger condition can include triggering the lane change based on the global path planning, there is no vehicle within the preset range before and after the longitudinal position of the target vehicle in the target lane, and the front and rear vehicle distances on the target lane meet the safety distance constraint.

[0091] Wherein, the global path planning triggers the lane change can be that when the global path planning requires to enter a certain target lane (for example, a left-turn exclusive lane), and the target lane is not the current lane, a lane change intention is generated, triggering the lane change; whether there is a vehicle within the preset range before and after the longitudinal position of the target vehicle in the target lane is determined according to the state information of the target vehicle; the front vehicle and the rear vehicle closest to the target vehicle in the target lane are found according to the state information of the target vehicle, the longitudinal distance between the front vehicle and the target vehicle and the longitudinal distance between the rear vehicle and the target vehicle are calculated, and whether the safety distance constraint is met is determined.

[0092] Optionally, a preset fixed steering angle is obtained, and the fixed steering angle is taken as the steering control signal when the lane change trigger condition is met.

[0093] In step 202, when the state information does not satisfy the lane changing trigger condition, a target planning position is obtained, and a steering control signal is determined according to the target planning position.

[0094] Optionally, the target planning position is a path point of the target vehicle at a next moment determined according to global path planning. The steering angle is obtained by geometric relationship.

[0095]

[0096] wherein, is the wheelbase of the target vehicle.

[0097] After the steering angle is determined according to the geometric relationship, the steering angle is taken as the steering control signal.

[0098] The above describes that when the state information satisfies the lane changing trigger condition, a fixed steering angle is taken as the steering control signal. When the state information does not satisfy the lane changing trigger condition, a target planning position is obtained, and a steering control signal is determined according to the target planning position, so that the first control strategy in the rule experience can be accurately determined according to the state information and the preset rule.

[0099] In an exemplary embodiment, the control strategy further includes a longitudinal control signal, and the longitudinal control signal is used to represent a throttle input signal or a brake signal. Figure 3 As shown in FIG. 3, the first control strategy determined according to the state information and the preset control rule further includes steps 301 to 303. Wherein:

[0100] In step 301, when the state information indicates that there is no vehicle in front of the target vehicle, a throttle input signal is determined according to a difference between a speed of the target vehicle and a preset expected speed.

[0101] Optionally, the longitudinal control signal can be determined according to a longitudinal control logic corresponding to the target vehicle, wherein the longitudinal control logic includes three cases: free cruise mode, following mode, and collision avoidance mode.

[0102] When there is no vehicle in front of the target vehicle, the free cruise mode is adopted, and the throttle input signal can be adjusted according to the speed difference. Specifically, a difference between the speed of the target vehicle and the preset expected speed is determined first, and then a product of a preset proportional coefficient and the difference is taken as the throttle input signal.

[0103] In step 302, when the state information indicates that a distance between the target vehicle and a preceding vehicle is less than a preset safety threshold, a brake signal is determined according to a speed of the target vehicle and a speed of the preceding vehicle.

[0104] ​Optionally, when the distance between the target vehicle and the front vehicle is less than the preset safety threshold, the braking signal is used to dynamically correct the speed of the target vehicle, wherein the target speed The target speed can be determined according to the speed of the front vehicle, the speed of the target vehicle, the distance between the target vehicle and the front vehicle, and the preset safety threshold, and can be represented by the following formula:

[0105]

[0106] wherein, is the speed of the front vehicle, and is an adjustment coefficient, is the speed of the target vehicle, and d is the distance between the target vehicle and the front vehicle, is the preset safety threshold.

[0107] In step 303, when the distance between the target vehicle and the front vehicle is less than the preset minimum distance, the preset maximum braking signal is determined as the braking signal.

[0108] Optionally, when the distance between the target vehicle and the front vehicle is less than the preset minimum distance, the preset maximum braking signal is obtained, and the preset maximum braking signal is output as the braking signal.

[0109] In an exemplary embodiment, as shown in Figure 4 the initial decision model is trained according to the double experience replay buffer and the reinforcement learning algorithm to obtain the decision model, including steps 401 to 402. Wherein:

[0110] In step 401, training samples are determined from the multiple sets of rule experience and / or multiple sets of interactive experience according to the current training number.

[0111] Optionally, when the initial decision model is trained according to the reinforcement learning algorithm, a batch of training samples are randomly obtained from the double experience replay buffer, and then the parameters in the initial policy network and the initial value network are updated according to the training samples.

[0112] In one possible implementation, the training samples are determined from the multiple sets of rule experience and / or multiple sets of interactive experience according to the current training number, including: in the case that the current training number is less than a preset threshold, the training samples are determined from the multiple sets of rule experience; in the case that the training number is not less than the preset threshold, a sampling ratio is determined according to the training number and a preset total training number, and the training samples are determined from the multiple sets of rule experience and the multiple sets of interactive experience according to the sampling ratio.

[0113] Optionally, according to the current training number, it is determined whether the training samples are determined from the rule experience or from the rule experience and the interactive experience.

[0114] Optionally, a number of preset training samples is acquired, and then when the number of training times is less than a preset threshold, a plurality of experiences are randomly selected from the rule experiences as the training samples; when the number of training times is not less than the preset threshold, the number of experiences determined from the rule experiences and the number of experiences determined from the interaction experiences are determined according to the number of training samples and a sampling ratio, so as to determine the training samples.

[0115] The sampling ratio can include a first sampling ratio and a second sampling ratio, a part of the training samples can be determined from the rule experiences according to the first sampling ratio, and a part of the training samples can be determined from the interaction experiences according to the second sampling ratio.

[0116] It can be understood that the first sampling ratio corresponding to the rule experiences decreases with the increase of the number of training times.

[0117] In another possible implementation manner, the training samples are determined from the plurality of rule experiences and / or the plurality of interaction experiences according to the current number of training times, including: in a case where the current number of training times is less than a preset threshold, the training samples are determined from the plurality of rule experiences; and in a case where the number of training times is not less than the preset threshold, the training samples are determined from the plurality of interaction experiences.

[0118] Step 402, training the initial decision model according to the training samples and the reinforcement learning algorithm.

[0119] Optionally, the initial decision model is trained according to the training samples and the DSACT algorithm, which will not be described herein again.

[0120] The above-mentioned determination of the training samples from the plurality of rule experiences and / or the plurality of interaction experiences according to the current number of training times and the training of the initial decision model according to the training samples and the reinforcement learning algorithm can rely on the rule experiences to determine the training samples and train the initial decision model at the initial stage of training. With the progress of training iteration, the interaction experiences can be added to the training samples, so that the training can gradually transit from complete dependence on rule experiences to dependence on the experience of the intelligent agent, and realize smooth switching from rule driving to autonomous learning.

[0121] In an exemplary embodiment, as shown in Figure 5 The rule experiences and the interaction experiences both include state information, a control strategy, a reward and next-time state information, the sampling ratio includes a first sampling ratio and a second sampling ratio, and the training samples are determined from the plurality of rule experiences and the plurality of interaction experiences according to the sampling ratio, including the following steps 501 to 503. Wherein:

[0122] Step 501, determining first training samples from the plurality of rule experiences according to the first sampling ratio.

[0123] Optionally, the first difference value can be determined according to the current training number and the preset threshold value, the second difference value can be determined according to the total training number and the preset threshold value, the second sampling ratio can be determined according to the ratio of the first difference value and the second difference value, and the first sampling ratio can be determined according to the difference between 1 and the second sampling ratio.

[0124] The first sampling ratio and the second sampling ratio can be represented by the following formulas

[0125]

[0126]

[0127] wherein t is the current training number, is the preset threshold value, is the total training number.

[0128] Optionally, after the first sampling ratio is determined, the first sample number can be determined according to the product of the first sampling ratio and the total sample number, and then the first sample number of rule experiences can be randomly determined from the rule experiences as the first training samples.

[0129] Step 502, determining the second training samples from the target interaction experience set according to the second sampling ratio, wherein the rewards of each interaction experience in the target interaction experience set are greater than the reward threshold value.

[0130] Optionally, the second sample number can be determined according to the product of the second sampling ratio and the total sample number, and then the second sample number of interaction experiences can be determined from the interaction experiences as the second training samples.

[0131] In one possible implementation manner, the reward threshold value can be determined according to the reward average value of the interaction experiences, the target interaction experience set can be determined according to the experiences with reward values greater than the reward average value, and then the second sample number of interaction experiences can be randomly determined from the target interaction experience set.

[0132] In another possible implementation manner, each interaction experience can be sequentially sorted according to the reward, the reward threshold value is the reward of the interaction experience corresponding to the third sample number, the third sample number is greater than or greater than or equal to the second sample number, and the embodiments of the present application do not perform. Then the first third sample number of interaction experiences are determined as the target interaction experience set.

[0133] wherein when the third sample number is greater than the second sample number, the second training samples can be randomly determined from the target interaction experience set, and when the third sample number is equal to the second sample number, the target interaction experience set can be determined as the second training samples.

[0134] Step 503, determining the training samples according to the first training samples and the second training samples.​​

[0135] Optionally, the first training sample and the second training sample are combined to determine the training sample.

[0136] The first training sample is determined from the multiple sets of rule experience according to the first sampling ratio, the second training sample is determined from the target interaction experience set according to the second sampling ratio, the rewards of each interaction experience in the target interaction experience set are all greater than the reward threshold, and the training sample is determined according to the first training sample and the second training sample. With the advancement of training iteration, the interaction experience can be added to the training sample, so that the training can gradually transition from completely relying on rule experience to relying on the experience of the intelligent agent itself, realizing smooth switching from rule-driven to autonomous learning. At the same time, the hierarchical sampling mechanism based on rewards can improve the training efficiency.

[0137] In one exemplary embodiment, as shown in Figure 6 The vehicle lane-changing decision control method further includes steps 601 to 602. Among them:

[0138] Step 601, determining next time state information according to the target control strategy and the current state information of the target vehicle.

[0139] Optionally, the target control strategy can be executed in a virtual environment based on the current state information of the target vehicle to obtain the next time state information of the target vehicle.

[0140] Exemplarily, the running state of the target vehicle can be determined according to a simplified dynamics model, and the motion state of the surrounding other vehicles can be determined according to a constant speed assumption model, which will not be repeated here.

[0141] Step 602, verifying the target control strategy according to the next time state information, and adjusting the target control strategy if the target control strategy does not meet the preset requirements.

[0142] Among them, verifying the target control strategy according to the next time state information can be determining whether the distance between the target vehicle and the surrounding other vehicles meets the preset safety threshold, and if not, the target control strategy needs to be adjusted.

[0143] Optionally, the distance between the target vehicle and the surrounding other vehicles can be determined according to the Euclidean distance, or can be determined according to the longitudinal distance between the target vehicle and the surrounding other vehicles, which is not limited by the embodiments of the present application.

[0144] In a possible implementation, if the target control strategy does not meet the preset requirement, the target control strategy can not be executed, and the decision model is updated according to the state information at the next moment and the double experience replay buffer, and then a new target control strategy is determined according to the current state information of the target vehicle and the updated decision model, until a target control strategy that meets the preset requirement is obtained.

[0145] In another possible implementation, if the target control strategy does not meet the preset requirement, the longitudinal control signal of the target control strategy can be adjusted according to a preset adjustment rule to obtain a new target control strategy.

[0146] For example, if it is determined according to the state information at the next moment that the distance between the target vehicle and the preceding vehicle is less than a preset safety threshold, the target braking signal is determined according to the distance between the target vehicle and the preceding vehicle and the preset safety threshold, and the target control strategy is adjusted according to the target braking signal.

[0147] The preceding vehicle can be a vehicle in front of the target vehicle in the current lane, or a vehicle in front of the target vehicle in the target lane. The distance between the target vehicle and the preceding vehicle can be determined according to the longitudinal distance.

[0148] When it is determined that the distance between the target vehicle and the preceding vehicle is less than the preset safety threshold, forced deceleration or braking logic can be used, the target braking signal can be determined according to the difference between the preset safety threshold and the distance between the target vehicle and the preceding vehicle, and the target braking signal The target braking signal can be determined by the following formula:

[0149]

[0150] wherein, is the preset safety threshold, is the distance between the target vehicle and the preceding vehicle, indicates that the value range of the target braking signal is between 0 and 1, if is less than 0, 0 is taken as the target braking signal, if is greater than 1, 1 is taken as the target braking signal, if is between 0 and 1, the is taken as the target braking signal.

[0151] The above determines the state information at the next moment according to the target control strategy and the current state information of the target vehicle, checks the target control strategy according to the state information at the next moment, and adjusts the target control strategy if the target control strategy does not meet the preset requirement. Even if the policy network outputs an aggressive action, it can also be screened before execution to ensure that the target vehicle always follows the basic safety constraints.

[0152] It should be understood that although each step in the flowchart involved in the above-described embodiments is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowchart involved in the above-described embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time but can be executed at different times, and the execution of these steps or stages is not necessarily sequential but can be performed alternately or alternately with at least some of the other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0153] Based on the same inventive concept, the embodiments of the present application also provide a vehicle lane change decision control device for implementing the above-mentioned vehicle lane change decision control method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more vehicle lane change decision control device embodiments provided below can refer to the limitations of the vehicle lane change decision control method in the above text, which will not be repeated here.

[0154] In one exemplary embodiment, as shown in Figure 7 A vehicle lane change decision control device 700 is provided, comprising: an acquisition module 701, a first determination module 702, a training module 703, and a second determination module 704, wherein:

[0155] The acquisition module 701 is configured to acquire current state information of the target vehicle, wherein the current state information includes the motion state of the target vehicle and the motion state of other vehicles within a preset range of the target vehicle.

[0156] The first determination module 702 is configured to determine a double experience replay buffer according to the current state information, wherein the double experience replay buffer is used to store a plurality of sets of rule experience and a plurality of sets of interaction experience, the rule experience is determined according to the current state information and a preset control rule, and the interaction experience is determined according to the current state information and an initial decision model.

[0157] The training module 703 is configured to train the initial decision model according to the double experience replay buffer and a reinforcement learning algorithm to obtain a decision model.

[0158] The second determination module 704 is configured to determine a target control strategy according to the current state information and the decision model.

[0159] In one of the embodiments, the training module 703 is specifically configured to determine the training sample from the multiple sets of rule experience and / or the multiple sets of interaction experience according to the current training number; and train the initial decision model according to the training sample and the reinforcement learning algorithm.

[0160] In one of the embodiments, the training module 703 is specifically configured to determine the training sample from the multiple sets of rule experience when the current training number is less than the preset threshold; and determine the sampling ratio according to the training number and the preset total training number when the training number is not less than the preset threshold, and determine the training sample from the multiple sets of rule experience and the multiple sets of interaction experience according to the sampling ratio.

[0161] In one of the embodiments, the rule experience and the interaction experience both include state information, a control policy, a reward, and next-time state information, the sampling ratio includes a first sampling ratio and a second sampling ratio, and the training module 703 is specifically configured to determine a first training sample from the multiple sets of rule experience according to the first sampling ratio; determine a second training sample from a target interaction experience set according to the second sampling ratio, the rewards of the interaction experiences in the target interaction experience set all being greater than a reward threshold; and determine the training sample according to the first training sample and the second training sample.

[0162] In one of the embodiments, the reinforcement learning algorithm is a distributed soft actor critic (DSACT) algorithm.

[0163] In one of the embodiments, the first determination module 702 is specifically configured to iteratively execute the experience generation process multiple times until a preset condition is met, and determine the double experience replay buffer according to the multiple iteration execution results; wherein the i-th experience generation process includes: determining a first control policy according to the state information and a preset control rule, executing the first control policy in the virtual environment, and determining a first reward and a first next-time state information of the target vehicle; wherein when i is equal to 1, the state information is the current state information, and when i is greater than 1, the state information is the first next-time state information determined in the (i-1)-th experience generation process; determining a second control policy according to the state information and the initial decision model, executing the second control policy in the virtual environment, and determining a second reward and a second next-time state information of the target vehicle; wherein when i is equal to 1, the state information is the current state information, and when i is greater than 1, the state information is the second next-time state information determined in the (i-1)-th experience generation process.

[0164] In one of the embodiments, the control policy includes a steering control signal, and the first determination module 702 is specifically configured to take a fixed steering angle as the steering control signal when the state information meets a lane change trigger condition; and obtain a target planning position and determine the steering control signal according to the target planning position when the state information does not meet the lane change trigger condition.

[0165] In one of the embodiments, the control strategy further comprises a longitudinal control signal, the longitudinal control signal being used to represent a throttle input signal or a brake signal, the first determining module 702 is specifically configured to determine the throttle input signal according to a difference between the speed of the target vehicle and a preset expected speed when it is determined according to the state information that there is no vehicle in front of the target vehicle; determine the brake signal according to the speed of the target vehicle and the speed of the preceding vehicle when it is determined according to the state information that the distance between the target vehicle and the preceding vehicle is less than a preset safety threshold; and determine the preset maximum brake signal as the brake signal when it is determined according to the state information that the distance between the target vehicle and the preceding vehicle is less than a preset minimum distance.

[0166] In one of the embodiments, the vehicle lane-changing decision control device further comprises a verification module, configured to determine next-time state information according to the target control strategy and the current state information of the target vehicle; verify the target control strategy according to the next-time state information, and adjust the target control strategy when the target control strategy does not meet preset requirements.

[0167] In one of the embodiments, the verification module is specifically configured to determine a target brake signal according to the distance between the target vehicle and the preceding vehicle and the preset safety threshold when it is determined according to the next-time state information that the distance between the target vehicle and the preceding vehicle is less than the preset safety threshold, and adjust the target control strategy according to the target brake signal.

[0168] The above-mentioned various modules in the vehicle lane-changing decision control device can be realized by software, hardware and combinations thereof in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0169] In one exemplary embodiment, a vehicle is provided, and a block diagram of the internal structure of the vehicle can be as shown in Figure 8 The vehicle includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the vehicle is used to provide computing and control capabilities. The memory of the vehicle includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the vehicle is used to store data. The input / output interface of the vehicle is used to exchange information between the processor and external devices. The communication interface of the vehicle is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a vehicle lane-changing decision control method.

[0170] Those skilled in the art can understand that Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] In an exemplary embodiment, a vehicle is provided, comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of any of the method embodiments described above when executing the computer program.

[0172] In an embodiment, a computer readable storage medium is provided, having stored thereon a computer program, the computer program being executed by a processor to implement the steps of any of the method embodiments described above.

[0173] In an embodiment, a computer program product is provided, comprising a computer program, the computer program being executed by a processor to implement the steps of any of the method embodiments described above.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0175] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0176] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A vehicle lane change decision control method, characterized by, The method comprises: obtaining current state information of a target vehicle, the current state information comprising a motion state of the target vehicle and a motion state of other vehicles within a preset range of the target vehicle; determining a double experience replay buffer according to the current state information, the double experience replay buffer being used to store a plurality of sets of rule experience and a plurality of sets of interaction experience, the rule experience being determined according to the current state information and a preset control rule, and the interaction experience being determined according to the current state information and an initial decision model; in a case where a current training number is less than a preset threshold, determining training samples from the plurality of sets of rule experience; the rule experience and the interaction experience each comprising state information, a control strategy, a reward, and next-time state information; in a case where the training number is not less than the preset threshold, determining a sampling ratio according to the training number and a preset total training number; the sampling ratio comprising a first sampling ratio and a second sampling ratio; determining first training samples from the plurality of sets of rule experience according to the first sampling ratio; determining second training samples from a target interaction experience set according to the second sampling ratio, the reward of each of the interaction experience in the target interaction experience set being greater than a reward threshold; determining training samples according to the first training samples and the second training samples; training the initial decision model according to the training samples and a reinforcement learning algorithm to obtain a decision model; determining a target control strategy according to the current state information and the decision model; the determining of the double experience replay buffer according to the current state information comprises: iteratively executing a plurality of times an experience generation process until a preset condition is met, and determining the double experience replay buffer according to a plurality of iteration execution results; wherein the i-th experience generation process comprises: determining a first control strategy according to state information and the preset control rule, executing the first control strategy in a virtual environment, and determining a first reward and first next-time state information of the target vehicle; wherein, when i is equal to 1, the state information is the current state information, and when i is greater than 1, the state information is the first next-time state information determined in the (i-1)-th experience generation process; determining a second control strategy according to the state information and the initial decision model, executing the second control strategy in the virtual environment, and determining a second reward and second next-time state information of the target vehicle; wherein, when i is equal to 1, the state information is the current state information, and when i is greater than 1, the state information is the second next-time state information determined in the (i-1)-th experience generation process.

2. The method of claim 1, wherein, The reinforcement learning algorithm is a distributed soft actor critic (DSACT) algorithm.

3. The method of claim 1, wherein, The control strategy comprises a steering control signal, and the determining of the first control strategy according to the state information and the preset control rule comprises: in a case where the state information satisfies a lane change triggering condition, taking a fixed steering angle as the steering control signal; in a case where the state information does not satisfy the lane change triggering condition, obtaining a target planning position, and determining the steering control signal according to the target planning position.

4. The method of claim 3, wherein, The control strategy further comprises a longitudinal control signal used to represent a throttle input signal or a brake signal, and the determining the first control strategy according to the state information and the preset control rule further comprises: determining the throttle input signal according to a difference between the speed of the target vehicle and a preset expected speed when it is determined according to the state information that there is no vehicle in front of the target vehicle; determining the brake signal according to the speed of the target vehicle and the speed of the front vehicle when it is determined according to the state information that the distance between the target vehicle and the front vehicle is less than a preset safety threshold; determining the brake signal according to the preset maximum brake signal when it is determined according to the state information that the distance between the target vehicle and the front vehicle is less than a preset minimum distance.

5. The method of claim 1, wherein, The method further comprises: determining next time state information according to the target control strategy and the current state information of the target vehicle; verifying the target control strategy according to the next time state information, and adjusting the target control strategy when the target control strategy does not meet preset requirements.

6. The method of claim 5, wherein, The adjusting the target control strategy when the target control strategy does not meet preset requirements comprises: determining a target brake signal according to the distance between the target vehicle and the front vehicle and the preset safety threshold when it is determined according to the next time state information that the distance between the target vehicle and the front vehicle is less than the preset safety threshold, and adjusting the target control strategy according to the target brake signal.

Images