Automatic driving strategy and system for lane-changing scenario considering adversarial attack

Abstract

The application discloses a lane-changing scene automatic driving strategy and system considering counter-attacks, comprising: acquiring state information of an automatic driving vehicle at a current time and behavior patterns of other vehicles in the same scene as the automatic driving vehicle as counter-intelligent agents; performing Markov game on the automatic driving vehicle and the counter-intelligent agents to obtain an execution action of the automatic driving vehicle; determining a decision result of the vehicle based on the output execution action; selecting a corresponding navigation point according to the decision result, a current running speed and a distance from a preceding vehicle; and generating a driving track of the vehicle by using a PID controller based on the navigation point and outputting corresponding motion control information. The application can complete lane-changing operation in the case that the automatic driving vehicle is attacked, detect vehicle decision by generating multiple counter-intelligent agents, retrain the automatic driving vehicle to prevent counter-attacks and enhance the robustness of the decision, and repeat the two steps to enhance the robustness of the system.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing. Specifically, it relates to an autonomous driving strategy and system for lane-changing scenarios that takes into account adversarial attacks. Background Technology

[0002] With the rapid development and widespread application of autonomous driving technology, its safety and reliability have become a major concern. Autonomous driving systems rely on a large number of sensors and complex algorithms to make decisions, and these components can all become targets for attack.

[0003] Autonomous driving faces various complex traffic environments, and lane-changing decisions are a critical operation. Malicious attacks on these decisions could lead to serious traffic accidents. Furthermore, there are cybersecurity threats. Hackers and other malicious attackers may attempt to interfere with and disrupt the lane-changing process of autonomous driving systems through various means, employing black-box or white-box attack methods to achieve malicious purposes.

[0004] Existing adversarial attack techniques for autonomous driving lane-changing have the following problems: 1. Poor adversarial decision-making effectiveness: When playing against attackers, it is difficult to make reasonable decisions, or the adversarial strategies have low robustness. 2. Vehicle perception noise: Unpredictable perception errors or measurement noise may mislead autonomous vehicles into making unsafe decisions, or even lead to catastrophic failures. 3. Inadequate detection and defense technologies: Defense mechanisms may not be flexible enough to cope with complex and ever-changing attack forms.

[0005] Therefore, an autonomous driving method is needed to improve the safety of vehicles making lane-changing decisions in attack scenarios. Summary of the Invention

[0006] This invention is proposed based on the above-mentioned needs of the prior art. The technical problem to be solved by this invention is to provide an autonomous driving strategy and system that considers adversarial attacks in lane-changing scenarios to improve the safety and reliability of vehicles in performing lane-changing decisions in attack scenarios, enhance the robustness of autonomous driving, reduce potential accident risks, and make it more adaptable to complex traffic environments.

[0007] To solve the above problems, the present invention is implemented using the following technical solution:

[0008] An autonomous driving strategy for lane-changing scenarios considering adversarial attacks, the method includes: acquiring the current state information of the autonomous vehicle and the behavior patterns of other vehicles in the same scenario as the autonomous vehicle, acting as adversarial agents; performing a Markov game between the autonomous vehicle and the adversarial agents to obtain the autonomous vehicle's execution actions, including: Step 1: freezing the parameters of a first policy network and optimizing the parameters of a second policy network by changing the behavior patterns of other vehicles to minimize the cumulative reward; Step 2: freezing the parameters of the second policy network and optimizing the parameters of the first policy network to maximize the cumulative reward; Step 3: processing the current state information using the first policy network to obtain the corresponding execution actions; repeating steps 1 to 3 until a preset number of loops is reached or the parameters no longer change, and outputting the execution actions; determining the vehicle's decision result based on the output execution actions, the decision result including following the preceding vehicle, waiting state, or lane-changing state; selecting the corresponding waypoint according to the decision result, the current operating speed, and the distance to the preceding vehicle; and generating the autonomous vehicle's driving trajectory using a PID controller based on the waypoints and outputting the corresponding motion control information.

[0009] Optionally, the optimization objective of the second policy network is: Among them, J(S) t ,π A S represents the optimization objective of the second policy network. t Represents the current state information, π A Let represent the second policy network, t represent the t-th time step, T0 represent the current time step, and γ represent the penalty term, with a value range of γ∈[0,1]. R represents the expected penalty for the current step time. E,t Let R represent the reward function for autonomous vehicles, α represent the adversarial factor, and R represent the reward function for autonomous vehicles. reg This represents the regularization term.

[0010] Optionally, the vehicle's decision result is determined based on the executed action of the output, and its expression is: Where π represents the first policy network, E represents the expected value, T is the total time step, t represents the t-th time step, and r(s) t ,a t ) indicates that in state s t Choose action a t The immediate reward, c represents the constraint function, s represents the current state, s ′ Let represent the state at the next moment, Δ represent the observed disturbance, and ∈ represent the expected minimum deviation.

[0011] Optionally, the cumulative reward includes sub-goal rewards and total task rewards; the sub-goal rewards include: the faster the autonomous vehicle reaches the destination, the greater the time reward; when the autonomous vehicle moves away from the expected destination or stops, the time reward decreases; when the autonomous vehicle enters an unsafe area, there is an unsafety penalty; the total task rewards include: if the autonomous vehicle collides, there is a collision penalty; if the autonomous vehicle successfully reaches the destination, there is a success reward; if the autonomous vehicle takes longer than a preset value to reach the destination, there is a timeout penalty.

[0012] Optionally, when the decision result is to follow or wait for the vehicle in front, waypoints are selected based on traffic density; when the decision result is to change lanes, waypoints are selected based on the speed of the autonomous vehicle.

[0013] Optionally, before steps one to three, the interaction count between the adversarial agent and the autonomous vehicle is updated, including: initially, randomly selecting n adversarial agents, each with the same number of interactions with the autonomous vehicle; obtaining the number of times each adversarial agent successfully confronts the autonomous vehicle and calculating the deduction value; ranking the adversarial agents from low to high based on the deduction value; changing the interaction count between the corresponding adversarial agent and the autonomous vehicle based on the ranking of different adversarial agents, the higher the ranking, the more challenging the adversarial agent, and the more interactions with the autonomous vehicle; during the execution of steps one to three, using the changed interaction count, allowing the adversarial agent to interact with the autonomous vehicle, and updating the interaction count again.

[0014] Optionally, it further includes: acquiring the state information of the autonomous vehicle over a historical time period; inputting the state information of the historical time period into an extended Kalman filter for training to compensate for observation noise in the state space; obtaining the predicted state information of the next moment through a reinforcement learning algorithm for the output execution action; and correcting the predicted state information of the next moment using the trained extended Kalman filter.

[0015] Optionally, the adversarial factor α is adjusted to generate multiple adversarial agents with different adversarial strengths, which then interact with the autonomous vehicle to stimulate the autonomous vehicle to make decision-making errors.

[0016] Optionally, the first policy network and the second policy network are trained using the epsilon-greedy reinforcement learning algorithm.

[0017] An autonomous driving system for lane-changing scenarios considering adversarial attacks includes: an acquisition module, which acquires the current state information of the autonomous vehicle and the behavior patterns of other vehicles in the same scenario as the autonomous vehicle, acting as an adversarial agent; an attack module, which performs a Markov game between the autonomous vehicle and the adversarial agent to obtain the autonomous vehicle's execution action, including: Step 1: Freezing the parameters of a first policy network and optimizing the parameters of a second policy network by changing the behavior patterns of other vehicles to minimize the cumulative reward; Step 2: Freezing the parameters of the second policy network and optimizing the parameters of the first policy network to maximize the cumulative reward; Step 3: Processing the current state information using the first policy network to obtain the corresponding execution action; repeating steps 1 to 3 until a preset number of loops is reached or the parameters no longer change, and outputting the execution action; an action decision module, which determines the vehicle's decision result based on the output execution action, the decision result including following the preceding vehicle, waiting state, or lane-changing state; and a trajectory planning module, which selects the corresponding waypoints based on the decision result, the current operating speed, and the distance to the preceding vehicle; and generating the autonomous vehicle's driving trajectory using a PID controller based on the waypoints and outputting the corresponding motion control information.

[0018] Compared with existing technologies, this invention proposes an autonomous driving strategy and system for lane-changing scenarios that considers adversarial attacks, enabling lane-changing operations to be completed even when the autonomous vehicle is attacked. The autonomous vehicle is trained using a Markov game-based approach with multiple adversarial agents to obtain the maximum expected reward. A hierarchical structure of decision-making, waypoint selection, and trajectory planning determines the driving path and motion state planning. Multiple adversarial agents are generated to detect erroneous vehicle decisions; the autonomous vehicle is retrained to defend against adversarial attacks and enhance its decision-making robustness; these two steps are repeated to further enhance the system's robustness. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of an autonomous driving strategy for lane-changing scenarios that takes into account adversarial attacks, provided in this embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the scope of protection of the present invention.

[0023] Example 1

[0024] This embodiment provides an autonomous driving strategy for lane-changing scenarios that considers adversarial attacks, including:

[0025] S1 acquires the current state information of the autonomous vehicle and the behavior patterns of other vehicles in the same scene as the autonomous vehicle, acting as an adversarial intelligent agent.

[0026] S2 performs a Markov game between the autonomous vehicle and the adversarial agent to obtain the actions to be performed by the autonomous vehicle.

[0027] This embodiment relates to a reinforcement learning algorithm. In the reinforcement learning training process for autonomous driving, the agent needs to maximize the expected reward while satisfying conditional constraints. The basic idea of ​​reinforcement learning is a Markov decision process. In each time step, the agent interacts with the current environmental state, performs actions, and transitions to a new state. The agent receives a reward based on the state transitions and actions taken. The sequence of state, action, and reward transitions forms a Markov trajectory. The goal of the reinforcement learning agent is to optimize the overall policy to obtain a greater reward value throughout the Markov trajectory.

[0028] To detect erroneous vehicle decisions, this embodiment improves the reinforcement learning algorithm. Specifically, it designs two participants: an autonomous vehicle M and an adversarial agent N that controls the scene parameters. The autonomous vehicle M's policy objective is to optimize for maximum cumulative reward to make better decisions, thus offsetting the adversarial agent's reward value. The adversarial agent N opposes the autonomous vehicle, and its policy objective is to minimize the autonomous vehicle M's cumulative reward, causing the autonomous vehicle to make incorrect decisions. Specifically, it opposes the autonomous vehicle by altering the behavior patterns of other vehicles in the future, such as changing vehicle speed or acceleration.

[0029] To achieve the above process, this embodiment constructs a model, which includes a first policy network and a second policy network. The first policy network reflects the decision-making network of the autonomous vehicle, and the second policy network reflects the decision-making network of the agent. The model uses seven elements from the set (S, A, p, r, c, Δ, γ) for modeling. Here, S is the set of states, called the state space, where the current state s∈S; A is the set of actions, called the action space, where action a∈A; p is the transition probability distribution of the next state; r: S×A→R represents the reward function; c represents the constraint function; Δ represents the observation perturbation, which is associated with the second policy network; and γ∈(0,1) is the penalty factor. The overall optimization objective of the model is:

[0030]

[0031] Among them, S t Represents the current state information, π A Represents the second policy network, π E G represents the first-policy network. E,t γ represents the cumulative reward for the autonomous vehicle. The cumulative score is the sum of the expected penalty and reward for the current step time T0, where t represents the t-th time step, T0 represents the current time step, and γ represents the penalty term, with a value range of γ∈[0,1]. R represents the expected penalty for the current step time. The longer t is from the current moment, the less important the immediate reward becomes. E,t This represents the reward function for autonomous vehicles.

[0032] The specific steps for training the model are as follows:

[0033] The S200 freezes the parameters of the first policy network with the goal of minimizing cumulative rewards, and optimizes the parameters of the second policy network by changing the behavior patterns of other vehicles.

[0034] By training adversarial agents, the vehicle's decisions are treated as an integral part of the environment, thus reducing the two-player game to a Markov game involving adversarial agents. In this embodiment, multiple adversarial agents can be generated to detect erroneous decisions made by the autonomous vehicle.

[0035] The optimization objective of the second strategy network is:

[0036]

[0037] Among them, S t Represents the current state information, π A R represents the second policy network. A,t The reward function represents the agent.

[0038] In one implementation, the reward function for the adversarial agent is established as: R A,t =-R E,t

[0039] In zero-sum games, irrational adversarial behavior may occur. To address this issue, another implementation incorporates a regularization term R. reg When the adversarial factor α is incorporated into the agent's reward mechanism, the established agent reward function is: R A,t =-αR E,t +R reg , where R reg The rationality of the constraint behavior, α controls the degree of adversarial behavior, the larger the value of α, the stronger the adversarial behavior of the agent.

[0040] The adjusted reward function ensures a balance between adversarial and real-world behavior. Over-aggression can be prevented by adjusting the intensity of adversarial behavior, while policies under various adversarial intensities can be trained to test vehicle driving performance. Therefore, multiple different adversarial agents can be generated.

[0041] S210 freezes the parameters of the second strategy network and optimizes the parameters of the first strategy network with the goal of maximizing cumulative rewards.

[0042] In this step, autonomous vehicles are trained to defend against adversarial attacks and enhance the robustness of their decision-making.

[0043] For the overall optimization objective, the parameters of the second policy network remain unchanged, and the goal is to find the parameters of the first policy network that maximize the cumulative reward of the autonomous vehicle.

[0044] S220 performs first policy network processing on the current state information to obtain the actions to be performed by the autonomous vehicle.

[0045] S230 repeats S200 and S220 until the preset number of loops is reached or the parameters no longer change, then outputs the executed action.

[0046] Multiple adversarial agents with different adversarial strengths are trained to identify decision-making errors in driving vehicles, and their robustness is enhanced by retraining to counteract these adversarial agents.

[0047] The vehicle retraining process based on deep reinforcement learning involves correcting the adversarial agent and updating the policy to defend against adversarial attacks. In this stage, the adversarial agent is considered an integral part of the surrounding environment, and the two-player game becomes a Markov decision process for the vehicle. The vehicle continuously corrects previous erroneous decisions and attempts to defend against adversarial attacks during retraining.

[0048] Furthermore, randomly selecting adversarial agents in each training cycle leads to low efficiency in the retraining process. Therefore, a priority adversarial selection method is introduced to address this issue. This method prioritizes more challenging adversarial agents in each training period. Specifically, initially, n adversarial agents are randomly selected, each interacting with the autonomous vehicle an equal number of times. The number of times each adversarial agent defeats the autonomous vehicle is obtained, and a penalty score is calculated. Based on the penalty scores, the adversarial agents are ranked from low to high. Based on the ranking of different adversarial agents, the interaction count of the corresponding adversarial agent is changed. The higher the ranking, the more challenging the adversarial agent, and the more interactions it has with the autonomous vehicle. During execution, the changed interaction count is used to allow the adversarial agent to interact with the autonomous vehicle, and the interaction count is updated again. In summary, the failure score and interaction count are updated in each training cycle.

[0049] This embodiment uses a hybrid reward mechanism as the training standard, dividing the cumulative reward into total task reward and sub-objective reward. Because the PID controller guarantees a smooth trajectory, no non-smoothness penalty is used. The total task reward and sub-objective reward are as follows:

[0050] In the sub-goal reward, for each time step, there is a time penalty -σ1 and a time step reward σ2 for moving towards the final destination. The faster the autonomous vehicle reaches the destination, the greater the time reward; when the autonomous vehicle moves away from the expected destination or stops moving, the time reward decreases; when the autonomous vehicle enters an unsafe area, there is an unsafe penalty exp-(d tr The unsafe area includes braking unsafe areas caused by vehicle physical limitations and road conditions.

[0051] In the overall task reward, for the termination condition, if the autonomous vehicle collides, there is a collision penalty -σ4; if the autonomous vehicle successfully reaches the destination, there is a success reward σ5; if the autonomous vehicle takes longer than a preset value to reach the destination, there is a timeout penalty -σ6; the termination condition is that the vehicle collides or the vehicle successfully reaches the destination.

[0052] This embodiment employs an epsilon-greedy reinforcement learning training strategy, aiming to balance exploration and exploitation. The adversarial agent's willingness to explore new strategies decreases with increasing iterations. Average cumulative reward is used for adjustment during training; when the average reward is high over a period, the reward value is reduced, and vice versa. This reduces convergence time and helps autonomous vehicles complete more exploration during training in complex scenarios.

[0053] S3 determines the vehicle's decision based on the executed actions of the output.

[0054] The decision outcome includes following the vehicle in front, waiting, or changing lanes.

[0055] The decision result for the vehicle is determined based on the executed action of the output, and its expression is:

[0056]

[0057] stE[c(s,s ′ ,Δ)]≤∈,

[0058] Where π represents the first policy network, E represents the expected value, T is the total time step, t represents the t-th time step, and r(s) t ,a t ) indicates that in state s t Choose action a t The immediate reward, c represents the constraint function, s represents the current state, s ′ Let represent the state at the next moment, Δ represent the observed disturbance, and ∈ represent the expected minimum deviation.

[0059] S4 selects the corresponding waypoint based on the decision result, current operating speed, and distance to the vehicle ahead.

[0060] Based on the selected decision, the trajectory planner chooses the final waypoint from the discrete waypoints. All final waypoints are generated at the center of the lane to prevent trajectory deviation and ensure stability. Once the final waypoint is selected, the target speed of the autonomous vehicle is calculated using maximum acceleration and deceleration constraints to ensure a smooth trajectory.

[0061] S5 uses waypoints and a PID controller to generate the driving trajectory of the autonomous vehicle and output the corresponding motion control information.

[0062] The target speed and final waypoint value are provided to the PID controller, which then generates longitudinal and lateral control. These sub-trajectories together constitute a complete operating trajectory, encompassing the full range of operations including following, waiting, and lane changing.

[0063] When the decision is to follow or wait for the vehicle ahead, waypoints are selected based on traffic density. The trajectory planner generates trajectories of variable length according to different driving scenarios. When traffic density is high, the autonomous vehicle selects a closer final waypoint, allowing it to plan sub-trajectories in a shorter time. In scenarios with low traffic density, the final waypoint location of the autonomous vehicle will be farther away. After planning, the final waypoint and target speed information of all trajectories are provided to the PID controller to achieve smooth trajectory tracking.

[0064] When the decision outcome is a lane change, a constrained Markov decision process is used to select a waypoint in the target lane using the autonomous vehicle's state information. When the autonomous vehicle's speed is high, a faster lane change operation is required, so a longer trajectory is selected; when the autonomous vehicle's speed is low, a shorter trajectory is selected to complete a sharper turn.

[0065] Furthermore, the model also includes an Extended Kalman Filter (EKF) to obtain the state information of the autonomous vehicle over historical time periods; the state information over historical time periods is input into the EKF for training to compensate for observation noise in the state space and mitigate the impact on dynamic driving behavior; the output execution action is used to obtain the predicted state information of the next moment through a reinforcement learning algorithm; and the trained Extended Kalman Filter is used to correct the predicted state information of the next moment.

[0066] Furthermore, historical state data can be used as input to a two-layer LSTM in the model to compensate for observation noise in the state space and improve learning efficiency under interactive modes and random driving conditions. To gain experience from historical data, a random update method is used, randomly extracting a sequence from the historical experience data to train the neural network.

[0067] Based on the above method, this embodiment uses an interactive lane-changing scenario for experimentation and simulation, simulating high, medium, and low traffic flow scenarios. In some training scenarios, self-channels are randomly disabled to increase the complexity of road conditions.

[0068] To simulate real-world driving behavior, multiple vehicles and their behavioral patterns were randomly generated using the traffic manager module of the open-source autonomous driving simulator CARLA. The traffic manager, based on a PID control algorithm, set all vehicles to autonomous driving mode. In this mode, all vehicles interact with other vehicles and react to their own actions, making the environment highly interactive. The "own vehicle" refers to the autonomous vehicle in steps S1-S5, and the behavioral patterns of other vehicles refer to the adversarial agents in steps S1-S5. Furthermore, the traffic manager can increase vehicle speed, reduce the distance to the vehicle in front, and randomly assign threat states to vehicles.

[0069] In the simulation scenario, an autonomous driving vehicle, along with its nearest target vehicle and obstacle vehicle, are selected as the state observation targets. Information about these three vehicles is contained in B∈{B1,B2,B3}, where B1, B2, and B3 represent the information of the autonomous driving vehicle, the target vehicle, and the obstacle vehicle, respectively. Based on this, the state space of the autonomous driving vehicle is constructed, including…

[0070] s = [v e ,lane ide ,v t ,d t ,d tr ,lane idt ]

[0071] Among them, v e Lane indicates the speed of the vehicle itself. ide The ID of the lane in which the vehicle is located, v t d represents the speed of the target vehicle. t d represents the distance between the vehicle and the target vehicle. tr Lane represents the gap between the lane and the safety threshold. idt This indicates the ID of the lane where the obstacle vehicle and the target vehicle are located. The target vehicle includes the vehicle adjacent to the autonomous vehicle after it changes lanes to the target lane, and the obstacle vehicle refers to a vehicle in the same lane as the target vehicle that obstructs the autonomous vehicle from changing lanes.

[0072] By interacting with other vehicles, the autonomous vehicle completes a Markov game with an adversarial agent, obtains the actions performed by the autonomous vehicle, and thus obtains the trajectory of the autonomous vehicle.

[0073] Compared with existing technologies, this embodiment proposes an autonomous driving strategy for lane-changing scenarios that considers adversarial attacks, enabling lane-changing operations to be completed even when the autonomous vehicle is attacked. The autonomous vehicle is trained using a Markov game-based approach with multiple adversarial agents to obtain the maximum expected reward. A hierarchical structure of decision-making, waypoint selection, and trajectory planning determines the driving path and motion state planning. A Long Short-Term Memory (LSTM) layer is used to mitigate the impact of observation noise and dynamic driving behavior. Multiple adversarial agents are generated to detect erroneous vehicle decisions; the autonomous vehicle is retrained to defend against adversarial attacks and enhance its decision-making robustness; these two steps are repeated to further enhance the system's robustness.

[0074] Example 2

[0075] This embodiment provides an autonomous driving system for lane-changing scenarios considering adversarial attacks to implement the method described in Embodiment 1. The system includes:

[0076] The acquisition module acquires the current state information of the autonomous vehicle and the behavior patterns of other vehicles in the same scene as the autonomous vehicle, acting as an adversarial intelligent agent.

[0077] The attack module performs a Markov game between the autonomous vehicle and the adversarial agent to obtain the autonomous vehicle's actions. The steps include: Step 1: Freezing the parameters of the first policy network to minimize the cumulative reward, and optimizing the parameters of the second policy network by changing the behavior patterns of other vehicles; Step 2: Freezing the parameters of the second policy network to maximize the cumulative reward, and optimizing the parameters of the first policy network; Step 3: Processing the current state information using the first policy network to obtain the corresponding action; Repeating steps 1 to 3 until a preset number of iterations is reached or the parameters no longer change, then outputting the action.

[0078] The action decision module determines the vehicle's decision result based on the output execution action. The decision result includes following the vehicle in front, waiting, or changing lanes.

[0079] The trajectory planning module selects the corresponding waypoints based on the decision results, the current operating speed, and the distance to the vehicle in front; based on the waypoints, it uses a PID controller to generate the driving trajectory of the autonomous vehicle and outputs the corresponding motion control information.

[0080] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An autonomous driving strategy for lane-changing scenarios considering adversarial attacks, characterized in that, include: The autonomous vehicle's current state information and the behavior patterns of other vehicles in the same scene as the autonomous vehicle are acquired to act as an adversarial intelligent agent. A Markov game is played between an autonomous vehicle and an adversarial agent to determine the actions to be taken by the autonomous vehicle. The steps include: Step 1: Freezing the parameters of the first policy network and minimizing the cumulative reward as the optimization objective of the second policy network. The parameters of the second policy network are then optimized by changing the behavior patterns of other vehicles. The optimization objective of the second policy network is: Among them, J(S) t ,π A S represents the optimization objective of the second policy network. t Represents the current state information, π A Let represent the second policy network, t represent the t-th time step, T0 represent the current time step, and γ represent the penalty term, with values ​​ranging from γ∈[0,1]. t-T0 R represents the expected penalty for the current step time. E,t Let R represent the reward function for autonomous vehicles, α represent the adversarial factor, and R represent the reward function for autonomous vehicles. reg Step 1: Represents the regularization term; Step 2: Freeze the parameters of the second policy network and optimize the parameters of the first policy network with the goal of maximizing the cumulative reward; Step 3: Process the current state information using the first policy network to obtain the corresponding execution action; Repeat steps 1 to 3 until the preset number of loops is reached or the parameters no longer change, and output the execution action; The vehicle's decision-making outcome is determined based on the executed action of the output, and the decision-making outcome includes following the vehicle in front, waiting, or changing lanes. Based on the decision result, current operating speed, and distance to the vehicle in front, the corresponding waypoint is selected. When the decision result is to follow or wait for the vehicle in front, the waypoint is selected based on traffic density; when the decision result is to change lanes, the waypoint is selected based on the speed of the autonomous vehicle. Based on the waypoints, the PID controller is used to generate the driving trajectory of the autonomous vehicle and output the corresponding motion control information.

2. The autonomous driving strategy for lane-changing scenarios considering adversarial attacks according to claim 1, characterized in that, The decision result for the vehicle is determined based on the executed action of the output, and its expression is: s.t.E[c(s,s ′ ,Δ)]≤∈, Where π represents the first policy network, E represents the expected value, T is the total time step, t represents the t-th time step, and r(s) t ,a t ) indicates that in state s t Choose action a t The immediate reward, c represents the constraint function, s represents the current state, s ′ Let represent the state at the next moment, Δ represent the observed disturbance, and ∈ represent the expected minimum deviation.

3. The autonomous driving strategy for lane-changing scenarios considering adversarial attacks according to claim 1, characterized in that, Cumulative rewards include sub-goal rewards and total task rewards; The sub-goal rewards include: the faster the autonomous vehicle reaches the destination, the greater the time reward; when the autonomous vehicle moves away from the expected destination or stops moving, the time reward decreases; when the autonomous vehicle enters an unsafe area, there is an unsafe penalty. The total task reward includes: a collision penalty if the autonomous vehicle collides; a success reward if the autonomous vehicle successfully reaches its destination; and a timeout penalty if the autonomous vehicle takes longer than a preset time to reach its destination.

4. The autonomous driving strategy for lane-changing scenarios considering adversarial attacks according to claim 1, characterized in that, Before steps one through three, the interaction count between the adversarial agent and the autonomous vehicle is updated, including: initially, randomly selecting n adversarial agents, each with the same number of interactions with the autonomous vehicle; obtaining the number of times each adversarial agent successfully confronts the autonomous vehicle and calculating the deduction value; ranking the adversarial agents from low to high based on the deduction value; and changing the interaction count between the corresponding adversarial agent and the autonomous vehicle based on the ranking of different adversarial agents. The higher the ranking, the more challenging the adversarial agent is, and the more interactions it has with the autonomous vehicle. During steps one through three, the modified number of interactions is used to allow the adversarial agent to interact with the autonomous vehicle, and the number of interactions is updated again.

5. The autonomous driving strategy for lane-changing scenarios considering adversarial attacks according to claim 1, characterized in that, Also includes: Obtain the status information of autonomous vehicles over historical time periods; The state information of historical time periods is input into the extended Kalman filter for training; The output action is executed to obtain the predicted state information for the next moment through a reinforcement learning algorithm; The trained extended Kalman filter is used to correct the predicted state information for the next time step.

6. The autonomous driving strategy for lane-changing scenarios considering adversarial attacks according to claim 1, characterized in that, Multiple adversarial agents with varying adversarial strengths are generated by adjusting the adversarial factor α, and these agents interact with autonomous vehicles to stimulate the vehicles to make incorrect decisions.

7. The autonomous driving strategy for lane-changing scenarios considering adversarial attacks according to claim 1, characterized in that, The first and second policy networks were trained using the epsilon-greedy reinforcement learning algorithm.

8. An autonomous driving system for lane-changing scenarios considering adversarial attacks, characterized in that, include: The acquisition module acquires the current state information of the autonomous vehicle and the behavior patterns of other vehicles in the same scene as the autonomous vehicle, acting as an adversarial intelligent agent. The attack module performs a Markov game between the autonomous vehicle and the adversarial agent to obtain the actions taken by the autonomous vehicle. This includes: Step 1: Freezing the parameters of the first policy network and minimizing the cumulative reward as the optimization objective of the second policy network. The parameters of the second policy network are then optimized by changing the behavior patterns of other vehicles. The optimization objective of the second policy network is: Among them, J(S) t ,π A S represents the optimization objective of the second policy network. t Represents the current state information, π A Let represent the second policy network, t represent the t-th time step, T0 represent the current time step, and γ represent the penalty term, with values ​​ranging from γ∈[0,1]. t-R0 R represents the expected penalty for the current step time. E,t Let R represent the reward function for autonomous vehicles, α represent the adversarial factor, and R represent the reward function for autonomous vehicles. reg Step 1: Represents the regularization term; Step 2: Freeze the parameters of the second policy network and optimize the parameters of the first policy network with the goal of maximizing the cumulative reward; Step 3: Process the current state information using the first policy network to obtain the corresponding execution action; Repeat steps 1 to 3 until the preset number of loops is reached or the parameters no longer change, and output the execution action; The action decision module determines the vehicle's decision result based on the output execution action, and the decision result includes following the vehicle in front, waiting state, or lane change state; The trajectory planning module selects corresponding waypoints based on the decision result, current operating speed, and distance to the vehicle in front. When the decision result is to follow or wait for the vehicle in front, the waypoint is selected based on traffic density; when the decision result is to change lanes, the waypoint is selected based on the speed of the autonomous vehicle. Based on the waypoints, the PID controller generates the driving trajectory of the autonomous vehicle and outputs the corresponding motion control information.

Images