Autonomous vehicle GNSS spoofing attack detection device and method based on reinforcement learning

Abstract

The invention discloses an automatic driving vehicle GNSS spoofing attack detection device based on reinforcement learning, which relates to the technical field of machine learning and satellite navigation and comprises a data processing module, a driving distance calculation module, a driving distance prediction module, a reinforcement learning module and a spoofing attack detection module. A directional attack data set is constructed by using a real driving data set of an automatic driving vehicle, a deep neural network is trained by using the attack data set and unattacked sensor data, a vehicle driving distance is predicted, so that a difference value between a predicted distance and a calculated distance is obtained, and a deep reinforcement learning model is trained by using the data set to obtain a deep reinforcement learning model. The invention further discloses an automatic driving vehicle GNSS spoofing attack detection method based on reinforcement learning, and the automatic driving vehicle GNSS spoofing attack detection method based on reinforcement learning can achieve accurate attack detection.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and satellite navigation technology, and in particular to a device and method for detecting GNSS spoofing attacks on an autonomous driving vehicle based on reinforcement learning. Background Art

[0002] With the advancement of communication and automation technology, the landscape of road transportation systems has fundamentally changed. Ground vehicles are becoming more and more automated and connected to each other and to transportation infrastructure. This provides an opportunity for travelers to move efficiently from one place to another and use their time for personal use while traveling. Positioning, navigation, and timing (PNT) services are key to navigation for autonomous vehicles (AVs). The Global Navigation Satellite System (GNSS) provides satellite-based PNT services. In the United States, GNSS is known as the Global Positioning System (GPS). GPS is a group of satellites launched by the United States Department of Defense in 1970 for military use. Some countries have their own satellites, such as GPS, to provide PNT services. In the United States, GPS provides two different services, the Standard Positioning Service (SPS) and the Precise Positioning Service (PPS). The PPS service can be used for government and military purposes, and autonomous vehicles are expected to use SPS, also known as civilian GPS, for their PNT services.

[0003] Autonomous vehicles require reliable and real-time PNT services. However, the robustness and reliability of GNSS-based PNT services depend on strong satellite signals and radio communications at the receiving end. The long distance between the satellite and the GNSS receiver results in reduced signal strength and reduced reliability of GNSS PNT services. GNSS signals are also susceptible to natural vulnerabilities, which are called unintentional vulnerabilities. For example, an autonomous vehicle may not be able to obtain GNSS signals when passing through a tunnel. In urban areas, tall buildings cause multipath propagation, which creates RF interference and reduces GNSS signal strength. In addition to these vulnerabilities, jamming and spoofing are common intentional threats to GNSS-based PNT services. Jamming attacks make it impossible for the receiver to obtain the real GNSS signal by drowning out the high-power corrupted signal. On the other hand, spoofing attackers can tamper with the real GNSS signal and may transmit false location information to the autonomous vehicle. Even the destination and route selection of the autonomous vehicle may be disrupted, and the vehicle may be directed to an unwanted destination. This will endanger the safety of AV users. Nowadays, attackers can use low-cost software-defined radios to carry out such spoofing attacks.

[0004] There are many methods to detect complex GNSS spoofing attacks by analyzing GNSS signals. Cryptographic mechanisms, coding-free cross-correlation metrics, signal statistical analysis, and antenna-based methods are common techniques for detecting spoofing attacks. In addition, researchers have used on-board inertial navigation systems (INS) and inertial measurement units (IMU) sensors such as gyroscopes and accelerometers to calculate the vehicle's position and acceleration by comparing with GNSS-based position and acceleration information to flag spoofing attacks. However, due to scaling factors and non-orthogonality errors, the position, velocity, and orientation information provided by these on-board sensors are less reliable. In addition, these errors accumulate over time. With the advancement of machine learning (ML) and deep learning (DL) models, several studies have investigated the potential of using ML and DL for complex spoofing attack detection. However, there is no research using deep reinforcement learning (RL) methods to detect complex turn-by-turn spoofing attacks. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a GNSS spoofing attack detection device and method for autonomous driving vehicles based on reinforcement learning. The present invention is based on low-cost on-board sensor data and learns action strategies by constructing a reinforcement learning model. It can effectively detect GNSS spoofing attacks without directly analyzing GNSS signal characteristics.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] In the first aspect, a GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning includes a data processing module, a driving distance estimation module, a driving distance prediction module, a reinforcement learning RL module, and a spoofing attack judgment module; wherein:

[0008] The data processing module is used to process the GPS latitude and longitude information obtained by the vehicle during driving, create position movement at different locations, and thus obtain a direction-by-direction deception attack data set and a non-attack data set;

[0009] The driving distance estimation module is used to calculate the ground truth value through GPS longitude and latitude information. The ground truth value refers to the driving distance d of the vehicle at two consecutive time stamps;

[0010] The driving distance prediction module is used to build a deep neural network DNN, use the non-attack data set to train the distance prediction deep neural network, and use the trained distance prediction deep neural network, the GPS longitude and latitude information of the vehicle during driving, and the steering wheel angle, speed and relative accelerator pedal position from the control area network CAN to predict the predicted distance S of the vehicle at two consecutive timestamps;

[0011] The reinforcement learning RL module is trained by the directional deception attack dataset and the non-attack dataset. After the training converges, it is used to update the threshold of the maximum prediction error of the driving distance between two consecutive timestamps in real time according to the real-time ground truth and the differential distance DD, DD = dS;

[0012] The deception attack judgment module is used to compare the real-time differential distance DD with the threshold obtained by the reinforcement learning RL module to determine whether the GNSS is under a deception attack. If the real-time differential distance DD is greater than the threshold, the GNSS is under a deception attack; otherwise, the GNSS is not under a deception attack.

[0013] As a further optimization scheme of the GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning described in the present invention, the reinforcement learning RL module includes a reinforcement learning agent, a reward calculation unit, an action strategy unit and a state calculation unit, wherein:

[0014] A reinforcement learning agent is used to obtain real-time ground truth and determine whether the GNSS is under spoofing attack using the real-time differential distance DD and the current state of the reinforcement learning agent, where the state refers to the threshold of the maximum prediction error of the driving distance between two consecutive timestamps;

[0015] The reward calculation unit is used to calculate the reward based on the result of judging whether the GNSS is under spoofing attack and the ground truth. The reward is used to train the deep learning network:

[0016] The action strategy unit is used to form a mapping relationship between states and actions, i.e., action strategies, through a deep learning network, ground truth, and differential distance; the actions include increasing the threshold, decreasing the threshold, or keeping the threshold unchanged;

[0017] The state calculation unit calculates the state at the next moment based on the state at the previous moment and the action taken.

[0018] As a further optimization scheme of the GNSS spoofing attack detection device for autonomous driving vehicles based on reinforcement learning described in the present invention, a turn-by-turn spoofing attack means that after the spoofer obtains the destination information of the target AV, it generates an erroneous GPS signal so that the GPS receiver of the AV is locked on the spoofing signal. After controlling the GPS signal, the spoofer creates a spoofing route that matches all the turns of the actual route.

[0019] As a further optimization scheme of the GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning described in the present invention, in the driving distance prediction module, a deep neural network constructed includes an input layer, an output layer, and three fully connected hidden layers. The vehicle driving distance calculated using the GPS longitude and latitude information during vehicle driving and the steering wheel angle, speed, and relative accelerator pedal position from the control area network CAN are normalized and used as the input layer neurons of the deep neural network. The number of neurons in the three fully connected hidden layers are 16, 8, and 4, respectively. The output of the deep neural network is the driving distance of two consecutive timestamps of the vehicle, and the rectified linear unit ReLU function is selected as the activation function.

[0020] As a further optimization scheme of the GNSS spoofing attack detection device for autonomous driving vehicles based on reinforcement learning described in the present invention, in the reinforcement learning module, the state input of the RL agent is a threshold defined according to the maximum prediction error of the driving distance between two consecutive timestamps. The RL agent obtains an action according to the state and the trained deep learning network. The action is used to update the threshold, thereby determining whether the GNSS is under a spoofing attack. The following configurations are used for reward calculation, state calculation, and RL action design:

[0021] a. Reward calculation

[0022] The reward is set as follows: when the ground truth is consistent with the judgment of the RL agent, a positive reward is given; otherwise, a negative reward is given;

[0023] b. State calculation

[0024] The states are configured as: the maximum prediction error of the travel distance between two consecutive timestamps;

[0025] c. RL action design

[0026] The RL actions are configured as: increase threshold, decrease threshold, and keep threshold unchanged.

[0027] Second, a reinforcement learning-based GNSS spoofing attack detection method for autonomous driving vehicles.

[0028] The GPS longitude and latitude information obtained by the vehicle during driving is processed to create position movement at different locations, thereby obtaining a direction-by-direction spoofing attack data set and a non-attack data set;

[0029] The ground truth is calculated through GPS longitude and latitude information. The ground truth refers to the distance d traveled by the vehicle at two consecutive time stamps.

[0030] A deep neural network DNN is constructed, and the deep neural network is trained using the direction-by-direction spoofing attack dataset and the non-attack dataset. The predicted distance S of the vehicle at two consecutive timestamps is predicted using the trained deep neural network, the GPS longitude and latitude information of the vehicle during driving, and the steering wheel angle, speed, and relative accelerator pedal position from the control area network CAN.

[0031] According to the real-time ground truth and the differential distance DD, DD = dS, the threshold of the maximum prediction error of the driving distance between two consecutive timestamps is updated in real time using the reinforcement learning RL module;

[0032] Compare the real-time differential distance DD with the threshold obtained by the reinforcement learning RL module to determine whether the GNSS is under a spoofing attack. If the real-time differential distance DD is greater than the threshold, the GNSS is under a spoofing attack; otherwise, the GNSS is not under a spoofing attack.

[0033] As a further optimization scheme of the GNSS spoofing attack detection method for autonomous driving vehicles based on reinforcement learning described in the present invention, the travel distance d of two consecutive timestamps is calculated using the haversine formula:

[0034]

[0035] Where d is the distance between two points on the Earth's surface, r is the radius of the Earth, and They represent longitude, ψ1 and ψ2 represent latitude.

[0036] As a further optimization scheme of the GNSS spoofing attack detection method for autonomous driving vehicles based on reinforcement learning described in the present invention, in the reinforcement learning module,

[0037] First, we obtain the real-time ground truth and use the real-time differential distance and the current state of the RL agent to determine whether the GNSS is under spoofing attack;

[0038] Then, the reward is calculated based on the judgment result and the ground truth, and the current Q-value function Q′(s) is updated using the current and next state, action, and reward. t , a t ):

[0039] Q′(s t , a t )=Q(s t , a t )+α(r t+1 +γmaxQ(s t+1 , a)-Q(s t , a t )

[0040] Among them, st is the current state, s t+1 is the state at the next moment, a t is the action taken, α is the learning rate, r t+1 is the reward at the next moment, γ is the discount factor, Q(*) is the Q-value function, the current Q-value is used to train the deep learning network, the next action is performed according to the action corresponding to the maximum Q-value output by the deep learning network, and the state of the RL agent is updated.

[0041] In the third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the steps of the method for detecting GNSS spoofing attacks on autonomous driving vehicles based on reinforcement learning are implemented as described in the second aspect above or any corresponding embodiment thereof.

[0042] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for detecting GNSS spoofing attacks for autonomous driving vehicles based on reinforcement learning are implemented as described in the second aspect above or any corresponding embodiment thereof.

[0043] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0044] This invention innovatively proposes to use the reinforcement learning model for GNSS spoofing attack detection of autonomous driving vehicles. By processing the GPS information in the real autonomous driving data set, a turn-by-turn attack data set is constructed for network training and testing. The RL agent is trained with the ground truth provided by GPS as a reference. According to the characteristics of the GNSS spoofing attack model, the main elements such as state quantity, reward function (reward), action, etc. are reasonably designed to obtain the optimal "state-action" strategy. At the same time, the sensor data is used to obtain the prediction error of the driving distance within two timestamps, which ensures the reliability of the reinforcement learning model in GNSS turn-by-turn attack detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a system structure diagram of the present invention.

[0046] Figure 2 This is a diagram of the reinforcement learning model. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] A GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning, comprising a data processing module, a driving distance estimation module, a driving distance prediction module, a reinforcement learning RL module and a spoofing attack judgment module; wherein:

[0049] The data processing module is used to process the GPS latitude and longitude information obtained by the vehicle during driving, create position movement at different locations, and thus obtain a direction-by-direction deception attack data set and a non-attack data set;

[0050] The driving distance estimation module is used to calculate the ground truth value through GPS longitude and latitude information. The ground truth value refers to the driving distance d of the vehicle at two consecutive time stamps;

[0051] The driving distance prediction module is used to build a deep neural network DNN, use the non-attack data set to train the distance prediction deep neural network, and use the trained distance prediction deep neural network, the GPS longitude and latitude information of the vehicle during driving, and the steering wheel angle, speed and relative accelerator pedal position from the control area network CAN to predict the predicted distance S of the vehicle at two consecutive timestamps;

[0052] The reinforcement learning RL module is trained by the directional deception attack dataset and the non-attack dataset. After the training converges, it is used to update the threshold of the maximum prediction error of the driving distance between two consecutive timestamps in real time according to the real-time ground truth and the differential distance DD, DD = dS;

[0053] The deception attack judgment module is used to compare the real-time differential distance DD with the threshold obtained by the reinforcement learning RL module to determine whether the GNSS is under a deception attack. If the real-time differential distance DD is greater than the threshold, the GNSS is under a deception attack; otherwise, the GNSS is not under a deception attack.

[0054] As a further optimization scheme of the GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning described in the present invention, the reinforcement learning RL module includes a reinforcement learning agent, a reward calculation unit, an action strategy unit and a state calculation unit, wherein:

[0055] A reinforcement learning agent is used to obtain real-time ground truth and determine whether the GNSS is under spoofing attack using the real-time differential distance DD and the current state of the reinforcement learning agent, where the state refers to the threshold of the maximum prediction error of the driving distance between two consecutive timestamps;

[0056] The reward calculation unit is used to calculate the reward based on the result of judging whether the GNSS is under spoofing attack and the ground truth. The reward is used to train the deep learning network:

[0057] The action strategy unit is used to form a mapping relationship between states and actions, i.e., action strategies, through a deep learning network, ground truth, and differential distance; the actions include increasing the threshold, decreasing the threshold, or keeping the threshold unchanged;

[0058] The state calculation unit calculates the state at the next moment based on the state at the previous moment and the action taken.

[0059] As a further optimization scheme of the GNSS spoofing attack detection device for autonomous driving vehicles based on reinforcement learning described in the present invention, a turn-by-turn spoofing attack means that after the spoofer obtains the destination information of the target AV, it generates an erroneous GPS signal so that the GPS receiver of the AV is locked on the spoofing signal. After controlling the GPS signal, the spoofer creates a spoofing route that matches all the turns of the actual route.

[0060] As a further optimization scheme of the GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning described in the present invention, in the driving distance prediction module, a deep neural network constructed includes an input layer, an output layer, and three fully connected hidden layers. The vehicle driving distance calculated using the GPS longitude and latitude information during vehicle driving and the steering wheel angle, speed, and relative accelerator pedal position from the control area network CAN are normalized and used as the input layer neurons of the deep neural network. The number of neurons in the three fully connected hidden layers are 16, 8, and 4, respectively. The output of the deep neural network is the driving distance of two consecutive timestamps of the vehicle, and the rectified linear unit ReLU function is selected as the activation function.

[0061] As a further optimization scheme of the GNSS spoofing attack detection device for autonomous driving vehicles based on reinforcement learning described in the present invention, in the reinforcement learning module, the state input of the RL agent is a threshold defined according to the maximum prediction error of the driving distance between two consecutive timestamps. The RL agent obtains an action according to the state and the trained deep learning network. The action is used to update the threshold, thereby determining whether the GNSS is under a spoofing attack. The following configurations are used for reward calculation, state calculation, and RL action design:

[0062] a. Reward calculation

[0063] The reward is set as follows: when the ground truth is consistent with the judgment of the RL agent, a positive reward is given; otherwise, a negative reward is given;

[0064] b. State calculation

[0065] The states are configured as: the maximum prediction error of the travel distance between two consecutive timestamps;

[0066] c. RL action design

[0067] The RL actions are configured as: increase threshold, decrease threshold, and keep threshold unchanged.

[0068] Second, a reinforcement learning-based GNSS spoofing attack detection method for autonomous driving vehicles.

[0069] The GPS longitude and latitude information obtained by the vehicle during driving is processed to create position movement at different locations, thereby obtaining a direction-by-direction spoofing attack data set and a non-attack data set;

[0070] The ground truth is calculated through GPS longitude and latitude information. The ground truth refers to the distance d traveled by the vehicle at two consecutive time stamps.

[0071] A deep neural network DNN is constructed, and the deep neural network is trained using the direction-by-direction spoofing attack dataset and the non-attack dataset. The predicted distance S of the vehicle at two consecutive timestamps is predicted using the trained deep neural network, the GPS longitude and latitude information of the vehicle during driving, and the steering wheel angle, speed, and relative accelerator pedal position from the control area network CAN.

[0072] According to the real-time ground truth and the differential distance DD, DD = dS, the threshold of the maximum prediction error of the driving distance between two consecutive timestamps is updated in real time using the reinforcement learning RL module;

[0073] Compare the real-time differential distance DD with the threshold obtained by the reinforcement learning RL module to determine whether the GNSS is under a spoofing attack. If the real-time differential distance DD is greater than the threshold, the GNSS is under a spoofing attack; otherwise, the GNSS is not under a spoofing attack.

[0074] As a further optimization scheme of the GNSS spoofing attack detection method for autonomous driving vehicles based on reinforcement learning described in the present invention, the travel distance d of two consecutive timestamps is calculated using the haversine formula:

[0075]

[0076] Where d is the distance between two points on the Earth's surface, r is the radius of the Earth, and They represent longitude, ψ1 and ψ2 represent latitude.

[0077] As a further optimization scheme of the GNSS spoofing attack detection method for autonomous driving vehicles based on reinforcement learning described in the present invention, in the reinforcement learning module,

[0078] First, we obtain the real-time ground truth and use the real-time differential distance and the current state of the RL agent to determine whether the GNSS is under spoofing attack;

[0079] Then, the reward is calculated based on the judgment result and the ground truth, and the current Q-value function Q′(s) is updated using the current and next state, action, and reward. t , a t ):

[0080] Q′(s t , a t )=Q(s t , a t )+α(r t+1 +γmaxQ(s t+1 , a)-Q(s t , a t )

[0081] Among them, s t is the current state, s t+1 is the state at the next moment, a t is the action taken, α is the learning rate, r t+1 is the reward at the next moment, γ is the discount factor, Q(*) is the Q-value function, the current Q-value is used to train the deep learning network, the next action is performed according to the action corresponding to the maximum Q-value output by the deep learning network, and the state of the RL agent is updated.

[0082] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned reinforcement learning-based autonomous driving vehicle GNSS spoofing attack detection method are implemented.

[0083] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned reinforcement learning-based autonomous driving vehicle GNSS spoofing attack detection method.

[0084] The specific embodiments are as follows: The present invention includes a data processing module, a driving distance estimation module, a driving distance prediction module, a reinforcement learning module and a deception attack detection module; wherein,

[0085] The data processing module is mainly used to process the GPS latitude and longitude information obtained by the vehicle in the urban environment, and create position movements ranging from one block to several blocks at different locations, so as to obtain the directional deception attack dataset and the non-attack dataset for model training and result verification. A total of ten different directional attack datasets are created to train and test the RL model. All datasets contain multiple directional attacks, but the number of attacks is different.

[0086] The driving distance estimation module mainly uses the processed GPS longitude and latitude information to estimate the vehicle's driving distance between two consecutive timestamps using the haversine formula:

[0087]

[0088] Where d is the distance between two points on the Earth's surface (in meters), r is the radius of the Earth (6378 km), and represents longitude (in radians), and ψ1 and ψ2 represent latitude (in radians).

[0089] The driving distance prediction module mainly synchronizes and normalizes the driving distance calculated using the GPS longitude and latitude information and the steering wheel angle (deg), speed (ft / s) and relative accelerator pedal position (%) from the control area network (CAN), and then trains them through a deep neural network (DNN). The deep neural network contains three fully connected hidden layers, which contain 16, 8, and 4 hidden layer neurons respectively. The rectified linear unit (ReLU) function is selected as the activation function, and the trained network is used to predict the distance traveled by the vehicle between two consecutive timestamps.

[0090] The reinforcement learning module is mainly used to estimate the threshold of the differential distance, that is, the maximum prediction error of the driving distance between two consecutive timestamps, such as Figure 1 As shown in the figure, the system determines whether the GNSS is under spoofing attack based on the calculated differential distance and the threshold estimated by the RL agent. The RL agent uses the threshold of the differential distance as the state input, and the RL obtains the RL action based on the state and the trained action strategy network. The action is used to update the threshold of the differential distance, thereby improving the accuracy of GNSS spoofing attack detection. Based on the understanding of the GNSS spoofing attack model, the following configuration is proposed:

[0091] a. Reward calculation

[0092] The reward can be set as follows: if the ground truth (GPS trust state) is consistent with the RL agent’s detection of a GNSS spoofing attack, a positive reward (+1) is given; otherwise, a larger negative reward (-100) is given.

[0093] b. State calculation

[0094] The state is configured as: the maximum prediction error of the travel distance between two consecutive timestamps.

[0095] c.RL action design

[0096] The RL actions are configured as: increase threshold, decrease threshold, and keep threshold unchanged.

[0097] The system block diagram of the reinforcement learning model of the present invention is as follows Figure 2 As shown in the figure, first, the real-time ground truth is obtained, and the real-time differential distance and the current state of the RL agent are used to determine whether the GNSS is under a spoofing attack; then, the reward is calculated based on the judgment result and the ground truth, the neural network of the RL agent is trained, and the next action is obtained and the state of the RL agent is updated.

[0098] The spoofing attack detection module is mainly used to compare the differential distance with the threshold obtained by the reinforcement learning RL module to determine whether the GNSS is under a spoofing attack. If the differential distance is greater than the threshold, the GNSS is under a spoofing attack; otherwise, the GNSS is not under a spoofing attack.

[0099] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0100] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0103] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0104] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A GNSS spoofing attack detection device for autonomous driving vehicles based on reinforcement learning, characterized in that: It includes a data processing module, a driving distance estimation module, a driving distance prediction module, a reinforcement learning RL module and a deception attack judgment module; among which, The data processing module is used to process the GPS latitude and longitude information obtained by the vehicle during driving, create position movement at different locations, and thus obtain a direction-by-direction deception attack data set and a non-attack data set; The driving distance estimation module is used to calculate the ground truth value through GPS longitude and latitude information. The ground truth value refers to the driving distance d of the vehicle at two consecutive time stamps; The driving distance prediction module is used to build a deep neural network DNN, use the non-attack data set to train the distance prediction deep neural network, and use the trained distance prediction deep neural network, the GPS longitude and latitude information of the vehicle during driving, and the steering wheel angle, speed and relative accelerator pedal position from the control area network CAN to predict the predicted distance S of the vehicle at two consecutive timestamps; The reinforcement learning RL module is trained by the directional deception attack dataset and the non-attack dataset. After the training converges, it is used to update the threshold of the maximum prediction error of the driving distance between two consecutive timestamps in real time according to the real-time ground truth and the differential distance DD, DD = dS; The deception attack judgment module is used to compare the real-time differential distance DD with the threshold obtained by the reinforcement learning RL module to determine whether the GNSS is under a deception attack. If the real-time differential distance DD is greater than the threshold, the GNSS is under a deception attack; otherwise, the GNSS is not under a deception attack.

2. The GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning RL module includes a reinforcement learning agent, a reward calculation unit, an action strategy unit, and a state calculation unit, among which: A reinforcement learning agent is used to obtain real-time ground truth and determine whether the GNSS is under spoofing attack using the real-time differential distance DD and the current state of the reinforcement learning agent, where the state refers to the threshold of the maximum prediction error of the driving distance between two consecutive timestamps; The reward calculation unit is used to calculate the reward based on the result of judging whether the GNSS is under spoofing attack and the ground truth. The reward is used to train the deep learning network: The action strategy unit is used to form a mapping relationship between states and actions, i.e., action strategies, through a deep learning network, ground truth, and differential distance; the actions include increasing the threshold, decreasing the threshold, or keeping the threshold unchanged; The state calculation unit calculates the state at the next moment based on the state at the previous moment and the action taken.

3. The GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning according to claim 1, characterized in that: A turn-by-turn spoofing attack is when a spoofer obtains the target AV’s destination information and generates a false GPS signal, causing the AV’s GPS receiver to lock onto the spoofed signal. After controlling the GPS signal, the spoofer creates a spoofed route that matches all the turns of the actual route.

4. The GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning according to claim 1, characterized in that: In the driving distance prediction module, the constructed deep neural network includes an input layer, an output layer and three fully connected hidden layers. The vehicle driving distance calculated using the GPS longitude and latitude information during vehicle driving and the steering wheel angle, speed and relative accelerator pedal position from the control area network CAN are normalized and used as the input layer neurons of the deep neural network. The number of neurons in the three fully connected hidden layers are 16, 8 and 4 respectively. The output of the deep neural network is the driving distance of two consecutive timestamps of the vehicle, and the rectified linear unit ReLU function is selected as the activation function.

5. The GNSS spoofing attack detection device for an autonomous driving vehicle based on reinforcement learning according to claim 2, characterized in that: In the reinforcement learning module, the state input of the RL agent is a threshold defined by the maximum prediction error of the driving distance between two consecutive timestamps. The RL agent obtains an action based on the state and the trained deep learning network. The action is used to update the threshold, which is used to determine whether the GNSS is under spoofing attack. The following configurations are used for reward calculation, state calculation, and RL action design: a. Reward calculation The reward is set as follows: when the ground truth is consistent with the judgment of the RL agent, a positive reward is given; otherwise, a negative reward is given; b. State calculation The states are configured as: the maximum prediction error of the travel distance between two consecutive timestamps; c. RL action design The RL actions are configured as: increase threshold, decrease threshold, and keep threshold unchanged.

6. A method for detecting GNSS spoofing attacks on autonomous driving vehicles based on reinforcement learning, characterized in that: The GPS longitude and latitude information obtained by the vehicle during driving is processed to create position movement at different locations, thereby obtaining a direction-by-direction spoofing attack data set and a non-attack data set; The ground truth is calculated through GPS longitude and latitude information. The ground truth refers to the distance d traveled by the vehicle at two consecutive time stamps. A deep neural network DNN is constructed, and the deep neural network is trained using the direction-by-direction spoofing attack dataset and the non-attack dataset. The predicted distance S of the vehicle at two consecutive timestamps is predicted using the trained deep neural network, the GPS longitude and latitude information of the vehicle during driving, and the steering wheel angle, speed, and relative accelerator pedal position from the control area network CAN. According to the real-time ground truth and the differential distance DD, DD = dS, the threshold of the maximum prediction error of the driving distance between two consecutive timestamps is updated in real time using the reinforcement learning RL module; Compare the real-time differential distance DD with the threshold obtained by the reinforcement learning RL module to determine whether the GNSS is under spoofing attack. If the real-time differential distance DD is greater than the threshold, the GNSS is under spoofing attack; Otherwise, GNSS is not vulnerable to spoofing attacks.

7. A reinforcement learning-based GNSS spoofing attack detection method for an autonomous driving vehicle according to claim 6, characterized in that: The travel distance d between two consecutive time stamps is calculated using the Haversine formula: Where d is the distance between two points on the Earth's surface, r is the radius of the Earth, and They represent longitude, ψ1 and ψ2 represent latitude.

8. A reinforcement learning-based GNSS spoofing attack detection method for an autonomous driving vehicle according to claim 6, characterized in that: In the reinforcement learning module, First, we obtain the real-time ground truth and use the real-time differential distance and the current state of the RL agent to determine whether the GNSS is under spoofing attack; Then, the reward is calculated based on the judgment result and the ground truth, and the current Q-value function Q′(s) is updated using the current and next state, action, and reward. t ,a t ): Q′(s t ,a t )=Q(s t ,a t )+α(r t+1 +γmaxQ(s t+1 ,a)-Q(s t ,a t ) Among them, s t is the current state, s t+1 is the state at the next moment, a t is the action taken, α is the learning rate, r t+1 is the reward at the next moment, γ is the discount factor, Q(*) is the Q-value function, the current Q-value is used to train the deep learning network, the next action is performed according to the action corresponding to the maximum Q-value output by the deep learning network, and the state of the RL agent is updated.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the reinforcement learning-based GNSS spoofing attack detection method for an autonomous driving vehicle are implemented as described in any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the reinforcement learning-based GNSS spoofing attack detection method for an autonomous driving vehicle are implemented as described in any one of claims 1 to 5.

Images