Internet of vehicles environment intelligent penetration test method and system based on generative adversarial imitation learning
By applying a generative adversarial imitation learning model in the Internet of Vehicles environment, combined with the experience knowledge base of Internet of Vehicles security experts, the convergence difficulties of state space and action space in penetration test are solved, automated and efficient penetration tests are realized, and security vulnerabilities are quickly discovered. , reducing labor costs.
Patent Information
- Application Number
- CN202410927934.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
AI Technical Summary
Penetration testing in the existing Internet of Vehicles infotainment environment has convergence difficulties caused by large-scale state space and high-dimensional discrete action space, and traditional methods rely on manual operations, which increases labor costs and is difficult to quickly detect potential threats.
Using a method based on generative adversarial imitation learning, the generative adversarial imitation learning model is used to integrate the experience knowledge base of Internet of Vehicles security experts on the basis of deep reinforcement learning or reinforcement learning, build a Markov game model, and use generators and discriminators to train intelligence. optimize penetration test paths to achieve automated and efficient simulation attacks.
It realizes efficient and accurate penetration testing path planning in the IVI environment of the Internet of Vehicles, reduces labor costs, can quickly discover potential security vulnerabilities, simulates expert-level penetration behavior, and improves testing efficiency and strategy quality.
Smart Images

Figure CN118944912A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent traffic safety technology, and in particular to an intelligent penetration testing method and system for an Internet of Vehicles environment based on generative adversarial imitation learning. Background Art
[0002] With the rapid development of information technology, especially the widespread application of Internet of Things (IoT) and Vehicular Network (Vehicular Network) technologies, the degree of vehicle informatization and intelligence has been continuously improved, but it has also brought increasingly serious network security risks. Due to its complexity, dynamism and openness, the IVI vehicle networking environment is vulnerable to various malicious attacks, making the security protection of vehicle systems and related infrastructure particularly critical. Traditional penetration testing (PT) methods have played an important role in identifying and exposing these potential threats, but they are highly dependent on the manual operations and decisions of security experts, which not only consumes a lot of manpower costs, but also often finds it difficult to quickly converge to an effective penetration strategy when faced with a large and complex state space and a high-dimensional discrete action space.
[0003] In view of this, in recent years, the industry has gradually turned to artificial intelligence algorithms such as reinforcement learning (RL) and deep reinforcement learning (DRL) to automate the penetration testing process, thereby reducing manpower investment and improving testing efficiency. However, the direct application of RL / DRL methods in actual IVI penetration scenarios still faces a series of challenges, such as increased convergence difficulty due to the huge action space and lack of verification in real environments. Therefore, it is urgent to develop a technical solution that can automatically and efficiently perform penetration testing in the IVI environment. Summary of the invention
[0004] To this end, the present invention provides an intelligent penetration testing method and system for an Internet of Vehicles environment based on generative adversarial imitation learning. By integrating a generative adversarial imitation learning mechanism, the penetration test of the Internet of Vehicles in-vehicle infotainment environment is realized on the basis of deep reinforcement learning or reinforcement learning. The problem of convergence difficulty caused by large-scale state space and high-dimensional discrete action space in the penetration test under the existing Internet of Vehicles in-vehicle infotainment environment is solved, and the simulation attack test in the in-vehicle entertainment environment with real-time changes and complex physical characteristics is met.
[0005] According to the design scheme provided by the present invention, on the one hand, a method for intelligent penetration testing of an Internet of Vehicles environment based on generative adversarial imitation learning is provided, comprising:
[0006] Receive a penetration test request of a target object, and collect Internet of Vehicles environment data of the target object according to the penetration test request, wherein the target object is a target vehicle or a target Internet of Vehicles environment, and the Internet of Vehicles environment is a network environment of an in-vehicle infotainment system;
[0007] Input the target object's Internet of Vehicles environment data into the pre-trained generative adversarial imitation learning model, and use the generative adversarial imitation learning model to simulate attacks on the target object;
[0008] Among them, the generative adversarial imitation learning model is a simulated attack agent trained based on pre-acquired Internet of Vehicles environmental data and the Internet of Vehicles security expert experience knowledge base.
[0009] As the intelligent penetration testing method for the Internet of Vehicles environment based on generative adversarial imitation learning of the present invention, further, the training process of the generative adversarial imitation learning model includes:
[0010] Perform penetration testing in a simulated network in an Internet of Vehicles environment, set a corresponding reward value according to the degree of intrusion during the penetration test, collect state-action pair sample data based on the reward value, and store the state-action pair sample data in an Internet of Vehicles security expert experience knowledge base, where the state refers to the vehicle computer configuration and vulnerability information in the Internet of Vehicles environment, and the action refers to the test-related operations performed on the vehicle computer;
[0011] Collect Internet of Vehicles environment data, and build a generative adversarial imitation learning model based on a generator and a discriminator. The generator determines the probability distribution of the agent's action space based on the Internet of Vehicles environment state data, and the discriminator obtains a discounted reward for guiding learning based on the agent's action and the state-action pairs in the vehicle network safety expert experience knowledge base;
[0012] The objective function of model training is constructed using the experience knowledge base loss of Internet of Vehicles security experts and the agent loss. Based on this objective function, the discriminator and generator network parameters are updated during model training.
[0013] As an intelligent penetration testing method for the Internet of Vehicles environment based on generative adversarial imitation learning of the present invention, further, the generator is a reinforcement learning model or a deep reinforcement learning model, which uses the reinforcement learning model or the deep reinforcement learning model to obtain the action corresponding to the state of the Internet of Vehicles environment, and inputs it into the discriminator for comparison with the state-action in the experience knowledge base of the Internet of Vehicles security experts.
[0014] As the intelligent penetration testing method for the Internet of Vehicles environment based on generative adversarial imitation learning of the present invention, further, the generator includes a policy network and a value network, and the policy network and the value network are set as a maximum-minimum game process during the training process. The objective function of the minimum-maximum game process is expressed as: Among them, D is the discriminator, π represents the learning strategy, and v E represents the expert strategy, log D(s, a) represents the judgment of the discriminator on the state-action pair (s, a) fact data, log(1-D(s, a)) represents the judgment of the discriminator on the state-action pair (s, a) generated data, L GAIL Loss for generative adversarial imitation learning.
[0015] As the intelligent penetration testing method for the Internet of Vehicles environment based on generative adversarial imitation learning of the present invention, further, the objective function of the model training is expressed as: in, It represents the entropy of the imitation strategy, λ is the control coefficient of the strategy regularization term, and λ≥0, ω and θ represent the discriminator and generator network parameters respectively, S is the state set, and A is the action set.
[0016] As the intelligent penetration testing method for the Internet of Vehicles environment based on generative adversarial imitation learning of the present invention, further, a generative adversarial imitation learning model is used to simulate an attack on a target object, including:
[0017] For the penetration test request of the target vehicle, the penetration test process is modeled as a Markov game model, where the state of the agent is set to the port scanning information and the action is set to the corresponding action in the penetration test payload sequence list;
[0018] A multi-threaded mechanism is used to train the intelligent agent using a deep reinforcement learning algorithm. The state in which the intelligent agent obtains the root permissions of the target vehicle's Internet of Vehicles environment through actions is used as a sign of the successful status of the penetration test attack, and a reward and punishment mechanism is used to incentivize the intelligent agent to participate in the penetration test of the target vehicle's Internet of Vehicles environment.
[0019] As the intelligent penetration testing method for the Internet of Vehicles environment based on generative adversarial imitation learning of the present invention, further, a generative adversarial imitation learning model is used to simulate an attack on a target object, including:
[0020] For the penetration test request of the Internet of Vehicles environment, the penetration test process is modeled as a Markov game model, in which the agent state is set to the vehicle computer configuration information and vulnerability information collected by the agent in different network environments, and the action is set to the operation of the agent to simulate the attack on the vehicle computer in the network environment. The simulated attack vector is formed according to the corresponding actions performed by the agent on different vehicle computers in the network environment;
[0021] The reinforcement learning algorithm is used to automatically generate penetration test paths and optimize path optimization. In the path optimization, the intelligent agent selects actions and obtains the action probability space, and obtains the reward value based on the value of the vehicle computer and the cost of performing actions on the vehicle computer. The optimal penetration path is explored by maximizing the cumulative reward, and the most valuable sensitive vehicle computer information is obtained by performing as few actions as possible.
[0022] On the other hand, the present invention also provides an intelligent penetration testing system for an Internet of Vehicles environment based on generative adversarial imitation learning, comprising: a test receiving module and a simulation test module, wherein:
[0023] A test receiving module, used to receive a penetration test request of a target object, and collect Internet of Vehicles environment data of the target object according to the penetration test request, wherein the target object is a target vehicle or a target Internet of Vehicles environment, and the Internet of Vehicles environment is a network environment of an in-vehicle infotainment system;
[0024] A simulation test module is used to input the target object's Internet of Vehicles environment data into a pre-trained generative adversarial imitation learning model, and use the generative adversarial imitation learning model to simulate attacks on the target object;
[0025] Among them, the generative adversarial imitation learning model is a simulated attack agent trained based on pre-acquired Internet of Vehicles environmental data and the Internet of Vehicles security expert experience knowledge base.
[0026] Beneficial effects of the present invention:
[0027] Aiming at the characteristics of the complex system of IVI of the Internet of Vehicles, the present invention constructs an experience knowledge base of IVI security experts of the Internet of Vehicles, and uses the experience knowledge base to store state-action pairs in different penetration scenarios, so that expert knowledge can participate in the decision-making process at a lower cost, and realize more efficient and accurate penetration test path planning; by modifying the overall loss update mechanism of the GAIL network, combining the loss function of the RL / DRL model and the Discriminator to optimize the performance of the RL / DRL model in the penetration test, the improved GAIL is applied to the DRL / RL penetration test of the IVI security of the Internet of Vehicles, making the training process more stable and efficient, and can automatically collect and use the expert knowledge base to guide the actions of the intelligent agent, so that in the ever-changing IVI environment of the Internet of Vehicles, the intelligent agent can quickly approach and simulate the expert-level penetration behavior, and then quickly discover potential security vulnerabilities, which has a good application prospect in the field of intelligent transportation safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is a schematic diagram of the intelligent penetration testing process of the Internet of Vehicles environment based on generative adversarial imitation learning in the embodiment;
[0029] Figure 2 This is a schematic diagram of the working principle of the intelligent penetration test in the Internet of Vehicles environment in the embodiment;
[0030] Figure 3 This is a schematic diagram of the penetration attack process in the embodiment;
[0031] Figure 4 Schematic diagram of the training process of the generative adversarial imitation learning model in the embodiment. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.
[0033] In-Vehicle Infotainment (IVI) is an integrated vehicle information processing system that uses a dedicated vehicle CPU, based on the vehicle bus system and Internet services. IVI can realize a series of applications including three-dimensional navigation, real-time traffic conditions, IPTV, assisted driving, fault detection, vehicle information, body control, mobile office, wireless communication, online entertainment functions and TSP services, which greatly improves the level of vehicle electrification, networking and intelligence. With the development of intelligent network technology and the expansion of integrated applications of application terminals and IVI, applications, protocols or IVI programs are vulnerable to attacks. IVI security is an important part of ensuring the network security of intelligent networked vehicles. The application of traditional penetration testing methods in the IVI environment requires security experts to participate deeply in every test link. This method that highly relies on expert manual operation and decision-making greatly increases the labor cost. When using the deep reinforcement learning (DRL) algorithm for automated penetration testing, the huge state space and high-dimensional discrete action space of the IVI environment may lead to a slow training process and difficult convergence. It is difficult for intelligent agents to learn effective penetration strategies in a short time. Most intelligent penetration testing technologies are verified in a virtual network environment. However, the IVI environment has the characteristics of real-time changes and complex physical characteristics, so the strategies learned in the virtual environment may not be well applied in the real environment. In the IVI field, collecting and organizing the penetration rules and practical experience developed by network security experts is a time-consuming and laborious task. How to effectively integrate these valuable knowledge base resources into the automated penetration testing model and reduce the cost of knowledge transfer is an urgent problem to be solved. To this end, the embodiments of the present invention, see Figure 1 As shown, a method for intelligent penetration testing of an Internet of Vehicles environment based on generative adversarial imitation learning is provided, comprising:
[0034] S101, receiving a penetration test request of a target object, and collecting Internet of Vehicles environment data of the target object according to the penetration test request, wherein the target object is a target vehicle or a target Internet of Vehicles environment, and the Internet of Vehicles environment is a network environment of an in-vehicle infotainment system;
[0035] S102, inputting the target object's Internet of Vehicles environment data into a pre-trained generative adversarial imitation learning model, and using the generative adversarial imitation learning model to perform a simulated attack on the target object;
[0036] Among them, the generative adversarial imitation learning model is a simulated attack agent trained based on pre-acquired Internet of Vehicles environmental data and the Internet of Vehicles security expert experience knowledge base.
[0037] Specifically, the training process of the generative adversarial imitation learning model can be designed to include:
[0038] Perform penetration testing in a simulated network in an Internet of Vehicles environment, set a corresponding reward value according to the degree of intrusion during the penetration test, collect state-action pair sample data based on the reward value, and store the state-action pair sample data in an Internet of Vehicles security expert experience knowledge base, where the state refers to the vehicle computer configuration and vulnerability information in the Internet of Vehicles environment, and the action refers to the test-related operations performed on the vehicle computer;
[0039] Collect Internet of Vehicles environment data, and build a generative adversarial imitation learning model based on a generator and a discriminator. The generator determines the probability distribution of the agent's action space based on the Internet of Vehicles environment state data, and the discriminator obtains a discounted reward for guiding learning based on the agent's action and the state-action pairs in the vehicle network safety expert experience knowledge base;
[0040] The objective function of model training is constructed using the experience knowledge base loss of Internet of Vehicles security experts and the agent loss. Based on this objective function, the discriminator and generator network parameters are updated during model training.
[0041] like Figure 2 As shown, in the embodiment of this case, an expert knowledge base is constructed to store the state-action data of successful cases in IVI penetration testing; an adversarial generative imitation learning algorithm is used to guide the intelligent agent to learn the optimal penetration strategy; the improved GAIL network can be applied to the DRL model to perform intelligent penetration testing in a single target vehicle computer, or the improved GAIL network can be applied to the RL model to perform intelligent penetration testing in simulated and real IVI environments.
[0042] Specifically, in constructing the expert knowledge base, proprietary tools can be used to automatically collect state-action pairs when the "post-exploit" is successfully executed under the IVI simulation network, and then save them into the expert knowledge base constructed online.
[0043] Specifically, executing automatic PT will result in three different results on a target vehicle: unsuccessful intrusion, successful intrusion, and successful post-intrusion. Figure 3 As shown in the figure, when the Kali server launches a penetration attack on the target vehicle computer, if stage 1 is executed successfully, the target server will establish a session connection with the Kali server at the same time as stage 2, and then Kali will use the payload to perform privilege escalation operations on the target vehicle computer to obtain root privileges. We call the privilege escalation process "post-exploit", which corresponds to stage 3. Assuming that stages 1, 2, and 3 are all executed successfully, it means that the Kali server has successfully invaded the target vehicle computer and performed subsequent privilege escalation operations through the payload, which means that the post-exploit is also successful. Different reward values can be set for mixed penetration results, and the highest value can be set for successful post-exploitation. Therefore, during the PT process, the reward value obtained by the agent can be used to confirm the penetration result at that time. In addition, the state-action pairs with the highest reward value are collected as expert sample data and stored in the expert knowledge base in the form of one-to-one correspondence between states and actions.
[0044] Collecting expert sample data in the IVI scenario also means storing the state-action pairs with the highest reward value when successfully exploiting the target sensitive vehicle computer at the lowest cost. In general, the construction of the PT expert knowledge base is similar to exploiting a single target vehicle computer and network. Specifically, exploiting a vehicle computer refers to performing relevant operations on the target vehicle computer, such as executing a payload and ultimately gaining control of the target vehicle computer. In a network scenario, after gaining control of a vehicle computer, the attacker can use the compromised vehicle computer as a stepping stone to move laterally based on the network connection relationship to gain control of the target sensitive vehicle computer. In other words, exploiting the network is like finding the best attack path, and the path shows the exploitation relationship between certain vehicle computers in different subnets. It is worth mentioning that the biggest difference between the scheme in this case and the attack graph is that the scheme in this case does not require prior knowledge of the network topology information, but instead determines the best attack path through continuous trial and error training.
[0045] In the training of GAIL network design based on Generative Adversarial Imitation Learning (GAIL), GAIL can be designed to contain two different neural networks: the generator and the discriminator. The generator can be composed of an arbitrary DRL / RL model. Taking the A3C model as an example, its Actor consists of a four-layer network. The hidden layer consists of three dense layers with output dimensions of 50, 100, and 200, connected by ReLU activation functions. The last layer is the output layer with Softmax activation function, and the output dimension corresponds to the number of actions. The network structure of the Critic is similar to that of the Actor. The only difference is that the last output layer is linear with a dimension of 1. The Discriminator is called the Reward Neural Network, which has a four-layer network, in which the structure of the three hidden layers is the same as that of the actor's hidden layers. The last output layer is activated by Sigmoid and outputs a specific probability value as a discounted reward.
[0046] like Figure 4 The training process of GAIL is shown in the figure. The Generator and Discriminator are trained alternately. The input of the Generator is the currently observed state, and the output is the corresponding action and the value of the current state. The input of the Discriminator is the state-action pairs of the agent and the PT expert. In addition, the state-action pairs of the expert samples are only used for training, and the output of the Discriminator is the discounted reward value. The main process of GAIL training can be described as follows: (1) The state-action pairs in the PT expert knowledge base and the state-action pairs generated by the DRL / RL online model are put into the discriminator and trained together. (2) The discriminator is trained by maximizing the standard action reward given by the PT expert sample and minimizing the action reward output by the agent. (3) The output of the discriminator is used to replace the reward function of the original model to guide the training direction. (4) The predicted distribution of the DRL model output is as close as possible to the trajectory of the PT expert knowledge base.
[0047] In the training process of imitating the PT expert strategy, the DRL / RL model is used to replace the generator G, and its output actions and states are paired into the discriminator and compared with the expert data. The output of the discriminator D:S×A→(0,1) is regarded as a reward signal to guide the strategy learning in imitation learning. Therefore, the training of the strategy and reward function is regarded as a game process, similar to the game between G and D in the generative adversarial network (GAN), and the objective function L GAIL (π,D) can be expressed as follows:
[0048]
[0049] Where π represents the strategy to be learned, π E Denotes the expert strategy. The first term log D(s, a) represents the judgment of the discriminator on the factual data, and the second term log(1-D(s, a)) represents the judgment on the generated data. Through such a maximum-minimum game process, G and D are alternately optimized to train the behavior network and the discriminant network. During the training process, the expert loss and the agent loss can be added to obtain the total loss, and the loss function can be minimized by gradient derivation, thereby reversely updating the parameters of the discriminant network and the agent network. The total loss can be expressed as:
[0050]
[0051] in represents the entropy of the imitation strategy, which is controlled by a constant λ (λ ≥ 0) and serves as the policy regularization term in the loss function. ω and θ represent the parameters of the discriminator and DRL / RL network, respectively.
[0052] Among them, for the penetration test request of the target vehicle, the penetration test process is modeled as a Markov game model, in which the state of the agent is set to scan information through the port, and the action is set to the corresponding action in the penetration test payload sequence list; the agent is trained using a multi-threaded mechanism using a deep reinforcement learning algorithm, and the state of the agent obtaining the root authority of the target vehicle's Internet of Vehicles environment through actions is used as a sign of the successful status of the penetration test attack, and a reward and punishment mechanism is used to incentivize the agent to participate in the penetration test of the target vehicle's Internet of Vehicles environment.
[0053] The improved GAIL network is applied to the DRL model to perform intelligent penetration testing in a single target vehicle. In the actual scenario of using a single target vehicle, the penetration testing process is first defined as an MDP. The state s of the agent is defined by the information obtained by Nmap through port scanning; secondly, the actions a selected by the agent in the PT are defined, which correspond to the list of payload sequences in the Metasploit Framework (MSF); finally, the reward r is set according to the penetration result of the applied payload. Different DRL models can be used to train the agent, such as A2C, A3C, DPPO, etc. In addition, a multi-threaded mechanism can be used for training to speed up the learning efficiency of the agent.
[0054] Here we take A3C and DPPO as examples. Each A3C and DPPO model consists of a 4-layer neural network: an input layer and a hidden layer, with three fully connected layers and an output layer. The state input of the model, namely the operating system, port service, product version, protocol, and vulnerability exploit module type, are all obtained by scanning the target with the vulnerability scanning tool Nmap. The action output of the model corresponds to the probability distribution and state value of each payload in Metasploit. For the Actor network in the model, Softmax can be used as the activation function, and the output of the Actor network is the probability distribution P(a) of each action; while the Critic network uses Linear as the activation function, and its output is the state value V(s) corresponding to the current state. In the training process of Actor and Critic, a discriminator is introduced to participate in the training of Actor and Critic. In the training process based on the DRL model and GAIL, there are specific differences between the A3C model and the DPPO model. The difference between the value function and the discounted reward output by the discriminator is used as the advantage function to guide the training of the policy network. The output of the GAIL model is consistent with the original model.
[0055] The DRL model is considered as an attacker during the learning process. Based on the input vulnerability information, the agent will select a valid payload to attack the target vehicle until further obtaining root privileges. Therefore, obtaining the root privileges of the target vehicle is a sign of a successful attack. More precisely, it is a post-intrusion after the privilege escalation operation is completed, at which point the agent will receive a reward. In addition, the reward value r also depends on whether the PT attack is effective.
[0056] like Figure 3 As shown, if the agent performs a penetration attack from the Kali server to the target vehicle computer, establishes a session connection with the Kali server, and then executes the payload, it is advantageous to obtain the root privileges of the target vehicle computer. This shows that post-exploitation is booming and the agent will receive the highest reward. On the contrary, if the agent only launches a penetration attack without establishing a connection with the Kali server, the reward obtained by the agent at this time will be lower. In addition, if the agent does not successfully exploit the target vehicle computer, its reward will be negative. The specific settings of the reward are as follows:
[0057] -After the vulnerability is successful, r = 100;
[0058] -The vulnerability was successfully exploited, but the root privilege of the target vehicle computer was not obtained r=1;
[0059] - Exploitation failed, r = -1;
[0060] - For other operations, r = -1 regardless of whether the exploit was successful.
[0061] Negative reward r = -1 means punishing the agent, and the agent will always try to maximize the reward. Therefore, if the agent is punished with a negative reward, it will try to reach the penetration goal as soon as possible. This corresponds to the situation where an actual attacker finds a vulnerability and successfully exploits it. In addition, as long as the agent receives a reward, it means that each round of training is over.
[0062] Among them, for the penetration test request of the Internet of Vehicles environment, the penetration test process is modeled as a Markov game model, wherein the agent state is set to the vehicle computer configuration information and vulnerability information collected by the agent in different network environments, and the action is set to the operation of the agent to simulate the attack on the vehicle computer in the network environment. The simulated attack vector is formed according to the corresponding actions performed by the agent on different vehicle computers in the network environment; the reinforcement learning algorithm is used to automatically generate the penetration test path and optimize the path optimization. In the path optimization, the agent selects actions and obtains the action probability space, and obtains the reward value based on the vehicle computer value and the cost of executing the action on the vehicle computer; the optimal penetration path is explored by maximizing the cumulative reward, and the most valuable sensitive vehicle computer information is obtained by executing as few actions as possible.
[0063] The improved GAIL network is applied to the RL model to perform intelligent penetration testing in simulation and real IVI environments. The penetration testing process can also be modeled as an MDP. First, the state is defined as the configuration details and vulnerability information of all vehicle computers observed by the agent in different network environments. The more vehicles there are, the larger the state scale. In addition, the action refers to the vulnerability scanning, vulnerability exploitation, and privilege escalation operations performed by the agent after reaching the target vehicle computer. The operation is defined as an attack vector, m,c means that the agent operates m on vehicle computer c. In each training step, the size of the agent's operation space reaches (P×Q), where P is the number of vehicles in the network and Q represents the number of operations that can be performed using the final target vehicle computer. After the agent performs a specific operation, the environment will feedback the corresponding reward value. The calculation method of the reward value is as follows, including the value of the vehicle computer and the cost of exploiting, escalating privileges, or scanning the vehicle computer.
[0064]
[0065] Based on the RL algorithm Q-learning, the penetration path is automatically generated and its pathfinding process is optimized. During the training process, the agent will select the corresponding representative vulnerability according to the service to calculate the vulnerability probability. In the above formula, C represents the set of vehicles in different networks that the agent can exploit, and M represents the set of operations that the agent can take. The attacker's goal is to maximize the cumulative reward and explore the optimal penetration path until the most valuable sensitive target vehicle is successfully exploited with as few operations as possible.
[0066]
[0067] Subsequently, GAIL is combined with the Q-learning algorithm. The Q-learning-GAIL algorithm is applicable to three different situations: small-scale networks (considering the presence of honeypots) and large-scale networks, to train agents to optimize the search for penetration paths. The Q-learning algorithm is different from the DRL algorithm. Its optimization process is to continuously minimize the target value Q tar and the actual evaluation value Q eva The difference between the two is T D-error. When calculating the current value Q eva When the immediate reward r is equal to the discounted maximum target value Q in the next state, tarnex Therefore, in order to improve the optimization effect, the immediate reward r and the reward r output by the discriminator are added. d Superposition is used to guide the learning of imitation strategies so that the action distribution of the RL model is as close as possible to the PT expert samples.
[0068] In the embodiments of this case, by constructing an expert knowledge base for IVI security, the state-action pairs in various penetration test scenarios are included, so that the experience of penetration test experts can guide the intelligent system to make decisions under low-cost conditions, significantly improving the test efficiency and strategy quality. The improved GAIL network is applied to the deep reinforcement learning (DRL) and reinforcement learning (RL) models of IVI penetration testing. By integrating the RL / DRL model with the loss function of the Discriminator, the learning process of the intelligent agent is optimized, so that it can more stably and efficiently approach the expert-level penetration strategy, thereby showing the best performance in actual testing. The GAIL framework in this case solution has been extensively verified in simulations and real-vehicle IVI environments of different scales (including but not limited to small networks with honeypot defense mechanisms, real target car machines without honeypot defense, and large-scale network scenarios simulated in high-fidelity network simulators), all of which have demonstrated industry-leading penetration testing capabilities. By automatically collecting and learning expert experience, the intelligent agent can quickly discover and exploit security vulnerabilities in complex and ever-changing IVI environments, starting from a target vehicle computer in a subnet, controlling other associated vehicle computers through lateral movement until ultimately controlling the sensitive target vehicle computer, thereby effectively verifying the superiority of the GAIL-based automated penetration testing process in IVI environments of different scales.
[0069] Furthermore, based on the above method, an embodiment of the present invention also provides an intelligent penetration testing system for an Internet of Vehicles environment based on generative adversarial imitation learning, comprising: a test receiving module and a simulation test module, wherein:
[0070] A test receiving module, used to receive a penetration test request of a target object, and collect Internet of Vehicles environment data of the target object according to the penetration test request, wherein the target object is a target vehicle or a target Internet of Vehicles environment, and the Internet of Vehicles environment is a network environment of an in-vehicle infotainment system;
[0071] A simulation test module is used to input the target object's Internet of Vehicles environment data into a pre-trained generative adversarial imitation learning model, and use the generative adversarial imitation learning model to simulate attacks on the target object;
[0072] Among them, the generative adversarial imitation learning model is a simulated attack agent trained based on pre-acquired Internet of Vehicles environmental data and the Internet of Vehicles security expert experience knowledge base.
[0073] Unless otherwise specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present invention.
[0074] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0075] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.
[0076] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.
[0077] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A method for intelligent penetration testing in an Internet of Vehicles environment based on generative adversarial imitation learning, characterized in that: Include: Receive a penetration test request of a target object, and collect Internet of Vehicles environment data of the target object according to the penetration test request, wherein the target object is a target vehicle or a target Internet of Vehicles environment, and the Internet of Vehicles environment is a network environment of an in-vehicle infotainment system; Input the target object's Internet of Vehicles environment data into the pre-trained generative adversarial imitation learning model, and use the generative adversarial imitation learning model to simulate attacks on the target object; Among them, the generative adversarial imitation learning model is a simulated attack agent trained based on pre-acquired Internet of Vehicles environmental data and the Internet of Vehicles security expert experience knowledge base.
2. According to claim 1, the intelligent penetration testing method for Internet of Vehicles environment based on generative adversarial imitation learning is characterized in that: The training process of the generative adversarial imitation learning model includes: Perform penetration testing in a simulated network in an Internet of Vehicles environment, set a corresponding reward value according to the degree of intrusion during the penetration test, collect state-action pair sample data based on the reward value, and store the state-action pair sample data in an Internet of Vehicles security expert experience knowledge base, where the state refers to the vehicle computer configuration and vulnerability information in the Internet of Vehicles environment, and the action refers to the test-related operations performed on the vehicle computer; Collect Internet of Vehicles environment data, and build a generative adversarial imitation learning model based on a generator and a discriminator. The generator determines the probability distribution of the agent's action space based on the Internet of Vehicles environment state data, and the discriminator obtains a discounted reward for guiding learning based on the agent's action and the state-action pairs in the vehicle network safety expert experience knowledge base; The objective function of model training is constructed using the experience knowledge base loss of Internet of Vehicles security experts and the agent loss. Based on this objective function, the discriminator and generator network parameters are updated during model training.
3. The intelligent penetration testing method for Internet of Vehicles environment based on generative adversarial imitation learning according to claim 2 is characterized in that: The generator is a reinforcement learning model or a deep reinforcement learning model, which uses the reinforcement learning model or the deep reinforcement learning model to obtain the action corresponding to the state of the Internet of Vehicles environment, and inputs it into the discriminator for comparison with the state-action in the experience knowledge base of the Internet of Vehicles security expert.
4. The intelligent penetration testing method for Internet of Vehicles environment based on generative adversarial imitation learning according to claim 3 is characterized in that: The generator includes a policy network and a value network. The policy network and the value network are set as a maximum-minimum game process during the training process. The objective function of the minimum-maximum game process is expressed as: Among them, D is the discriminator, π represents the learning strategy, and π E represents the expert strategy, log D(s, a) represents the judgment of the discriminator on the state-action pair (s, a) fact data, log(1-D(s, a)) represents the judgment of the discriminator on the state-action pair (s, a) generated data, L GAIL Loss for generative adversarial imitation learning.
5. The method for intelligent penetration testing of Internet of Vehicles environment based on generative adversarial imitation learning according to claim 4 is characterized in that: The objective function of model training is expressed as: in, It represents the entropy of the imitation strategy, λ is the control coefficient of the strategy regularization term, and λ≥0, ω and θ represent the discriminator and generator network parameters respectively, S is the state set, and A is the action set.
6. The intelligent penetration testing method for Internet of Vehicles environment based on generative adversarial imitation learning according to claim 1 is characterized in that: Use the generative adversarial imitation learning model to simulate attacks on the target object, including: For the penetration test request of the target vehicle, the penetration test process is modeled as a Markov game model, where the state of the agent is set to the port scanning information and the action is set to the corresponding action in the penetration test payload sequence list; A multi-threaded mechanism is used to train the intelligent agent using a deep reinforcement learning algorithm. The state in which the intelligent agent obtains the root permissions of the target vehicle's Internet of Vehicles environment through actions is used as a sign of the successful status of the penetration test attack, and a reward and punishment mechanism is used to incentivize the intelligent agent to participate in the penetration test of the target vehicle's Internet of Vehicles environment.
7. The intelligent penetration testing method for Internet of Vehicles environment based on generative adversarial imitation learning according to claim 1 is characterized in that: Use the generative adversarial imitation learning model to simulate attacks on the target object, including: For the penetration test request of the Internet of Vehicles environment, the penetration test process is modeled as a Markov game model, in which the agent state is set to the vehicle computer configuration information and vulnerability information collected by the agent in different network environments, and the action is set to the operation of the agent to simulate the attack on the vehicle computer in the network environment. The simulated attack vector is formed according to the corresponding actions performed by the agent on different vehicle computers in the network environment; The reinforcement learning algorithm is used to automatically generate penetration test paths and optimize path optimization. In the path optimization, the intelligent agent selects actions and obtains the action probability space, and obtains the reward value based on the value of the vehicle computer and the cost of performing actions on the vehicle computer. The optimal penetration path is explored by maximizing the cumulative reward, and the most valuable sensitive vehicle computer information is obtained by performing as few actions as possible.
8. An intelligent penetration testing system for Internet of Vehicles environment based on generative adversarial imitation learning, characterized in that: Contains: test receiving module and simulation test module, among which, A test receiving module, used to receive a penetration test request of a target object, and collect Internet of Vehicles environment data of the target object according to the penetration test request, wherein the target object is a target vehicle or a target Internet of Vehicles environment, and the Internet of Vehicles environment is a network environment of an in-vehicle infotainment system; A simulation test module is used to input the target object's Internet of Vehicles environment data into a pre-trained generative adversarial imitation learning model, and use the generative adversarial imitation learning model to simulate attacks on the target object; Among them, the generative adversarial imitation learning model is a simulated attack agent trained based on pre-acquired Internet of Vehicles environmental data and the Internet of Vehicles security expert experience knowledge base.
9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.
Citation Information
Cited By
Transferable automatic penetration test method fusing reinforcement learning and HER algorithm and medium
CN119377971A
Attack optimization method and device of confrontation system, electronic equipment and storage medium
CN119892518A
Personified virtual test scene multi-background vehicle collaborative decision-making method
CN120805390A
Aircraft dispatching strategy generation method, device, equipment, medium and product
CN120823730A
