In-vehicle network intrusion detection method and system based on federal reinforcement learning

Abstract

The invention discloses an in-vehicle network intrusion detection method based on federal reinforcement learning. The method comprises the following steps: acquiring a historical in-vehicle data set and preprocessing the historical in-vehicle data set; performing key feature extraction on the preprocessed historical vehicle-mounted data set according to a random forest algorithm to obtain a key feature subset; performing initial training on a DQN model based on the key feature subset to obtain an initial DQN model; distributing the initial DQN model to each vehicle end, so that each vehicle end calls a corresponding real vehicle-mounted data set to retrain the initial DQN model, and uploads the retrained model parameters to a cloud end; and receiving model parameters uploaded by the vehicle ends, aggregating the model parameters by adopting a parameter aggregation algorithm, retraining the initial DQN model to obtain an updated DQN model, and continuously distributing the updated DQN model to the vehicle ends to realize iterative optimization of the model. On the basis of the method, the data privacy is guaranteed, and meanwhile, the adaptability of the DQN model to the heterogeneous vehicle-mounted environment and the intrusion detection precision are improved.

Description

Technical Field

[0001] This invention relates to the field of vehicle network security technology, specifically to a vehicle network intrusion detection method and system based on federated reinforcement learning. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, automotive Ethernet, with its advantages of high bandwidth and low latency, has gradually become the core architecture for connecting in-vehicle infotainment systems, ADAS (Advanced Driver Assistance Systems), and powertrain control systems. However, the openness and heterogeneity of in-vehicle networks expose them to severe security threats: attackers can interfere with normal vehicle communication and even control vehicle driving status by forging network data packets and launching DDoS attacks, causing traffic accidents.

[0003] The harm caused by in-vehicle cyberattacks is mainly manifested in the following aspects: First, attacks directly affect the vehicle control system, causing abnormalities in steering, braking, and other functions, threatening the lives of drivers and passengers; second, attacks may steal in-vehicle privacy data (such as driving trajectory and driving habits), infringing on user information security; third, large-scale in-vehicle cyberattacks may cause regional traffic paralysis, resulting in widespread social impact. Therefore, developing efficient and secure in-vehicle network intrusion detection systems is crucial to ensuring the safe operation of intelligent connected vehicles.

[0004] Traditional vehicle intrusion detection methods, such as rule-based feature matching and centralized machine learning models, can identify known attacks to some extent, but they have obvious drawbacks: 1) Rule matching relies on manually preset attack features, which is difficult to deal with new and unknown attacks; 2) Centralized models need to collect raw data from the vehicle to the cloud for training, which poses a risk of data privacy leakage; 3) The vehicle network environment is heterogeneous (different brands and models of vehicles have different communication protocols), and centralized models have poor adaptability, which can easily affect detection accuracy.

[0005] In recent years, the rise of federated learning and reinforcement learning technologies has provided new directions for vehicle-mounted intrusion detection. Federated learning enables distributed training where "the data remains stationary while the model moves," with the vehicle only uploading model parameters, thus ensuring data privacy. Reinforcement learning optimizes model strategies through dynamic trial and error, adapting to unknown attack scenarios. However, current technologies have not yet effectively combined these two approaches for vehicle-mounted network intrusion detection, and issues such as feature redundancy and low model iteration efficiency still exist. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, such as data privacy leakage in vehicular networks, poor model adaptability, and low intrusion detection accuracy, this invention provides a vehicular network intrusion detection method and system based on federated reinforcement learning. Through standardized data preprocessing, precise feature selection, and federated DQN model training and iteration, a closed-loop detection system is constructed, encompassing "data preprocessing—feature selection—cloud initialization—vehicle-side training—cloud aggregation." This system not only protects data privacy but also improves the adaptability of the DQN model to heterogeneous vehicular environments and enhances intrusion detection accuracy. It can effectively identify malicious attack behaviors in vehicular networks, providing strong protection for vehicular network security.

[0007] According to one aspect of the present invention, a method for vehicular network intrusion detection based on federated reinforcement learning is provided, comprising: step S1, acquiring and preprocessing a historical vehicular dataset; step S2, extracting key features from the preprocessed historical vehicular dataset using a random forest algorithm to obtain a key feature subset; step S3, initially training a DQN model based on the key feature subset to obtain an initial DQN model; step S4, distributing the initial DQN model to at least two vehicle terminals, so that each vehicle terminal calls the corresponding real vehicular dataset to retrain the initial DQN model, and uploading the retrained model parameters to the cloud; step S5, aggregating the model parameters uploaded by each vehicle terminal using a parameter aggregation algorithm, retraining the initial DQN model based on the aggregated model parameters to obtain an updated DQN model, further distributing the updated DQN model to each vehicle terminal, and returning to step S4 to achieve iterative optimization of the DQN model, thereby realizing vehicular network intrusion detection for the corresponding vehicle terminal based on the optimized DQN model.

[0008] Further, step S1 includes: acquiring a historical vehicle data set, including several historical vehicle data samples; deleting missing value samples from the historical vehicle data set; parsing and converting the historical vehicle data samples after deleting missing value samples into decimal; unifying the length of each historical vehicle data sample after parsing and conversion, including: when the length of the current historical vehicle data sample is greater than a preset length, truncating the first preset length of bytes from the current historical vehicle data sample; when the length of the current historical vehicle data sample is less than the preset length, increasing the length of the current historical vehicle data sample to the preset length by padding with zeros.

[0009] Furthermore, each historical vehicle data sample in the historical vehicle dataset consists of several vehicle features, including CAN ID, vehicle speed, time interval, and load length. Step S2 includes: constructing multiple decision trees based on the preprocessed historical vehicle dataset; wherein, during the construction of each decision tree, the information gain of each vehicle feature is first calculated, and the construction of nodes in each decision tree is completed based on the information gain of each vehicle feature; the cumulative information gain of each vehicle feature in all decision trees is summarized, and the importance score of each vehicle feature is calculated by combining the total information gain of all vehicle features; vehicle features with an importance score greater than a preset threshold are selected from each historical vehicle data sample to obtain the corresponding key feature samples, and then form a key feature subset.

[0010] Furthermore, the formula for calculating the importance score is as follows:

[0011] ,

[0012] in, The importance score of the vehicle feature F is represented by its significance score. This represents the total number of decision trees, where m represents the index of each decision tree. This represents the total number of all vehicle-mounted features involved in the calculation. Let f represent the information gain of the vehicle feature F in the m-th decision tree, and let f represent the index of the vehicle feature.

[0013] Further, in step S3, the DQN model includes an evaluation Q-network and a target Q-network. The evaluation Q-network is used to calculate the estimated Q-value corresponding to the current key feature sample, and the target Q-network is used to calculate the target Q-value based on its fixed model parameters. The estimated Q-value and the target Q-value are used to construct a loss function to guide the update of the model parameters in the evaluation Q-network. Both the evaluation Q-network and the target Q-network adopt a fully connected structure consisting of an input layer, a hidden layer, and an output layer. The input layer is used to extract shallow features from the key feature subset output in step S2. The hidden layer is used to extract deep features from the shallow features through nonlinear transformation of multiple neurons. The output layer is used to map the deep features to the final intrusion detection judgment result.

[0014] Further, step S3 includes: step S31, dividing the key feature subset into a training set and a validation set; step S32, initializing the learning rate, discount factor, and greedy coefficient of the DQN model; step S33, traversing the key feature samples in the training set, inputting the current key feature sample into the evaluation Q network to obtain the evaluation Q value of all actions; step S34, selecting an action based on the evaluation Q value of all actions according to the greedy coefficient, and calculating the reward value based on the matching degree between the selected action and the true label; step S35, storing the experience tuple composed of the current key feature sample, the selected action, the reward value, and the next key feature sample into the experience replay pool; step S36, using the next key feature sample, the selected action, the reward value, and the discount factor, calculating the target Q network through the target Q network. Value; Step S37, calculate the evaluation Q value through the evaluation Q network, and compare the evaluation Q value with the target value. The mean squared error of the value is used as the loss function to update the model parameters of the evaluation Q network; in step S38, after a preset number of update steps, the model parameter set of the evaluation Q network is synchronized to the model parameters of the target Q network; in step S39, the accuracy of the current DQN model is verified using the validation set, and training is stopped when the preset termination condition is reached to obtain the initial DQN model.

[0015] Furthermore, each vehicle terminal calls the corresponding real in-vehicle dataset to retrain the initial DQN model, and uploads the retrained model parameters to the cloud, including: receiving the initial DQN model and calling the local dataset after preprocessing and feature filtering; fixing the input layer parameters and hidden layer parameters of the initial DQN model and fine-tuning the output layer parameters; using the gradient descent algorithm to minimize the prediction error, calculating the loss value, and stopping training when the loss value is less than a preset threshold; extracting the trained model parameters and uploading them to the cloud through an encrypted channel.

[0016] Further, step S5 includes: calculating global parameters using a federated averaging algorithm based on the model parameters uploaded by each vehicle terminal; loading the global parameters into the initial DQN model, and retraining the initial DQN model using a cloud-annotated attack sample dataset to obtain an updated DQN model; distributing the updated DQN model to each vehicle terminal, and returning to step S4 to achieve iterative optimization of the DQN model, thereby realizing in-vehicle network intrusion detection for the corresponding vehicle terminal based on the optimized DQN model.

[0017] According to one aspect of the present invention, a vehicle network intrusion detection system based on federated reinforcement learning is provided, comprising: a data acquisition and preprocessing module for acquiring and preprocessing historical vehicle datasets; a key feature extraction module for extracting key features from the preprocessed historical vehicle datasets using a random forest algorithm to obtain a key feature subset; a cloud initialization module for initial training of the DQN model based on the key feature subset to obtain an initial DQN model; and a vehicle-side retraining module for distributing the initial DQN model to each vehicle, so that each vehicle calls a corresponding real vehicle dataset to retrain the initial DQN model, and uploads the retrained model parameters to the cloud. The cloud aggregation module receives model parameters uploaded by each vehicle terminal, aggregates the model parameters using a parameter aggregation algorithm, retrains the initial DQN model based on the aggregated model parameters to obtain an updated DQN model, and then distributes the updated DQN model to each vehicle terminal for repeated cloud-vehicle collaborative training to achieve iterative optimization of the DQN model. Finally, based on the optimized DQN model, the corresponding vehicle terminal can perform in-vehicle network intrusion detection.

[0018] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the aforementioned vehicular network intrusion detection method based on federated reinforcement learning.

[0019] The above technical solution constructs a closed-loop detection system of "data preprocessing—feature selection—cloud initialization—vehicle retraining—cloud aggregation". Specifically: In the data preprocessing stage, historical vehicle datasets are cleaned, formatted, and length-uniformed to ensure consistency and lay the foundation for subsequent DQN model training; in the feature selection stage, a subset of key features is selected using the random forest algorithm to reduce redundant information interference and improve model training efficiency; in the cloud initialization stage, the cloud trains an initial DQN model based on the selected subset of key features, distributes it to the vehicle, and then the vehicle retrains using local real vehicle datasets and uploads the model parameters; in the cloud aggregation stage, the cloud continuously improves the model detection performance through iterative aggregation to adapt to diverse vehicle network attack scenarios. It is evident that because this invention achieves "cloud-vehicle" collaborative training based on a federated learning architecture, the initial training uses historical vehicle datasets from public datasets, and subsequent training combines real vehicle datasets collected from each vehicle for local training and performance verification, thus ensuring adaptability to actual vehicle scenarios. Meanwhile, the vehicle does not need to upload its own real vehicle information, thus ensuring privacy and security.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] (1) By introducing a federated reinforcement learning framework, collaborative intelligent decision-making between the vehicle and the cloud is realized, and global model training is completed under the premise of protecting the privacy data of each vehicle, which effectively improves the distributed intrusion detection capability of the system.

[0022] (2) The adaptive optimization mechanism based on reinforcement learning can dynamically adjust the detection strategy according to the real-time vehicle network status, realize closed-loop optimization from environmental perception, strategy decision-making to abnormal response, and significantly improve the real-time performance and accuracy of detection.

[0023] (3) By utilizing the federated parameter aggregation and policy sharing mechanism among multiple vehicles, the DQN model can have stronger generalization and robustness when facing different vehicle models, different network topologies and new attack samples, thereby enhancing the overall defense capability of the system against complex vehicle attack scenarios. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a vehicle network intrusion detection method based on federated reinforcement learning, provided as an embodiment of the present invention.

[0026] Figure 2 This is a flowchart of the initial training of the cloud-based DQN model provided in this embodiment of the invention. Detailed Implementation

[0027] It should be noted that:

[0028] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0031] Please refer to the appendix for details. Figure 1 This invention provides a method for intrusion detection in vehicular networks based on federated reinforcement learning, specifically including the following steps:

[0032] Step S1: Obtain historical vehicle dataset and perform preprocessing.

[0033] It should be noted that the training and testing of this invention uses a dual-source data approach: "public dataset + real-world collected data". The public dataset (e.g., the TOW-IDS dataset) stores historical vehicle data samples collected via vehicular Ethernet. Therefore, several historical vehicle data samples can be directly obtained from the public dataset to obtain the historical vehicle dataset. The real-world collected data (including normal driving traffic, simulated DDoS attack traffic, replay attack traffic, etc.) refers to the actual vehicle data collected from each vehicle. During the iterative optimization of the DQN model (Deep Q-Network), the initial training uses the historical vehicle dataset from the public dataset for initial training. Subsequent training combines the real-world vehicle dataset collected from each vehicle for local training and performance verification on each vehicle, ensuring adaptation to real-world vehicle scenarios. However, both the historical vehicle dataset from the public dataset and the real-world vehicle dataset collected from each vehicle require preprocessing before being used for training the DQN model. Both the historical vehicle dataset and the real-world vehicle dataset consist of several vehicle data samples, each of which is a vehicle network data packet.

[0034] In step S1, the cloud obtains historical vehicle datasets from public datasets and preprocesses them. Specifically, firstly, the historical vehicle datasets are cleaned, including deleting missing value samples. Missing value samples refer to historical vehicle data samples where key fields (such as source IP, destination port, protocol type, frame length, etc.) are empty or invalid. This step ensures the quality of input data and avoids model bias or training instability caused by missing features. Subsequently, since the frame data lengths generated by different ECU nodes and gateway devices vary significantly during actual vehicle Ethernet communication, directly inputting them into the DQN model would lead to inconsistent feature dimensions. To ensure that the input data dimensions are consistent and match the model structure, this invention parses each historical vehicle data sample from pcap format into decimal, and then unifies the length of the decimal historical vehicle dataset to a preset length. pcap format is a general file format used to store vehicle network data packets, storing the original vehicle network data packets in binary form.

[0035] Furthermore, the lengths of the parsed and converted historical vehicle data samples are standardized, including: if the length of the current historical vehicle data sample is greater than a preset length, the first preset length of bytes are truncated from the current historical vehicle data sample; if the length of the current historical vehicle data sample is equal to the preset length, no length processing is performed; if the length of the current historical vehicle data sample is less than the preset length, the length of the current historical vehicle data sample is increased to the preset length by padding with zeros. It is understood that the preset length can be set according to requirements and is not limited here. In this embodiment, the preset length is... The corresponding formula is:

[0036] ,

[0037] in, The length of the current historical vehicle data sample before length processing. The length of the current historical vehicle data sample after length processing.

[0038] Step S2: Extract key features from the preprocessed historical vehicle dataset using the random forest algorithm to obtain a subset of key features.

[0039] In step S2, in the context of vehicular network data, the process of selecting features in a random forest can be viewed as a merit-based selection process: multiple decision trees are constructed in the cloud based on the preprocessed historical vehicular dataset. Each historical vehicular data sample in the historical dataset consists of several vehicular features and a true category label. The vehicular features include, but are not limited to, CAN ID, vehicle speed, time interval, and payload length. The true category label is the true category of the historical vehicular data sample (e.g., normal / attack). The construction process of the decision tree is essentially a continuous process of selecting the optimal vehicular features for node splitting. The basis for selecting the optimal vehicular features is their contribution to reducing data clutter (i.e., improving purity), and this contribution is measured by information gain. Therefore, during the construction of each decision tree, the information gain of each vehicle feature is repeatedly calculated to complete the construction of each node in the decision tree based on the information gain of each vehicle feature; the cumulative information gain of each vehicle feature in all decision trees is summarized, and the importance score of each vehicle feature is calculated by combining the total information gain of all vehicle features; vehicle features with importance scores greater than a preset threshold (i.e., vehicle features with high importance) are selected to form a subset of key features.

[0040] Furthermore, the decision tree employs binary splitting during the splitting process. A parent node (P) is divided into two child nodes (C1, C2) based on the rule corresponding to the selected optimal vehicle feature. A complete decision tree contains multiple layers of this splitting structure, forming a multi-level parent-child node relationship from the root node to the leaf nodes. Specifically, the decision tree is a combination of a series of judgment rules, for example, splitting first by rotational speed (feature A) and then by load length (feature B). The parent node (P) represents the vehicle network data packets to be classified (containing normal traffic and attack traffic, mixed together). The child nodes (C1, C2) represent the two sets of vehicle network data packets classified according to the rule corresponding to a certain optimal vehicle feature (e.g., Payload_Size > 100). Entropy (Ent): Describes the "chaos" of the data; all identical data represents "pure data," while half good and half bad data is called "chaotic data." Information gain (IG): How much "pure" the data becomes after classification using a certain vehicle feature (the more pure the data, the better the vehicle feature). Vehicle features (F): A column (i.e. an attribute) of the historical vehicle dataset, such as load length.

[0041] Furthermore, key features are extracted from the preprocessed historical vehicle dataset using the random forest algorithm, with the corresponding formula being:

[0042] ① Calculate the information gain of each vehicle feature. The corresponding information gain formula is:

[0043] ,

[0044] in, The entropy of the parent node (P) , child nodes , entropy, , Representing child nodes respectively , The proportion of samples, Information gain representing the vehicle-mounted characteristic F.

[0045] It should be noted that each decision tree includes multiple levels of parent-child node relationships. The formula for calculating information gain in a multi-level decision tree is exactly the same as that for a single-level node; the core is to subtract the weighted entropy of all child nodes from the entropy of the parent node, regardless of the number of levels in the decision tree. Furthermore, the formula for calculating entropy is... ,in, For category In the set The sample proportion in the data is denoted by n, which represents the total number of categories. Each category refers to the true category label of the vehicle data sample. In this embodiment, the true category label uses binary classification: normal or attack. In some feasible embodiments, the true category label can also use multi-class classification, such as normal, DoS attack, scanning attack, or other specific attack types.

[0046] ② Summarize the cumulative information gain of each vehicle feature across all decision trees, and combine it with the total information gain of all vehicle features to calculate the importance score of each vehicle feature. The corresponding feature importance score formula is as follows:

[0047] Let the vehicle-mounted characteristic F be in The total information gain in the decision trees is The total information gain of all features is Then the feature The formula for importance score is:

[0048]

[0049] in, The importance score of the vehicle feature F is represented by m, where m represents the index of the decision tree. This represents the total number of decision trees. This represents the information gain of the vehicle feature F in the m-th decision tree. This represents the total number of all vehicle features involved in the calculation, where f represents the serial number of the vehicle feature.

[0050] ③ Select the importance score from each historical vehicle data sample. ( A preset threshold (usually 5%) is used to select vehicle features to obtain corresponding key feature samples, which are then used to form a key feature subset. Understandably, the preset threshold can be set according to actual needs and is not limited here.

[0051] Step S3: Initially train the DQN model based on the key feature subset to obtain the initial DQN model.

[0052] In step S3, a DQN model is constructed in the cloud, and initial training is performed on the DQN model based on the key feature subset selected in step S2. The DQN model includes an evaluation Q-network Q(s,a;θ) and a target Q-network Q'(s,a;θ'). The evaluation Q-network calculates the estimated Q-value corresponding to the current key feature sample in real time. The estimated Q-value represents the evaluation Q-network's score for the action. The evaluation Q-network also dynamically updates the model parameters to make the estimated Q-value continuously approach the true Q-value. The core function of the target Q-network is to provide a stable target Q-value to ensure training convergence. It calculates the target Q-value based on its fixed model parameters. The estimated Q-value and the target Q-value are used to construct the loss function to guide the update of the model parameters in the evaluation Q-network. The target Q-network is not updated synchronously with the evaluation Q-network, but rather its parameters are copied at intervals to avoid training instability caused by frequent fluctuations in the target Q-value.

[0053] Furthermore, both the evaluation Q-network and the target Q-network adopt a fully connected structure of "input layer - hidden layer - output layer". The dimension of the input layer is equal to the dimension of the key feature subset after screening in step S2, and the output layer corresponds to the Q-value for determining the two types of actions: "normal traffic" (a0) and "attack traffic" (a1). The input layer is used to read the key features of network traffic packets, such as packet length, transmission protocol type, source / destination address, payload features, etc., converting these unstructured or discrete data into numerical vectors that the model can recognize. At the same time, it may perform normalization, feature screening, and other operations to lay the foundation for subsequent feature extraction, thereby extracting shallow features. The hidden layer is the core feature extraction and learning module of the DQN model, responsible for mining deep patterns in the data. It extracts complex deep features from the shallow features of the input layer through nonlinear transformations of multiple layers of neurons. For example, the differences between normal traffic and intrusion traffic in terms of temporal patterns, interaction frequency, and data distribution are key to distinguishing normal communication from malicious attacks (such as DoS, injection attacks, etc.). The number of hidden layers and neurons directly affects the DQN model's ability to recognize complex attack patterns. The output layer is the decision and result output module of the DQN model. It maps the deep features extracted from the hidden layer to specific intrusion detection judgment results, i.e., Q-values ​​(numerical values). It's important to note that while the direct outputs of the evaluation Q-network and the target Q-network are Q-values, in practical applications, these Q-values ​​are converted into category labels. The conversion process involves selecting the action corresponding to the maximum Q-value based on a greedy strategy or the Argmax function, and mapping it to the text label "attack" or "normal." Therefore, the output of the DQN model is presented as a probability value or a clear category label, providing the vehicle system with a basis for judging whether intrusion behavior has occurred, supporting subsequent security response operations.

[0054] Furthermore, the initial training process of the DQN model in the cloud is as follows (e.g.) Figure 2(as shown)

[0055] Step S31: Divide the key feature subset into a training set and a validation set. In this embodiment, the key feature subset obtained in step S2 is divided into a training set in a 7:3 ratio. With the validation set .

[0056] Step S32, initialize the learning rate of the DQN model. Discount factor The capacity of the experience replay pool and the greedy algorithm coefficient. In this embodiment, the learning rate... Discount factor Experience replay pool capacity Greed coefficient (Linearly decays to 0.1). It should be noted that the learning rate... Discount factor The initial values ​​of the experience replay pool capacity and the greedy coefficient can be set according to actual needs, and are not limited here.

[0057] Step S33, Traverse the training set The key feature samples are input into the evaluation Q-network. The DQN model defines key feature samples as states s, and classifying the current key feature sample as action a, including actions such as classifying the current key feature sample as normal traffic or attack traffic. The evaluation Q network scores all actions to obtain the evaluation Q value of all actions.

[0058] Step S34, based on the evaluation Q-value of all actions... Coefficient selection action The reward value is calculated based on the degree of match between the selected action and the real label.

[0059] In step S34, the action with the highest Q-value under the current key feature sample is selected according to the greedy coefficient. According to the action With real labels Matching degree calculation reward value Each key feature sample in the key feature subset carries its own true label. The selected action falls into three categories: if the true label of the current key feature sample is attack traffic / normal traffic, and the judgment result corresponding to the selected action is also attack traffic / normal traffic, then the reward value is... =1; if the true label of the current key feature sample is normal traffic ( However, the judgment result corresponding to the selected action is attack traffic. Then the reward value =-1; if the true label of the current key feature sample is attack traffic. However, the judgment result corresponding to the selected action is normal flow. Then the reward value =-2. The corresponding formula is as follows:

[0060] ,

[0061] Step S35: Store the experience tuple consisting of the current key feature sample, the selected action, the reward value, and the next key feature sample into the experience replay pool.

[0062] In step S35, the current key feature samples are... Selected action Reward Value Next key feature sample The empirical tuples storage It should be noted that steps S33 to S35 are actually a repetitive process, in which experience tuples are continuously added to the experience replay pool until the pool capacity reaches a preset capacity (e.g., half the capacity of the experience replay pool). ( ), randomly sample several empirical tuples from the empirical replay pool. For example, sample An empirical tuple.

[0063] Step S36: Using the next key feature sample, the selected action, the reward value, and the discount factor, calculate the target value through the target Q-network. value.

[0064] ,

[0065] in, For the goal value, This represents the output value of the target Q-network. This is the discount factor.

[0066] Step S37: Calculate the evaluation Q-value using the evaluation Q-network, and then compare the evaluation Q-value with the target... The mean square error of the value is used as the loss function Update and evaluate the Q network Model parameters Among them, model parameters This includes, but is not limited to, the weights and biases of neurons in the network.

[0067] Step S38, each Preset update steps The set of model parameters for evaluating Q-networks will be used. synchronous Model parameters of the target Q-network Here, the preset update step number refers to the number of iterations, for example, The step represents the evaluation of the Q network. Model parameters The model parameters of the target Q network are updated every 100 times. Updated once simultaneously.

[0068] Step S39, using the validation set To verify the accuracy of the current DQN model, training stops when a preset termination condition is met (e.g., the accuracy of the DQN model is >95% for 10 consecutive rounds), thus obtaining the initial DQN model. It should be noted that the preset update steps and preset termination condition can be set according to actual needs and are not limited here.

[0069] Step S4: Distribute the initial DQN model to each vehicle terminal so that each vehicle terminal can call the corresponding real vehicle-mounted dataset to retrain the initial DQN model and upload the retrained model parameters to the cloud.

[0070] In step S4, the cloud distributes the initially trained DQN model to each vehicle. Each vehicle then uses the locally preprocessed and feature-selected real vehicle dataset to retrain the initial DQN model and uploads the retrained model parameters to the cloud. The local preprocessing and feature selection methods are the same as those used for the cloud-based historical vehicle dataset in step S1, and will not be described in detail here.

[0071] Furthermore, each vehicle uses the corresponding real-world in-vehicle dataset to retrain the initial DQN model, and then uploads the retrained model parameters to the cloud. This includes:

[0072] Step S41: Each vehicle receives the initial DQN model and calls the preprocessed and feature-selected real vehicle dataset. .

[0073] Step S42: Fix the input layer parameters and hidden layer parameters of the initial DQN model at each vehicle end, and fine-tune the output layer parameters.

[0074] Step S43: Each vehicle uses the gradient descent algorithm to minimize the prediction error of the initial DQN model, and simultaneously calculates the loss value. Training stops when the loss value is less than a preset threshold. In this embodiment, the training rounds are set to 20-50 rounds, and the loss value is calculated in each round. When the loss value Training should be stopped at this time.

[0075] Step S44: Each vehicle extracts the trained model parameters and uploads them to the cloud via an encrypted channel. In this embodiment, the extracted trained model parameters (including but not limited to the weights of each layer) are... Bias It is uploaded to the cloud via a TLS 1.3 encrypted channel.

[0076] Step S5: Receive model parameters uploaded by each vehicle terminal, aggregate the model parameters using a parameter aggregation algorithm, retrain the initial DQN model based on the aggregated model parameters to obtain an updated DQN model, distribute the updated DQN model to each vehicle terminal, and return to step S4 to achieve iterative optimization of the DQN model, and then realize vehicle network intrusion detection on the corresponding vehicle terminal based on the optimized DQN model.

[0077] In step S5, the cloud implements a "cloud aggregation - model update - vehicle iteration" loop without an iteration termination condition. Steps S4 and S5 need to be repeatedly executed to continuously iterate and optimize the DQN model, thereby facilitating the detection of new attacks. Understandably, the DQN model retrained in step S4 is essentially the vehicle network intrusion detection model, used for intrusion detection on the corresponding vehicle. However, since the DQN model in step S4 is retrained based on the initial DQN model distributed in the cloud, and the initial DQN model distributed in the cloud has limited accuracy, the accuracy of the retrained DQN model is also limited, and the results of vehicle network intrusion detection are naturally inaccurate. Therefore, the main purpose of step S4 is to obtain updated model parameters through retraining and upload them to the cloud so that in the subsequent step S5, the cloud globally updates the initial DQN model based on the model parameters uploaded by each vehicle, and then redistributes the updated DQN model to each vehicle. Each vehicle continues to retrain based on the new DQN model and local data, thereby continuously improving detection accuracy and achieving continuous model optimization. This results in better detection results for subsequent updates to the vehicle network intrusion detection model on each vehicle.

[0078] Step S51, Parameter Aggregation: Based on the model parameters uploaded by each vehicle, the global parameters are calculated using a federated averaging algorithm. The corresponding formula is:

[0079] ,

[0080] in, Indicates a global parameter. Indicates the serial number of the vehicle end. This indicates the total number of vehicles participating in the training. Indicates the first Sample size per vehicle Indicates the first Model parameters uploaded by each vehicle.

[0081] Step S52, Model Retraining: Load the global parameters into the initial DQN model, and retrain the initial DQN model using the cloud-annotated attack sample dataset to obtain the updated DQN model.

[0082] In step 52, the global parameters are... Loading to the initial DQN model refers to loading the model parameters of the initial model. The values ​​are directly assigned to the global parameters calculated by the weighted average. Cloud-based labeled attack sample dataset The dataset comes from the historical vehicle data set in step S1. In this embodiment, the initial DQN model is retrained for 10-20 rounds to optimize generalization ability.

[0083] Step S53, Iterative Distribution: Distribute the updated DQN model to each vehicle terminal so that each vehicle terminal replaces the old DQN model and retrains the DQN model again. Return to step S4 to achieve iterative optimization of the DQN model, and then realize the vehicle network intrusion detection of the corresponding vehicle terminal based on the optimized DQN model.

[0084] Based on the same technical concept as the aforementioned embodiments, this invention also provides a vehicle network intrusion detection system based on federated reinforcement learning, comprising: a data acquisition and preprocessing module for acquiring and preprocessing historical vehicle datasets; a key feature extraction module for extracting key features from the preprocessed historical vehicle datasets using a random forest algorithm to obtain a key feature subset; a cloud initialization module for initial training of the DQN model based on the key feature subset to obtain an initial DQN model; a vehicle-side retraining module for distributing the initial DQN model to each vehicle, enabling each vehicle to call the corresponding real vehicle dataset to retrain the initial DQN model, and uploading the retrained model parameters to the cloud; and a cloud aggregation module for receiving the model parameters uploaded by each vehicle, aggregating the model parameters using a parameter aggregation algorithm, retraining the initial DQN model based on the aggregated model parameters to obtain an updated DQN model, and continuing to distribute the updated DQN model to each vehicle, repeating the collaborative training between the cloud and the vehicle to achieve iterative optimization of the DQN model.

[0085] Based on the same technical concept as the foregoing embodiments, the present invention also provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute the aforementioned vehicular network intrusion detection method based on federated reinforcement learning.

[0086] In summary, this invention combines the distributed training characteristics of federated learning with the dynamic optimization capabilities of reinforcement learning to construct a closed-loop detection system encompassing "data preprocessing—feature selection—cloud initialization—vehicle retraining—cloud aggregation." Specifically: In the data preprocessing stage, historical vehicle datasets are cleaned, format-converted, and length-uniformed to ensure consistency, laying the foundation for subsequent DQN model training; in the feature selection stage, a subset of key features is selected using the random forest algorithm to reduce redundant information interference and improve model training efficiency; in the cloud initialization stage, the cloud trains an initial DQN model based on the selected subset of key features, distributes it to the vehicle, and then retrains the model using local real vehicle datasets and uploads the model parameters; in the cloud aggregation stage, the cloud continuously improves model detection performance through iterative aggregation to adapt to diverse vehicle network attack scenarios. Furthermore, because this invention achieves "cloud-vehicle" collaborative training based on a federated learning architecture, the initial training uses historical vehicle datasets from public datasets, and subsequent training combines real vehicle datasets collected from each vehicle for local training and performance verification, ensuring adaptation to actual vehicle scenarios. Meanwhile, vehicle-side data does not need to upload its own real vehicle information, ensuring privacy and security. Therefore, this invention improves the adaptability of the DQN model to heterogeneous vehicle environments and the accuracy of intrusion detection while protecting data privacy, and can effectively identify malicious attack behaviors in vehicle networks.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for intrusion detection in vehicular networks based on federated reinforcement learning, characterized in that, include: Step S1: Obtain historical vehicle data sets and perform preprocessing; Step S2: Extract key features from the preprocessed historical vehicle dataset using the random forest algorithm to obtain a subset of key features; Step S3: Perform initial training on the DQN model based on the key feature subset to obtain the initial DQN model; Step S4: Distribute the initial DQN model to at least two vehicle terminals, so that each vehicle terminal can call the corresponding real vehicle dataset to retrain the initial DQN model, and upload the retrained model parameters to the cloud. Step S5: The parameter aggregation algorithm is used to aggregate the model parameters uploaded by each vehicle terminal. The initial DQN model is retrained based on the aggregated model parameters to obtain the updated DQN model. The updated DQN model is then distributed to each vehicle terminal. The process returns to step S4 to achieve iterative optimization of the DQN model. Finally, the vehicle network intrusion detection of the corresponding vehicle terminal is achieved based on the optimized DQN model.

2. The intrusion detection method for vehicular networks based on federated reinforcement learning as described in claim 1, characterized in that, Step S1 includes: Obtain historical vehicle data sets, including several historical vehicle data samples; Delete missing value samples from the historical vehicle dataset; The historical vehicle data samples after deleting missing value samples are parsed and converted into decimal; The length of each historical vehicle data sample after parsing and conversion is unified, including: when the length of the current historical vehicle data sample is greater than the preset length, the first preset length of bytes in the current historical vehicle data sample is extracted; when the length of the current historical vehicle data sample is less than the preset length, the length of the current historical vehicle data sample is increased to the preset length by padding with zeros.

3. The intrusion detection method for vehicular networks based on federated reinforcement learning as described in claim 1, characterized in that, Each historical vehicle data sample in the historical vehicle dataset consists of several vehicle features, including CANID, vehicle speed, time interval, and load length. Step S2 includes: Multiple decision trees are constructed based on the preprocessed historical vehicle data set. In the construction of each decision tree, the information gain of each vehicle feature is calculated first, and the nodes in each decision tree are constructed based on the information gain of each vehicle feature. The cumulative information gain of each vehicle feature across all decision trees is summarized, and the total information gain of all vehicle features is combined to calculate the importance score of each vehicle feature. Vehicle features with importance scores greater than a preset threshold are selected from each historical vehicle data sample to obtain corresponding key feature samples, which are then used to form a key feature subset.

4. The intrusion detection method for vehicular networks based on federated reinforcement learning as described in claim 3, characterized in that, The formula for calculating importance score is: ， in, The importance score of the vehicle feature F is represented by its weight. This represents the total number of decision trees, where m represents the index of each decision tree. This represents the total number of all vehicle-mounted features involved in the calculation. Let f represent the information gain of the vehicle feature F in the m-th decision tree, and let f represent the index of the vehicle feature.

5. The intrusion detection method for vehicular networks based on federated reinforcement learning as described in claim 1, characterized in that, In step S3, the DQN model includes an evaluation Q-network and a target Q-network. The evaluation Q-network is used to calculate the estimated Q-value corresponding to the current key feature sample, and the target Q-network is used to calculate the target Q-value based on its fixed model parameters. The estimated Q-value and the target Q-value are used to construct a loss function to guide the update of the model parameters in the evaluation Q-network. Both the evaluation Q-network and the target Q-network adopt a fully connected structure consisting of an input layer, a hidden layer, and an output layer. The input layer is used to extract shallow features from the key feature subset output in step S2. The hidden layer is used to extract deep features from the shallow features through nonlinear transformation of multiple neurons. The output layer is used to map the deep features to the final intrusion detection judgment result.

6. The intrusion detection method for vehicular networks based on federated reinforcement learning as described in claim 5, characterized in that, Step S3 includes: Step S31: Divide the key feature subset into a training set and a validation set; Step S32: Initialize the learning rate, discount factor, and greedy coefficient of the DQN model; Step S33: Traverse the key feature samples in the training set, input the current key feature sample into the evaluation Q network to obtain the evaluation Q value of all actions; Step S34: Select an action based on the evaluation Q value of all actions according to the greedy coefficient, and calculate the reward value according to the matching degree between the selected action and the real label. Step S35: Store the experience tuple consisting of the current key feature sample, the selected action, the reward value, and the next key feature sample into the experience replay pool. Step S36: Using the next key feature sample, the selected action, the reward value, and the discount factor, calculate the target value through the target Q-network. value; Step S37: Calculate the evaluation Q-value using the evaluation Q-network, and then compare the evaluation Q-value with the target... The mean squared error of the values ​​is used as the loss function to update the model parameters of the Q-network for evaluation. Step S38: After a preset number of update steps, synchronize the model parameter set of the evaluation Q network to the model parameters of the target Q network. Step S39: Validate the accuracy of the current DQN model using the validation set. Stop training when the preset termination condition is met to obtain the initial DQN model.

7. The intrusion detection method for vehicular networks based on federated reinforcement learning as described in claim 1, characterized in that, Each vehicle terminal calls the corresponding real in-vehicle dataset to retrain the initial DQN model, and uploads the retrained model parameters to the cloud, including: Receive the initial DQN model and call the preprocessed and feature-selected local dataset; The input layer parameters and hidden layer parameters of the initial DQN model are fixed, and the output layer parameters are fine-tuned. The gradient descent algorithm is used to minimize the prediction error, the loss value is calculated, and training is stopped when the loss value is less than a preset threshold. Extract the trained model parameters and upload them to the cloud via an encrypted channel.

8. The intrusion detection method for vehicular networks based on federated reinforcement learning as described in claim 1, characterized in that, Step S5 includes: Based on the model parameters uploaded by each vehicle, the global parameters are calculated using a federated averaging algorithm. The global parameters are loaded into the initial DQN model, and the initial DQN model is retrained using the cloud-annotated attack sample dataset to obtain the updated DQN model. The updated DQN model is distributed to each vehicle terminal, and the process returns to step S4 to achieve iterative optimization of the DQN model. Then, based on the optimized DQN model, the in-vehicle network intrusion detection of the corresponding vehicle terminal is realized.

9. A vehicle network intrusion detection system based on federated reinforcement learning, characterized in that, include: The data acquisition and preprocessing module is used to acquire historical vehicle-mounted datasets and perform preprocessing. The key feature extraction module is used to extract key features from the preprocessed historical vehicle dataset using the random forest algorithm to obtain a subset of key features. The cloud initialization module is used to perform initial training on the DQN model based on the key feature subset to obtain an initial DQN model; The vehicle-side retraining module is used to distribute the initial DQN model to each vehicle, so that each vehicle can call the corresponding real vehicle dataset to retrain the initial DQN model, and upload the retrained model parameters to the cloud. The cloud aggregation module receives model parameters uploaded by each vehicle terminal, aggregates the model parameters using a parameter aggregation algorithm, retrains the initial DQN model based on the aggregated model parameters to obtain an updated DQN model, and then distributes the updated DQN model to each vehicle terminal for repeated cloud-vehicle collaborative training to achieve iterative optimization of the DQN model. Finally, based on the optimized DQN model, the corresponding vehicle terminal can perform in-vehicle network intrusion detection.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the intrusion detection method for vehicular networks based on federated reinforcement learning as described in any one of claims 1 to 8.

Images