Ics intrusion detection system and method fusing reinforcement learning and feature selection optimization

Abstract

The application discloses an ICS intrusion detection system and method fusing reinforcement learning and feature selection optimization, carries out binary coding and population initialization on data feature selection of an industrial control system (ICS) historical data set, carries out offline training through SVM-reinforcement learning, takes the accuracy obtained on a verification set as a fitness function, designs cross operation and mutation operation based on cumulative probability to update the population, and obtains an optimal feature set after iterative optimization; feature selection is carried out on an ICS real-time data set based on the optimal feature set, online intrusion detection testing is carried out on the real-time data set through support vector machine (SVM)-reinforcement learning, and thus an intrusion detection performance index is obtained. The application adopts a new mode based on SVM-reinforcement learning, and fuses intelligent optimization of optimal feature selection on this basis, and improves the intelligent design level and the precision of intrusion detection of the ICS intrusion detection system.

Description

An ICS Intrusion Detection System and Method Integrating Reinforcement Learning and Feature Selection Optimization Technical Field

[0001] This invention relates to intelligent intrusion detection technology in the field of industrial control system information security, and particularly to an ICS intrusion detection system and method that integrates reinforcement learning and feature selection optimization. Background Technology

[0002] Industry is the leading sector of the national economy, and Industrial Control Systems (ICS) are used to manage, guide, and regulate the behavior of automated industrial processes. Therefore, ICS are crucial for the safe, reliable, and efficient operation of industry; they can be considered the brain of industry. However, the high degree of integration between information technology and industrialization today has made industrial control systems more precise and efficient, but it also presents increasingly severe security threats.

[0003] Intrusion Detection Systems (IDS) monitor system operation and analyze various attack objectives, behaviors, and results to ensure the confidentiality, integrity, and availability of system resources. IDS is a core technology for ensuring the security of industrial control systems, and therefore has received widespread attention from both academia and industry. However, traditional IDS often require complex training processes and their accuracy needs improvement when dealing with the current high-real-time, large-volume, and complex multi-dimensional industrial IoT data. Therefore, how to perform optimal feature selection to efficiently compress ICS datasets and improve the operational efficiency and performance of IDS has become one of the urgent technical challenges to be solved by academia and industry.

[0004] In recent years, deep learning-based intrusion detection technology for industrial control systems (ICS) has become a research hotspot in both academia and industry. Deep reinforcement learning not only combines deep neural networks and reinforcement learning, enabling it to maximize rewards from known network environments, but also possesses exploratory capabilities, automatically mining more valuable information within the network environment, and exhibiting fast model convergence. Support Vector Machines (SVMs) demonstrate good generalization performance, are less prone to overfitting, and perform well even with limited and high-dimensional data. However, currently, there are few research reports in the field of ICS intrusion detection that integrate the novel SVM-reinforcement learning model.

[0005] In summary, the novel ICS intrusion detection technology that integrates reinforcement learning and intelligent feature selection optimization has significant value and urgently needs further development. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing an ICS intrusion detection system and method that integrates reinforcement learning and feature selection optimization.

[0007] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides an ICS intrusion detection system that integrates reinforcement learning and feature selection optimization. The system includes an ICS intrusion detection data acquisition module, an offline feature selection optimization module based on SVM-reinforcement learning, and an online intrusion detection module based on optimal features using SVM-reinforcement learning.

[0008] The ICS intrusion detection data acquisition module is used to collect historical data on the production process monitoring status from the historical database of the industrial control system. After data parsing, the source dataset is obtained and labeled as X. The data is then normalized according to formula (1) to obtain an offline training dataset, which is then labeled as X. o X o Randomly partition the dataset proportionally to obtain the training set X. o1 With validation set X o2 The training set X o1 With validation set X o2 The input is fed into the SVM-reinforcement learning-based feature selection offline optimization module;

[0009]

[0010] Among them, X oj Represents the offline training dataset X o The feature of the j-th dimension, X j Let X represent the feature of the j-th dimension of the source dataset X. jmax With X jmin They represent X respectively j The minimum and maximum values ​​of X, where d represents X. o The maximum value of the dimension;

[0011] The ICS intrusion detection data acquisition module is used to collect real-time monitoring data of the production process from the ICS real-time database. After data parsing and data normalization, an online detection dataset is obtained, and the online detection dataset is input into the SVM-reinforcement learning online intrusion detection module based on optimal features.

[0012] The SVM-reinforcement learning-based feature selection offline optimization module is used to encode the data features of the ICS historical dataset into binary and initialize the population. It is then trained offline using SVM-reinforcement learning, and the accuracy obtained on the validation set is used as the fitness function. The population is updated based on the cumulative probability of crossover and mutation operations. After iterative optimization, the optimal feature set is obtained, and the obtained optimal feature set is input into the SVM-reinforcement learning-based online intrusion detection module.

[0013] The SVM-reinforcement learning online intrusion detection module based on optimal features is used to select features from the ICS real-time dataset based on the optimal feature set, and to conduct online intrusion detection tests on the real-time dataset through SVM-reinforcement learning to obtain intrusion detection performance indicators.

[0014] A second aspect of this invention provides an ICS intrusion detection method that integrates reinforcement learning and feature selection optimization using the above-described system, comprising the following steps:

[0015] (1) Set the parameter values ​​for the feature selection offline optimization module based on SVM-reinforcement learning. The parameter values ​​include population size N, number of iterations I, and crossover rate P. c Variation rate P m The training rounds EP for reinforcement learning, and the network parameters of the reinforcement learning actor and the critic, wherein the network parameters of the reinforcement learning actor and the critic include the number of network layers D, the learning rate lr1 of the reinforcement learning actor, the learning rate lr2 of the critic, the loss function loss, the optimizer optimizer1 of the reinforcement learning actor, and the optimizer optimizer2 of the critic.

[0016] (2) Take the N randomly generated binary-coded individuals as the initial population Q = {Ind} j For each individual in the sequence {j = 1, 2, ..., N}, the encoding is binary, and the encoding format is Ind. j =(a1,a2,…,a i ,…,a n ), here Ind j a represents the j-th individual in the population. i This represents the encoding of each gene in an individual, 1≤i≤n, where n represents the maximum number of features in the dataset, and a i The specific definition formula (2) is expressed as follows:

[0017]

[0018] Among them, each Ind j The initialization is to randomly initialize n a's. iThat is, a binary string of length n is randomly generated, and the information in the binary string represents the selection combination of n features in the dataset;

[0019] (3) The performance of population Q is evaluated. The specific evaluation process is as follows:

[0020] (3.1) Based on the binary encoding information corresponding to each individual in population Q, the offline training dataset X is... o1 conduct

[0021] After feature selection, the data is used as input for offline training of SVM-reinforcement learning;

[0022] (3.2) The implementation process of SVM-reinforcement learning offline training is as follows:

[0023] Initialize the reinforcement learning environment class `env`. This class has two attributes: the group size of the data.

[0024] `batch_size` and `state_shape` are the dimensions of the data; the `env` class has two functions: a state update function.

[0025] upstate and action function act; the return value of the state update function upstate is the state state, and the function of the state update function upstate is to randomly select a batch of data of size batch_size from the dataset, name it state, and return it.

[0026] The action function `act` takes an action and a state as input, and outputs the next state and a reward.

[0027] The function `act` is implemented by comparing the action and the label of the state. If the action value and the label value are equal, the reward for the action is 1; if they are not equal, the reward for the action is 0. Then, `upstate` is called to update the state and get state +1, and the reward and state +1 are returned.

[0028] The reinforcement learning actor network is initialized with a 3-layer network. The first layer is the input layer, the second layer is the hidden layer with a network size of hidden_size, and uses the Gaussian kernel function as the mapping. The loss function of the network is the hinge loss function, which corresponds one-to-one with the SVM based on the Gaussian kernel function. The third layer is the output layer, the optimizer is optimizer1, and the learning rate is lr1.

[0029] The critic initialization uses a 3-layer network, all of which are fully connected layers. The second hidden layer has a network size of hidden_size, an optimizer of optimizer2, and a learning rate of lr2.

[0030] The env class calls the state update function upstate to get the state, and then selects a random number r = random(0,1), where random(0,1) means randomly selecting a decimal number between 0 and 1;

[0031] The specific calculation of action is shown in formula (3):

[0032]

[0033] Where ε is a hyperparameter used to determine the choice of action, n D Represents the offline training dataset X o1 The total number of data points after feature selection, α h Indicates the action of the previous step, y h This indicates the data labels from the previous step, δ is a parameter of the Gaussian function, and c is a bias value. The state represents the total population, and s represents... A state in the context of a process.

[0034] Let A represent the state randomly selected in the h-th iteration, and let A represent the overall set of actions. Env executes the action function act, takes action as input, and outputs reward and the next state, called state+1. Then, the critic evaluates the reinforcement learning actor based on state, reward, and state+1, according to formula (4), to obtain td_error, thereby updating the reinforcement learning.

[0035] Actors:

[0036] td_error = reward + d × LAM × (V - -V)(4)

[0037] Where V represents the critics' evaluation of the state, V -The value represents the critic's evaluation of state+1. LAM is a hyperparameter representing the degree of connection between the state and state+1 data. d indicates whether to add the subsequent results. When d=0, the subsequent results are discarded. When d is a decimal between 0 and 1, the subsequent results are added according to the weights. Based on the obtained td_error, the reinforcement learning actor updates its own network parameter weights using action, state, and td_error.

[0038] (3.3) Perform offline training for the SVM-reinforcement learning EP rounds on the validation set X. o2 Perform verification tests to verify set X o2 Using the accuracy rate as the standard, the final accuracy rate for each individual is calculated and used as the individual fitness value f for each individual. j (j = 1, 2, ..., N), and determine the individual with the highest accuracy as the optimal individual Indi. best ;

[0039] (4) Calculate the probability p of each individual in population Q being inherited into the next generation population according to formula (5). j (j = 1, 2, ..., N), calculate the cumulative probability q of each individual according to formula (6). j :

[0040]

[0041]

[0042] (5) Generate a random number between 0 and 1, r1 = random(0,1). If r1 ≤ crossover rate P c If the condition is met, then a crossover operation is performed; otherwise, no crossover operation is performed. The specific steps for implementing the crossover operation are as follows: A random number r2 = random(0,1) between 0 and 1 is generated. If... Then select the j1-th individual from the population Q; randomly generate a random number between 0 and 1, r3 = random(0,1) and r3 ≠ r2, if Then, select the j2-th individual from the population Q, and then randomly generate an integer k = randint(1,n) between 1 and n, where randint(1,n) represents randomly generating an integer between 1 and n. Then, assign the code a corresponding to the k-th position of the j1-th and j2-th individuals to this code. k Perform a crossover operation, swapping values ​​while keeping other values ​​unchanged; label the resulting population as Q. c ;

[0043] (6) From population Qc A random individual is selected from the sample, and a random number r4 = random(0,1) is generated. If r4 ≤ the mutation rate P, then the result is positive. m If the mutation operation is successful, then the individual is mutated; otherwise, no mutation operation is performed. The mutation operation specifically involves: randomly generating an integer m = randint(1, n) between 1 and n, and encoding the m-th position of the individual to be mutated using the given code a. m Perform binary mutation, if the original a m If the value is 0, then the mutated a m If the original a becomes 1, m If the value is 1, then the mutated a m Change to 0; mark the mutated population as Q. m ;

[0044] (7) Unconditionally accept Q = Q m ;

[0045] (8) Repeat steps (3) to (7) until the set number of iterations I is met;

[0046] (9) Obtain the optimal individual Indi best To obtain Indi best The optimal feature set corresponding to the binary encoded sequence;

[0047] (10) Based on the optimal feature set, the ICS real-time dataset X r Perform feature selection to obtain a subset of data X t ;

[0048] (11) Apply SVM-reinforcement learning to the obtained data subset X according to step (3.2). t Online intrusion detection tests were conducted, alarms were triggered for detected abnormal data, and the accuracy, recall, and F1 score of the intrusion detection were calculated according to formulas (7) to (9):

[0049]

[0050]

[0051]

[0052] Wherein, TP represents the number of normal ICS data samples correctly predicted as normal categories, TN represents the number of abnormal ICS data samples correctly predicted as abnormal categories, FP represents the number of abnormal ICS data samples incorrectly predicted as normal categories, and FN represents the number of normal ICS data samples incorrectly predicted as abnormal categories.

[0053] The beneficial effects of this invention are that it reduces redundancy in the ICS dataset through intelligent feature selection optimization, thereby removing relevant noise and non-message data, reducing the space and time complexity of the algorithm, and alleviating the overfitting problem. Moreover, this invention adopts a new SVM-reinforcement learning model, which fully leverages the advantages of support vector machines in processing high-dimensional and small amounts of data, and also integrates the adaptability and exploratory nature of reinforcement learning, thereby improving the intelligent design level and intrusion detection accuracy of the ICS intrusion detection system. Attached Figure Description

[0054] Figure 1 is a schematic diagram of the principle of the ICS intrusion detection system and method that integrates reinforcement learning and feature selection optimization;

[0055] Figure 2 is a schematic diagram of binary encoding for feature selection optimization;

[0056] Figure 3 is a schematic diagram illustrating the principle of reinforcement learning methods;

[0057] Figure 4 is a schematic diagram of the cross operation;

[0058] Figure 5 is a schematic diagram of the mutation operation;

[0059] Figure 6 is a comparison of the accuracy, recall, and F1 score between the technology of the present invention and existing single Support Vector Machine (SVM) and single Reinforcement Learning (RL). Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Figure 1 shows a schematic diagram of the principle of an ICS intrusion detection system and method that integrates reinforcement learning and feature selection optimization, using a typical ICS system as an example.

[0062] Referring to Figure 1, the present invention integrates reinforcement learning and feature selection optimization in an ICS intrusion detection system. The system includes an Industrial Control Systems (ICS) intrusion detection data acquisition module, a feature selection offline optimization module based on Support Vector Machine (SVM)-reinforcement learning, and an online intrusion detection module based on optimal features using SVM-reinforcement learning.

[0063] In this embodiment, the ICS intrusion detection data acquisition module collects historical data on the production process monitoring status from the historical database of the industrial control system. After data parsing, the source dataset is obtained and labeled as X. The data is then normalized according to formula (1) to obtain the offline training dataset, which is also labeled as X. o X o Randomly partition the dataset proportionally to obtain the training set X. o1 With validation set X o2 , training set X o1 With validation set X o2 The input is fed into the SVM-reinforcement learning-based feature selection offline optimization module;

[0064]

[0065] Among them, X oj Represents the offline training dataset X o The feature of the j-th dimension, X j Let X represent the feature of the j-th dimension of the source dataset X. jmax With X jmin They represent X respectively j The minimum and maximum values ​​of X, where d represents X. o The maximum value of the dimension is d = 51 in this embodiment.

[0066] In this embodiment, the ICS intrusion detection data acquisition module collects real-time monitoring data of the production process from the ICS real-time database. After data parsing and normalization, an online detection dataset is obtained, which is then input into the SVM-reinforcement learning online intrusion detection module based on optimal features.

[0067] In this embodiment, the feature selection offline optimization module based on SVM-reinforcement learning performs binary encoding and population initialization on the data feature selection of the ICS historical dataset, performs offline training through support vector machine SVM-reinforcement learning, uses the accuracy obtained on the validation set as the fitness function, designs crossover and mutation operations based on cumulative probability to update the population, and obtains the optimal feature set after iterative optimization. The obtained optimal feature set is then input into the SVM-reinforcement learning online intrusion detection module based on the optimal features.

[0068] In this embodiment, the SVM-reinforcement learning online intrusion detection module based on optimal features selects features from the ICS real-time dataset based on the optimal feature set, and performs online intrusion detection testing on the real-time dataset through SVM-reinforcement learning to obtain intrusion detection performance indicators.

[0069] It is worth mentioning that this invention also provides an ICS intrusion detection method that integrates reinforcement learning and feature selection optimization. This method applies the ICS intrusion detection system described in the above embodiments and specifically includes the following steps:

[0070] (1) Set the parameter values ​​for the feature selection offline optimization module based on SVM-reinforcement learning. The parameter values ​​include population size N=10, number of iterations I=20, and crossover rate P. c =0.7, Variation rate P m =0.02, the training epochs of reinforcement learning are EP=300, the reinforcement learning actor network adopts a 3-layer network, namely the input layer, the hidden layer adopts Gaussian kernel function mapping, and the network loss function adopts the hinge loss function, which corresponds one-to-one with the Gaussian kernel function support vector machine. The learning rate lr1=0.001, the optimizer optimizer1=adaptive moment estimation (adam) optimizer, the critic network parameters include the number of network layers D=3, all three layers are fully connected layers, the optimizer optimizer2=adam, and the learning rate lr2=0.001.

[0071] It should be understood that other parameter values ​​can also be selected, and the specific settings can be made according to the actual situation.

[0072] (2) Take the randomly generated N = 10 binary-coded individuals as the initial population Q = {Ind} j For each individual in the sequence {j = 1, 2, ..., N}, the encoding is binary, and the encoding format is Ind. j =(a1,a2,…,a i ,…,a n), here Ind j a represents the j-th individual in the population. i This represents the encoding of each gene in an individual, 1≤i≤n, where n represents the maximum number of features in the dataset, and a i The specific definition formula (2) is shown below:

[0073]

[0074] Each Ind j The initialization is to randomly initialize n a's. i That is, a binary string of length n is randomly generated. The information of this binary string represents the selection combination of n features in the dataset. As shown in Figure 2, a randomly generated individual coding instance is displayed, where the feature selection information is: [3,4,6,12,16,17,21,22,23,28,29,31,33,34,36,37,41,44,45,46,47,50]. Then, the individual gene coding information obtained by the binary coding method described in the above formula (2) is: (001101000001000110001110000110101101100010011110010).

[0075] (3) The performance of population Q is evaluated. The specific evaluation process is as follows:

[0076] (3.1) Based on the binary encoding information corresponding to each individual in population Q, the offline training dataset X is... o1 After feature selection, the data is used as input for offline training of SVM-reinforcement learning.

[0077] (3.2) The implementation process of offline training of SVM-reinforcement learning is as follows:

[0078] Figure 3 illustrates the principle of reinforcement learning. The reinforcement learning environment class `env` is initialized, which has two attributes: the batch size (`batch_size`) and the state shape (`state_shape`). The `env` class also has two functions: a state update function (`upstate`) and an action function (`act`). The `upstate` function returns the state (`state`), which randomly selects a batch of data of size `batch_size` from the dataset and names it as `state`. The `act` function takes an action (`action`) and the state as inputs, and outputs the next state (labeled `state+1`) and a reward. The `act` function works by comparing the action's label with the state's label. If the action's label value is equal to the state's label value, the reward for that action is set to 1; otherwise, the reward is set to 0. Then, `upstate` is called again to update the state and obtain `state+1`, returning both the reward and `state+1`.

[0079] The actor network is initialized with a 3-layer network. The first layer is the input layer. The second layer is the hidden layer with a hidden_size of 128. It uses a Gaussian kernel function as the mapping and the network loss uses the hinge loss function, which corresponds one-to-one with the support vector machine (SVM) based on the Gaussian kernel function. The third layer is the output layer. The optimizer is optimizer1 and the learning rate is lr1.

[0080] The critic is initialized with a 3-layer network, all of which are fully connected layers. The second hidden layer has a network size of hindden_size = 128, and the optimizer is optimizer2 with a learning rate of lr2.

[0081] The env class calls the function upstate to get the state, and then selects a random number r = random(0,1), where r = random(0,1) means randomly selecting a decimal number between 0 and 1.

[0082] The specific calculation of action is shown in formula (3):

[0083]

[0084] Where ε is a hyperparameter used to determine the choice of action, n D Represents the offline training dataset X o1The total number of data points after feature selection, α h Indicates the action of the previous step, y h This indicates the data labels from the previous step, δ is a parameter of the Gaussian function, and c is a bias value. The state represents the total population, and s represents... A state in the context of a process.

[0085] Let A represent the state chosen randomly in the h-th iteration, and let A represent the total set of actions. Env executes the action function `act`, takes an action as input, and outputs a reward and the next state, called `state+1`. Then, the critic evaluates the state, reward, and...

[0086] state+1, and use formula (4) to obtain td_error to evaluate the reinforcement learning agent, thereby updating the reinforcement learning algorithm.

[0087] Actors:

[0088] td_error = reward + d × LAM × (V - -V)(4)

[0089] Where V represents the critics' evaluation of the state, V - The value represents the critic's evaluation of state+1. LAM is a hyperparameter representing the degree of connection between the state and state+1 data. d indicates whether to add the subsequent results. When d=0, the subsequent results are discarded. When d is a decimal between 0 and 1, the subsequent results are added according to the weights. Based on the obtained td_error, the reinforcement learning actor updates its own network parameter weights using action, state, and td_error.

[0090] (3.3) Perform offline training for the SVM-reinforcement learning EP rounds on the validation set X. o2 Perform verification tests to verify set X o2 Using the accuracy rate as the standard, the final accuracy rate for each individual is calculated and used as the individual fitness value f for each individual. j (j = 1, 2, ..., N), and determine the individual with the highest accuracy as the optimal individual Indi. best ;

[0091] (4) Calculate the probability p of each individual in population Q being inherited into the next generation population according to formula (5). j (j = 1, 2, ..., N), calculate the cumulative probability q of each individual according to formula (6). j :

[0092]

[0093]

[0094] (5) Generate a random number between 0 and 1, r1 = random(0,1). If r1 ≤ crossover rate P c If the condition is met, then a crossover operation is performed; otherwise, no crossover operation is performed. The specific steps for implementing the crossover operation are as follows: A random number r2 = random(0,1) between 0 and 1 is generated. If... Then select the j1-th individual from the population Q; randomly generate a random number between 0 and 1, r3 = random(0,1) and r3 ≠ r2, if Then, select the j2-th individual from the population Q, and then randomly generate an integer k = randint(1,n) between 1 and n, where randint(1,n) represents randomly generating an integer between 1 and n. Then, assign the code a corresponding to the k-th position of the j1-th and j2-th individuals to this code. k The values ​​are swapped, while other values ​​remain unchanged; a schematic diagram of the crossover operation is shown in Figure 4; the population after the crossover operation is labeled as Q. c ;

[0095] (6) From population Q c A random individual is selected from the sample, and a random number r4 = random(0,1) is generated. If r4 ≤ the mutation rate P, then the result is positive. m If the mutation operation is successful, then the individual is mutated; otherwise, no mutation operation is performed. The specific process of the mutation operation is as follows: Randomly generate an integer m = randint(1,n) between 1 and n, and assign the code a corresponding to the m-th position of the individual to be mutated to the variable. m Perform binary mutation, if the original a m If the value is 0, then the mutated a m If the original a becomes 1, m If the value is 1, then the mutated a m The value becomes 0, as shown in Figure 5; the mutated population is labeled as Q. m .

[0096] (7) Unconditionally accept Q = Q m .

[0097] (8) Repeat steps (3) to (7) until the set number of iterations I is met.

[0098] (9) Obtain the optimal individual Indi best To obtain Indi bestThe optimal feature set corresponding to the binary encoded sequence.

[0099] (10) Based on the optimal feature set, the ICS real-time dataset X r Perform feature selection to obtain a data subset X t .

[0100] (11) Apply SVM-reinforcement learning to the obtained data subset X according to step (3.2). t Online intrusion detection tests were conducted, alarms were triggered for detected abnormal data, and the accuracy, recall, and F1 score of the intrusion detection were calculated according to formulas (7) to (9):

[0101]

[0102]

[0103]

[0104] Wherein, TP represents the number of normal ICS data samples correctly predicted as normal categories, TN represents the number of abnormal ICS data samples correctly predicted as abnormal categories, FP represents the number of abnormal ICS data samples incorrectly predicted as normal categories, and FN represents the number of normal ICS data samples incorrectly predicted as abnormal categories.

[0105] By comparing and analyzing the experimental results of the above embodiments using the technology of the present invention with those of existing single SVM and single RL technologies, the comparison chart of the accuracy, recall and F1 score of the technology of the present invention with those of existing single SVM and single RL technologies is shown in Figure 6. It can be found that compared with traditional single support vector machine SVM and single reinforcement learning RL technologies, the ICS intrusion detection performance indicators such as accuracy, recall and F1 score obtained by the technology of the present invention are the best.

[0106] In this embodiment, the final performance metrics obtained are: accuracy = 98.90%, recall = 98.90%, and F1 score = 99.30%.

[0107] In summary, compared with existing technologies, this invention not only reduces redundancy in the ICS dataset through intelligent optimization of feature selection, thereby removing relevant noise and non-message data, reducing the space and time complexity of the algorithm, and alleviating the overfitting problem, but also adopts a new SVM-reinforcement learning model, which fully leverages the advantages of SVM in processing high-dimensional and small amounts of data, while also integrating the adaptability and exploratory nature of reinforcement learning, thus improving the intelligent design level and intrusion detection accuracy of the ICS intrusion detection system.

[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An ICS intrusion detection system that integrates reinforcement learning and feature selection optimization, characterized in that, The system includes an ICS intrusion detection data acquisition module, an SVM-reinforcement learning-based feature selection offline optimization module, and an optimal feature-based SVM-reinforcement learning-based online intrusion detection module. The ICS intrusion detection data acquisition module is used to collect historical data on production process monitoring status from the historical database of the industrial control system. After data parsing, it obtains the source dataset, labeled as X, and performs data normalization according to formula (1) to obtain an offline training dataset, which is then labeled as X. o X o Randomly partition the dataset proportionally to obtain the training set X. o1 With validation set X o2 The training set X o1 With validation set X o2 The input is fed into the SVM-reinforcement learning-based feature selection offline optimization module; (1) Where X oj Represents the offline training dataset X o The feature of the j-th dimension, X j Represents the feature of the j-th dimension of the source dataset X. and They represent X respectively j The minimum and maximum values ​​of X, where d represents X. o The maximum value of the dimension; the ICS intrusion detection data acquisition module is used to collect real-time monitoring data of the production process from the ICS real-time database. After data parsing and normalization, an online detection dataset is obtained. This online detection dataset is input into the SVM-reinforcement learning-based online intrusion detection module; the SVM-reinforcement learning-based feature selection offline optimization module is used to binary encode and initialize the data features of the ICS historical dataset, perform offline training through SVM-reinforcement learning, use the accuracy obtained on the validation set as the fitness function, update the population based on crossover and mutation operations based on cumulative probability, and after iterative optimization, from The optimal feature set is obtained and then input into the online intrusion detection module based on the optimal feature set using SVM-reinforcement learning. The online intrusion detection module based on the optimal feature set is used to perform feature selection on the ICS real-time dataset based on the optimal feature set, and to perform online intrusion detection testing on the real-time dataset using SVM-reinforcement learning to obtain intrusion detection performance indicators. The ICS intrusion detection method based on the fusion of reinforcement learning and feature selection optimization of the system includes the following steps: (1) setting the parameter values ​​of the feature selection offline optimization module based on SVM-reinforcement learning, wherein the parameter values ​​include population size N, iteration optimization number I, and crossover rate P. c Variation rate P m The training rounds EP of reinforcement learning, the network parameters of reinforcement learning actors and critics, the network parameters of reinforcement learning actors and critics include the number of network layers D, the learning rate lr1 of reinforcement learning actors, the learning rate lr2 of critics, the loss function loss, the optimizer optimizer1 of reinforcement learning actors, and the optimizer optimizer2 of critics; (2) take N randomly generated binary encoded individuals as the initial population Q={Ind j For each individual instance (j=1, 2, …, N), a binary encoding is used, with the encoding format being Ind. j =(a1,a2, …,a i , …,a n ), here Ind j a represents the j-th individual in the population. i This represents the encoding of each gene in an individual, 1≤i≤n, where n represents the maximum number of features in the dataset, and a i The specific definition formula (2) is expressed as follows: (2) Wherein, each Ind j The initialization is to randomly initialize n a's. i That is, a binary string of length n is randomly generated, and the information of the binary string represents the selection combination of n features in the dataset; (3) The performance of the population Q is evaluated, and the specific evaluation process is as follows: (3.1) Based on the binary encoding information corresponding to each individual in the population Q, the offline training dataset X is converted into a binary string of length n. o1 After feature selection, it is used as the data input for offline training of SVM-reinforcement learning; (3.2) The implementation process of offline training of SVM-reinforcement learning is as follows: Initialize the reinforcement learning environment env class, which has two attributes: the data group size batch_size and the data dimension size state_shape; the env class has two functions: the state update function upstate and the action function act; the return value of the state update function upstate is the state state, and the function of the state update function upstate is to randomly select a batch_size of data from the dataset and name it state and return it; the input parameters of the action function act are the action and state, and the output is the next state and the reward. The function of the action function act is to compare the label of action and state. If the action value and the label of state are different, the action function will be selected. If the el values ​​are equal, the reward for the action is 1; if they are not equal, the reward for the action is 0, and then upstate is called to update the state to get state+1, and the reward and state+1 are returned. The reinforcement learning actor network is initialized with a 3-layer network. The first layer is the input layer, the second hidden layer has a network size of hidden_size, uses the Gaussian kernel function as the mapping, and the loss function of the network uses the hinge loss function, which corresponds one-to-one with the SVM based on the Gaussian kernel function. The third layer is the output layer, the optimizer is optimizer1, and the learning rate is lr1. The critic is initialized with a 3-layer network. All 3 layers are fully connected layers. The second hidden layer has a network size of hidden_size, the optimizer is optimizer2, and the learning rate is lr2. The env class calls the state update function upstate to get the state, and then selects a random number r = random(0,1). random(0,1) means randomly selecting a decimal number between 0 and 1. The specific calculation of the action is shown in formula (3): (3) Where ε is a hyperparameter used to determine the choice of action, n D Represents the offline training dataset X o1 The total number of data after feature selection This indicates the action from the previous step. This indicates the data labels from the previous step, δ is a parameter of the Gaussian function, and c is a bias value. The state represents the total population, and s represents... A state in the context of a process. Let A represent the state randomly selected in the h-th iteration, and let A represent the overall set of actions. Env executes the action function act, takes action as input, and outputs reward and the next state, called state+1. Then, the critic evaluates the reinforcement learning actor based on state, reward, and state+1, according to formula (4), to obtain td_error, and updates the reinforcement learning actor accordingly. (4) Where V represents the critic's evaluation of the state, The evaluation value of state+1 is given by the critic. LAM is a hyperparameter representing the degree of connection between the state and state+1 data. d indicates whether to add the subsequent results. That is, when d=0, the subsequent results are discarded. When d is a decimal between 0 and 1, the subsequent results are added according to the weight. According to the obtained td_error, the reinforcement learning actor updates its own network parameter weights using action, state and td_error. (3.3) Perform offline training of SVM-reinforcement learning EP rounds for the validation set X. o2 Perform verification tests to verify set X o2 Using the accuracy rate as the standard, the final accuracy rate for each individual is calculated and used as the individual fitness value f for each individual. j j=1, 2, ..., N, and determine the individual with the highest accuracy as the optimal individual Indi best (4) Calculate the probability p of each individual in population Q being inherited into the next generation according to formula (5). j j=1,2,…,N, calculate the cumulative probability q of each individual according to formula (6). j : (5) (6) (5) Randomly generate a random number between 0 and 1, r1=random(0,1), if r1≤cross rate P c If the condition is met, then a crossover operation is performed; otherwise, no crossover operation is performed. The specific steps for implementing the crossover operation are as follows: A random number r2 = random(0,1) between 0 and 1 is generated. If... Then select the j1-th individual from the population Q; randomly generate a random number between 0 and 1 r3=random(0,1) and r3≠r2, if Then, select the j2th individual from the population Q, and then randomly generate an integer k = randint(1,n) between 1 and n, where randint(1,n) represents randomly generating an integer between 1 and n. Then, assign the code a corresponding to the k-th position of the j1st and j2nd individuals to each individual. k Perform a crossover operation, swapping values ​​while keeping other values ​​unchanged; label the resulting population as Q. c (6) From population Q c A random individual is selected from the sample, and a random number r4 = random(0,1) is generated. If r4 ≤ the mutation rate P, then the result is positive. m If the mutation operation is successful, then the individual is mutated; otherwise, no mutation operation is performed. The mutation operation specifically involves: randomly generating an integer m = randint(1, n) between 1 and n, and encoding the m-th position of the individual to be mutated using the given code a. m Perform binary mutation, if the original a m If the value is 0, then the mutated a m If the original a becomes 1, m If the value is 1, then the mutated a m Change to 0; mark the mutated population as Q. m (7) Unconditionally accept Q=Q m (8) Repeat steps (3) to (7) until the set number of iterations I is met; (9) Obtain the optimal individual Indi best To obtain Indi best The optimal feature set corresponding to the binary encoded sequence; (10) Based on the optimal feature set, the ICS real-time dataset X r Perform feature selection to obtain a subset of data X t (11) Apply SVM-reinforcement learning to the obtained data subset X according to step (3.2). t Conduct online intrusion detection tests, issue alarms for detected abnormal data, and calculate the accuracy, recall, and F1 score of intrusion detection according to formulas (7) to (9): (7) (8) (9) Wherein, TP represents the number of normal ICS data samples correctly predicted as normal classification, TN represents the number of abnormal ICS data samples correctly predicted as abnormal classification, FP represents the number of abnormal ICS data samples incorrectly predicted as normal classification, and FN represents the number of normal ICS data samples incorrectly predicted as abnormal classification.

Images