Risk IP identification method based on hybrid genetic ensemble learning

Abstract

The invention discloses a risk IP identification method based on hybrid genetic ensemble learning, and relates to the field of risk IP identification, and the method comprises the steps: obtaining unmarked risk IP data, marking the unmarked risk data through employing the data in historical information, and finally constructing a multi-source data warehouse; training by using a multi-source data warehouse to obtain a plurality of base classifiers for performing classification prediction on new IP data; screening each base classifier by adopting an improved genetic algorithm; and classifying new IP data by using an integrated model constructed by the retained base classifier. According to the method, crossover, variation and other operators in a traditional genetic algorithm are improved, simulated annealing and adaptive operation are introduced, and the speed and efficiency of the algorithm in the optimal solution solving process are improved. Based on a selective integration idea, a base classifier is selected and integrated by optimizing a weight factor in an optimal solution of an algorithm, a selective integration algorithm model is established, and the performance of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of risky IP identification, and in particular to a risky IP identification method and system based on hybrid genetic ensemble learning. Background Art

[0002] In the current network security environment, enterprises are facing increasingly severe threats of network attacks. Although the traditional external IP blocking method based on geographical characteristics can reduce the risk of attacks from specific areas to a certain extent, this method relies on manual judgment and operation, and has problems such as low efficiency and high misjudgment rate. With the continuous evolution and complexity of network attack methods, defense strategies that rely solely on geographical characteristics can no longer meet actual needs. Enterprises urgently need an automated and intelligent risk IP identification and defense solution to improve the efficiency and accuracy of network security defense.

[0003] At present, the technologies for risky IP identification mainly include blacklist and whitelist methods, rule matching methods, and machine learning methods: Blacklist and whitelist method: Maintain a blacklist to prevent known malicious IPs from accessing the network, and use a whitelist to allow trusted IPs to access. This method is simple and easy to implement, but it cannot identify new, unknown risky IPs, and the maintenance cost of the blacklist and whitelist is high.

[0004] Rule matching method: Use predefined rules to detect abnormal behaviors in network traffic, such as abnormal traffic patterns, abnormal port access, etc. This method requires a lot of prior knowledge, and the updating and maintenance of rules is also a tedious task, which is prone to missed reports and false positives.

[0005] Machine learning methods: In recent years, machine learning algorithms have been increasingly used in network security. By learning from a large amount of historical data, they can automatically identify potential risky IP addresses. However, when dealing with complex network environments, a single machine learning algorithm often has problems such as insufficient generalization ability and low accuracy.

[0006] The most similar implementation scheme to this technology is the risk IP identification method based on traditional genetic algorithms and ensemble learning. These methods have solved the limitations of traditional methods to a certain extent, but traditional genetic algorithms are prone to fall into local optimality when searching for the optimal solution, and the computational efficiency is low. In addition, the operator design of the genetic algorithm also affects the performance of the algorithm. Summary of the invention

[0007] In view of the above-mentioned problems, the present invention is proposed.

[0008] Therefore, the problem to be solved by the present invention is that the existing solutions are prone to omissions and false positives, insufficient generalization ability, and low accuracy when identifying risky IPs.

[0009] To solve the above technical problems, the present invention provides the following technical solutions: an improved genetic algorithm method for risk IP identification, which includes, in the step of selecting the crossover probability, selecting the corresponding crossover probability based on the fitness relationship; in the step of adjusting the mutation probability, adjusting the mutation probability based on the fitness relationship; in the key link of selecting excellent individuals, integrating the concept of simulated annealing to calculate the acceptance probability of new individuals.

[0010] As a preferred solution of the improved genetic algorithm method for risk IP identification described in the present invention, wherein: the crossover probability corresponding to the selection includes calculating the average fitness of the contemporary population , individual fitness When the individual fitness is less than the average fitness of the contemporary population, a fixed crossover probability is used. , when the individual fitness is greater than or equal to the average fitness of the contemporary population, Correction, dynamic crossover probability It is expressed as, , in, Expressed as the maximum fitness in the contemporary population.

[0011] As a preferred solution of the improved genetic algorithm method for risk IP identification described in the present invention, wherein: the adjustment of mutation probability includes, when the individual fitness is less than the average fitness of the contemporary population, using a fixed mutation coefficient , when the individual fitness is greater than or equal to the average fitness of the contemporary population, Correction, dynamic adjustment of mutation probability It is expressed as, .

[0012] As a preferred solution of the improved genetic algorithm method for risk IP identification described in the present invention, wherein: the calculation of the acceptance probability of a new individual includes setting the fitness threshold for the acceptance of the newly generated individual to ,when When the new individual is fully accepted, otherwise, the new individual has a certain probability of being accepted, and the acceptance probability It is expressed as, , in, It is expressed as the IP risk control parameter, that is, the thermodynamic temperature during simulated annealing. The calculation formula is as follows: , in, Expressed as IP risk variation coefficient, Expressed as initial risk.

[0013] In order to solve the above technical problems, the present invention provides the following technical solutions: a risk IP identification method based on improved hybrid genetic ensemble learning, which includes obtaining unlabeled risk IP data, labeling the unlabeled risk data using data in historical intelligence, and finally building a multi-source data warehouse; using the multi-source data warehouse for training to obtain several base classifiers for classifying and predicting new IP data; An improved genetic algorithm is used to screen each base classifier; an integrated model constructed using the retained base classifiers is used to classify new IP data.

[0014] As a preferred solution of the risk IP identification method based on improved hybrid genetic ensemble learning described in the present invention, the improved genetic algorithm includes: when performing the step of crossover probability selection, the corresponding crossover probability is selected based on the fitness relationship; when adjusting the mutation probability, the mutation probability is adjusted based on the fitness relationship; when selecting the key link of excellent individuals, the concept of simulated annealing is integrated to calculate the acceptance probability of new individuals; the fitness calculation method is to use the verification set to calculate the prediction error of the prediction results of each base classifier, and take the inverse of the prediction error as the fitness function, that is, the inverse of the error percentage in the entire verification set is expressed as the fitness function, , in, Expressed as prediction error percentage.

[0015] As a preferred solution of the risk IP identification method based on improved hybrid genetic ensemble learning described in the present invention, the screening of each base classifier includes using an improved genetic algorithm to obtain the optimal weight combination of the base classifier, screening according to the corresponding relationship between the weight and the base classifier, and the base classifier received with a weight of 1 is represented as significant and retained; the base classifier received with a weight of 0 is represented as insignificant and discarded.

[0016] Another object of the present invention is to provide a risk IP identification system based on improved hybrid genetic ensemble learning, which can learn new IP data and perform predictive classification on the new IP data.

[0017] To solve the above technical problems, the present invention provides the following technical solutions: a system for risk IP identification method based on improved hybrid genetic ensemble learning, comprising: a database establishment module, a base classifier construction model, a screening module and a prediction module; the database establishment module obtains unlabeled risk IP data, uses the data in the historical intelligence to mark the unlabeled risk data, and finally constructs a multi-source data warehouse; the base classifier construction model is trained using the multi-source data warehouse to obtain several base classifiers for classification and prediction of new IP data; the screening module uses an improved genetic algorithm to screen each base classifier; the prediction module uses the integrated model constructed by the retained base classifiers to classify the new IP data.

[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the risk IP identification method based on improved hybrid genetic ensemble learning are implemented as described above.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the risk IP identification method based on improved hybrid genetic ensemble learning as described above.

[0020] The beneficial effects of the present invention are as follows: the present invention adopts a binary encoding method, which reduces the space for searching for the optimal solution and improves the search efficiency of the genetic algorithm.

[0021] The crossover and mutation operators in the traditional genetic algorithm were improved, and simulated annealing and adaptive operations were introduced to further improve the speed and efficiency of the algorithm in the process of finding the optimal solution.

[0022] Based on the idea of ​​selective integration, by optimizing the weight factors in the optimal solution of the algorithm, selecting and integrating base classifiers, a selective integration algorithm model was established to enhance the performance of the overall model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 This is a flow chart of the risk IP identification method based on improved hybrid genetic ensemble learning in Example 1.

[0024] Figure 2 This is an algorithm flow chart of an improved genetic algorithm method for risky IP identification in Example 1. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and which provides a risk IP identification method based on improved hybrid genetic ensemble learning, including: Figure 1 As shown: Step 1: Obtain unlabeled risk IP data, use data from historical intelligence to label the unlabeled risk data, and finally build a multi-source data warehouse.

[0028] This invention innovatively adopts the idea of ​​selective integration. The algorithm regards the identification of risky IPs as a binary classification problem, obtains unlabeled risky IP data from the log collection of network devices, such as switches, firewalls, intrusion prevention systems, etc., and obtains labeled risky IP data from enterprise information notifications and historical security intelligence. Then, the unlabeled risk data is labeled using the data in the historical intelligence, and finally a multi-source data warehouse is constructed.

[0029] Step 2: Use the multi-source data warehouse for training to obtain several base classifiers for classification prediction of new IP data.

[0030] The algorithm of the present invention uses a gradient boosted decision tree (GBDT) as a base classifier. Based on the boosting idea in ensemble learning, GBDT is obtained by iterating the forward distributed algorithm of the CART decision tree. In each iteration, a new decision tree is generated in the gradient direction where the residual decreases. In addition, GBDT greatly integrates the concepts of regression trees and boosted trees, showing strong generalization and classification capabilities. Ultimately, the GBDT model that integrates multiple decision trees is able to classify and predict new IP data. The model will identify IP addresses that are significantly different from normal behavior patterns and mark them as abnormal or risky IPs. As a basic classifier for ensemble learning, GBDT performs better than other weak classifiers in improving the convergence speed and accuracy of the model.

[0031] Step 3: Use the improved genetic algorithm to screen each base classifier.

[0032] like Figure 2 As shown, the role of the improved genetic algorithm of the present invention in abnormal IP detection is achieved by optimizing the weights of each base classifier in selective ensemble learning. Its output is the optimal weight factor in selective ensemble learning, which determines which basic classifiers (in this method, gradient boosted decision tree GBDT) will be integrated and which will be discarded. The basic idea of ​​hybrid genetics is: first, the algorithm randomly generates a binary coding combination as the initial population. Then, all base classifiers are weighted according to the initial population. Through the validation set, the accuracy of the selective ensemble classification under these weight factors is calculated. Based on the calculation accuracy, the fitness is calculated by the fitness calculation formula. The algorithm will be checked if it has reached the generation cutoff. If the cutoff point is not reached, a bionic adaptive crossover is performed based on a certain probability to generate a new population. In addition, some individuals will undergo bionic mutations according to the probability of adaptive mutation to generate an updated population. Finally, excellent individuals are selected by imitating natural selection through the simulated annealing algorithm. After the selection, the selected population will again calculate its fitness, determine its cutoff point, and iterate its genes until it meets the cutoff condition and is able to produce the best individuals, which in our case is the optimal weight distribution of the base classifier.

[0033] The prediction error of each base classifier prediction result is calculated using the validation set, and the inverse of the prediction error is used as the fitness function, that is, the inverse of the error percentage in the entire validation set is expressed as the fitness function: , in, Expressed as prediction error percentage.

[0034] From the above formula, we can see that the smaller the prediction error or the higher the classification accuracy, the more suitable the individual gene is. Therefore, this more suitable individual gene is more likely to be retained in the next generation. The core of the adaptive crossover operator is the selection of crossover probability. When the average fitness of the population is Greater than individual fitness When , a fixed and large crossover probability is used. On the contrary, when the overall average fitness is less than or equal to the individual fitness, the crossover probability is calculated by the function. In this way, not only the diversity of the population can be guaranteed, but also the excellent population can be retained. Dynamic crossover probability of adaptive crossover operator The expression of is, , in, It is expressed as the maximum fitness in the contemporary population. From the above formula, we can see that when the individual fitness is less than the average fitness of the contemporary population When a fixed crossover probability is used . In the range of [0.5, 0.9]. When the individual fitness is greater than or equal to the average fitness When , the corresponding crossover probability Less than fixed This indicates that the crossover probability of excellent individual genes is low, and as individual fitness increases As the individual fitness approaches When , the crossover probability approaches 0, so that excellent genes can be retained.

[0035] The adaptive mutation operator adjusts the mutation probability according to the fitness of the population, which is conducive to the inheritance of excellent genes. In the early stage of evolution, when the overall fitness of the population is low, the operator uses a larger mutation probability, so that more diverse individuals can be produced, creating a larger space for finding the optimal solution. With the later stage of evolution, when the overall fitness of the population is high, the operator uses a relatively small mutation probability to retain excellent individuals, further promoting convergence to find the global optimum. The corresponding formula is as follows: , in is the mutation coefficient, ranging from 0 to 0.05. Just as in nature, the mutation position of genes in chromosomes is randomly selected, the operator also uses random numbers to select the mutation position of individuals. When a gene position is selected, the algorithm causes the corresponding gene in the binary code to mutate, that is, converting the number from 1 to 0 and vice versa.

[0036] As a key step in selecting excellent individuals, the selection operator incorporates the concept of simulated annealing. The core of simulated annealing is the Metropolis acceptance criterion. The basic concept of the operator is: Assume that the fitness of the newly generated individual is , the threshold is .when When , the new individual is completely accepted. Otherwise, the new individual has a certain probability of being accepted. The acceptance probability is as follows: , In the formula is the IP risk control parameter, i.e. the thermodynamic temperature during simulated annealing. The calculation formula is as follows: , in, is the IP risk variation coefficient, is the initial risk. The selected parameters are is 1000, and k is 5.0.

[0037] The optimal weight combination of base classifiers is obtained through the above hybrid genetic algorithm.

[0038] According to the corresponding relationship between the weight and the base classifier, the base classifier receiving the weight of 1 is indicated as significant and retained; the base classifier receiving the weight of 0 is indicated as insignificant and discarded. The basic classifier of this embodiment is a gradient enhanced decision tree. The final classification model is a set of basic classifiers obtained by the Majority Vote algorithm of bagging.

[0039] Step 4: Use the ensemble model built using the retained base classifiers to classify new IP data. The model will identify abnormal or risky IPs based on the learned patterns.

[0040] Example 2 is the second example of the present invention, which is different from the first example in that the risk IP identification method based on improved hybrid genetic ensemble learning also includes: in order to verify and illustrate the technical effects adopted in this method, this example adopts traditional technical solutions and the method of the present invention for comparative testing, and compares the test results by means of scientific argumentation to verify the real effect of this method.

[0041] Because traditional methods such as blacklist and whitelist methods and rule matching methods rely on known risk lists, they have limited detection capabilities for newly emerging malicious IPs or those that are not on the list, so this embodiment is only compared with other machine learning methods.

[0042] The improved hybrid genetic ensemble learning risk IP identification method of the present invention, together with the traditional IP risk profiling, gradient boosting decision tree and graph-based machine learning method, were loaded into the experimental model for simulation. The accuracy of risk IP identification was used as the evaluation criterion. The experimental data obtained are shown in Table 1.

[0043] Table 1: Experimental data comparison table , It can be seen from the above experimental data that the method of the present invention is better in terms of the accuracy of risky IP identification than the traditional machine learning method. This is due to the fact that the present invention improves the crossover, mutation and other operators in the traditional genetic algorithm, introduces simulated annealing and adaptive operations, and improves the speed and efficiency of the algorithm in the process of solving the optimal solution.

[0044] Embodiment 3 is the third embodiment of the present invention, which is different from the previous two embodiments in that: a system for risk IP identification method based on improved hybrid genetic ensemble learning includes a database establishment module, a base classifier construction model, a screening module and a prediction module; the database establishment module obtains unlabeled risk IP data, uses the data in historical intelligence to mark the unlabeled risk data, and finally builds a multi-source data warehouse; the base classifier construction model is trained using the multi-source data warehouse to obtain several base classifiers for classification and prediction of new IP data; the screening module uses an improved genetic algorithm to screen each base classifier; the prediction module uses the integrated model constructed by the retained base classifiers to classify the new IP data.

[0045] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0046] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0047] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0048] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0049] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An improved genetic algorithm method for risky IP identification, characterized by: include, When performing the crossover probability selection step, the corresponding crossover probability is selected based on the fitness relationship; In the step of adjusting the mutation probability, the mutation probability is adjusted based on the fitness relationship; In the key step of selecting excellent individuals, the concept of simulated annealing is integrated to calculate the acceptance probability of new individuals.

2. The improved genetic algorithm method for risky IP identification according to claim 1, characterized in that: The crossover probability corresponding to the selection includes calculating the average fitness of the contemporary population , individual fitness When the individual fitness is less than the average fitness of the contemporary population, a fixed crossover probability is used. , when the individual fitness is greater than or equal to the average fitness of the contemporary population, Correction, dynamic crossover probability It is expressed as, ， in, Expressed as the maximum fitness in the contemporary population.

3. The improved genetic algorithm method for risky IP identification according to claim 2, characterized in that: The adjustment of mutation probability includes adopting a fixed mutation coefficient when the individual fitness is less than the average fitness of the contemporary population. , when the individual fitness is greater than or equal to the average fitness of the contemporary population, Correction, dynamic adjustment of mutation probability It is expressed as, 。 4. The improved genetic algorithm method for risky IP identification according to claim 3, characterized in that: The calculation of the acceptance probability of the new individual includes setting the fitness threshold of the newly generated individual to be accepted as ,when When the new individual is fully accepted, otherwise, the new individual has a certain probability of being accepted, and the acceptance probability It is expressed as, ， in, It is expressed as the IP risk control parameter, that is, the thermodynamic temperature during simulated annealing. The calculation formula is as follows: ， in, Expressed as IP risk variation coefficient, Expressed as initial risk.

5. The risk IP identification method based on improved hybrid genetic ensemble learning is characterized by: include, Obtain unlabeled risk IP data, use data from historical intelligence to label the unlabeled risk data, and ultimately build a multi-source data warehouse; Using multi-source data warehouse for training, several base classifiers are obtained for classifying and predicting new IP data; An improved genetic algorithm is used to screen each base classifier; The ensemble model constructed using the retained base classifiers is used to classify new IP data.

6. The risk IP identification method based on improved hybrid genetic ensemble learning as claimed in claim 5, characterized in that: The improved genetic algorithm includes: when performing the step of selecting the crossover probability, selecting the corresponding crossover probability based on the fitness relationship; when adjusting the mutation probability, adjusting the mutation probability based on the fitness relationship; when selecting the key link of excellent individuals, integrating the concept of simulated annealing to calculate the acceptance probability of new individuals; The fitness calculation method is to use the validation set to calculate the prediction error of each base classifier prediction result, and take the inverse of the prediction error as the fitness function, that is, the inverse of the error percentage in the entire validation set is expressed as the fitness function: ， in, Expressed as prediction error percentage.

7. The risk IP identification method based on improved hybrid genetic ensemble learning as claimed in claim 6, characterized in that: The screening of each base classifier includes using an improved genetic algorithm to obtain the optimal weight combination of the base classifier, screening according to the corresponding relationship between the weight and the base classifier, and receiving a base classifier with a weight of 1 as significant and retained; receiving a base classifier with a weight of 0 as insignificant and discarded.

8. A system using the risk IP identification method based on improved hybrid genetic ensemble learning as claimed in any one of claims 5 to 7, characterized in that: It includes a database building module, a base classifier building model, a screening module and a prediction module; The database building module obtains unlabeled risk IP data, labels the unlabeled risk data using data from historical intelligence, and ultimately builds a multi-source data warehouse; The base classifier construction model is trained using a multi-source data warehouse to obtain a number of base classifiers for classifying and predicting new IP data; The screening module uses an improved genetic algorithm to screen each base classifier; The prediction module classifies new IP data using an integrated model constructed using the retained base classifiers.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the risk IP identification method based on improved hybrid genetic ensemble learning described in any one of claims 5 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the risk IP identification method based on improved hybrid genetic ensemble learning described in any one of claims 5 to 7 are implemented.

Images