Abnormal host detection method, system and device based on AE algorithm and medium

Abstract

The invention discloses an abnormal host detection method, system and device based on an AE algorithm and a medium, mainly relates to the technical field of abnormal host detection, and is used for solving the problems that an existing recognition scheme cannot effectively solve violation leakage events caused in organizations and cannot respond in time in a short time before or after the events occur. Comprising the steps of processing host data of a corresponding type into a preset uniform format based on a preset data uniform program, and generating a host data set; selecting a preset number of to-be-processed host features with the highest correlation with the preset abnormal target features from the host data set; screening out multiple collinearity features from the to-be-processed host features to generate a to-be-processed host set; obtaining an annotation result of the to-be-processed host set, taking the annotated to-be-processed host set as sample data to train an AE algorithm, and obtaining a trained AE algorithm; and inputting the current to-be-processed host set into the trained AE algorithm to obtain an output anomaly detection result.

Description

Technical Field

[0001] The present application relates to the technical field of abnormal host detection, and in particular to an abnormal host detection method, system, device and medium based on an AE algorithm. Background Art

[0002] With the development of network technology, data security and information security of network data are becoming more and more prominent. At present, there are mature solutions for various types of external network attacks, but violations and leaks caused by internal organizations are endless, and there is no effective means to detect them. Cyber ​​criminals may disguise themselves as legitimate users, break through network boundaries, steal network credentials, implant malware, or cause network security threats within the organization due to mistakes made by internal personnel.

[0003] Abnormal host identification is one of the effective means to deal with new threats, but the current traditional gateway-based security architecture: real-time monitoring of network traffic through gateway devices, analysis of data packets such as source, destination, content and other characteristics. Use traffic analysis technology, such as deep packet inspection (DPI) and traffic behavior analysis (NBA), to identify abnormal traffic patterns or suspicious behaviors.

[0004] However, the above traditional identification solutions cannot effectively solve the illegal leakage incidents caused by the organization, and cannot respond in time before or after the incident, resulting in the occurrence of information leakage security incidents. Therefore, there is an urgent need for an abnormal host detection method, system, device and medium based on the AE (AutoEncoder) algorithm to solve the above technical problems. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present application provides an abnormal host detection method, system, device and medium based on the AE algorithm to solve the problem that the existing identification scheme cannot effectively solve the illegal leakage incidents caused within the organization and cannot respond in time in a short time before or after the incident occurs.

[0006] In a first aspect, the present application provides an abnormal host detection method based on an AE algorithm, the method comprising: The host data of a preset historical time period is obtained; wherein the host data includes at least: host local operation and network behavior data; the preset data unified program corresponding to each type of host data is called, and the host data of the corresponding type is processed into a preset unified format based on the preset data unified program to generate a host data set; a preset number of host features to be processed with the highest correlation with preset abnormal target features are selected from the host data set; multicollinear features are screened out from the host features to be processed to generate a host set to be processed; the labeling result of the host set to be processed is obtained, and the labeled host set to be processed is used as sample data to train the AE algorithm to obtain a trained AE algorithm; the current host data is obtained in real time to generate the current host set to be processed; the current host set to be processed is input into the trained AE algorithm to obtain the output abnormality detection result.

[0007] The abnormal host detection method provided in the embodiment of the present application selects the host features to be processed that are most highly correlated with the preset abnormal target features from the host data set. This step ensures that the features used to train the AE algorithm are highly relevant to the anomaly detection task, thereby improving the efficiency and accuracy of the algorithm. Screening out multicollinear features avoids redundancy and interference between features, allowing the AE algorithm to focus more on learning key features and improving the generalization ability of the model. By training the AE algorithm, the pattern of abnormal host behavior can be learned, and after obtaining the current host data in real time, the anomaly detection results are output. In addition, the present application generates the current set of hosts to be processed by obtaining the current host data in real time, thereby reducing the amount of data actually processed by the algorithm, helping to respond in a timely manner in a short time before or after the event occurs, and reducing potential losses.

[0008] In one implementation of the present application, a preset number of host features to be processed that have the highest correlation with preset abnormal target features are selected from the host data set, specifically including: Obtaining preset abnormal target parameters; extracting target feature data corresponding to the preset abnormal target parameters from the host data set; Numerical processing is performed on the target feature data and the preset abnormal target features to obtain numerical target feature data and the preset abnormal target features; The numerical target feature data and the preset abnormal target feature are input into the chi-square verification algorithm to obtain a preset number of host features to be processed that have the highest correlation with the preset abnormal target feature.

[0009] In one implementation of the present application, multicollinear features are screened out from the host features to be processed to generate a host set to be processed, which specifically includes: According to the similarity between the current host feature to be processed and other host features to be processed, any one of the host features to be processed whose similarity is greater than a preset similarity threshold is deleted, until the similarity between any two host features to be processed in the remaining host features to be processed is less than or equal to the preset similarity threshold; The remaining host characteristics to be processed are stored as a host set to be processed.

[0010] In one implementation of the present application, current host data is acquired in real time to generate a current host set to be processed, specifically including: The current host data is obtained in real time, and the current host data is imported into the corresponding preset data unified program to obtain the current host data set, obtain the current host features to be processed, perform multicollinearity feature screening, and generate the current host set to be processed.

[0011] In a second aspect, the present application provides an abnormal host detection system based on the AE algorithm, the system comprising: An acquisition module is used to acquire host data of a preset historical time period; wherein the host data includes at least: host local operation and network behavior data; a generation module is used to call a preset data unified program corresponding to each type of host data, and based on the preset data unified program, process the corresponding type of host data into a preset unified format to generate a host data set; select a preset number of host features to be processed that have the highest correlation with preset abnormal target features from the host data set; screen out multicollinear features from the host features to be processed to generate a host set to be processed; an acquisition module is used to acquire the labeling results of the host set to be processed, and use the labeled host set to be processed as sample data to train the AE algorithm to obtain a trained AE algorithm; acquire the current host data in real time to generate the current host set to be processed; input the current host set to be processed into the trained AE algorithm to obtain the output abnormality detection result.

[0012] In one implementation of the present application, the generating module includes an obtaining unit, Used to obtain preset abnormal target parameters; extract target feature data corresponding to the preset abnormal target parameters from the host data set; Numerical processing is performed on the target feature data and the preset abnormal target features to obtain numerical target feature data and the preset abnormal target features; The numerical target feature data and the preset abnormal target feature are input into the chi-square verification algorithm to obtain a preset number of host features to be processed that have the highest correlation with the preset abnormal target feature.

[0013] In one implementation of the present application, the generation module includes a generation unit, for deleting any one of a pair of host features to be processed whose similarity is greater than a preset similarity threshold according to the similarity between the current host feature to be processed and other host features to be processed, until the similarity between any two host features to be processed in the remaining host features to be processed is less than or equal to the preset similarity threshold; The remaining host characteristics to be processed are stored as a host set to be processed.

[0014] In one implementation of the present application, the acquisition module includes a real-time data processing unit, It is used to obtain the current host data in real time, import the current host data into the corresponding preset data unified program, obtain the current host data set, obtain the current host features to be processed, perform multicollinearity feature screening, and generate the current host set to be processed.

[0015] In a third aspect, the present application provides an abnormal host detection device based on the AE algorithm, the device comprising: processor; and a memory having executable codes stored thereon, which, when executed, causes the processor to execute any of the above-mentioned abnormal host detection methods based on the AE algorithm.

[0016] In a fourth aspect, the present application provides a non-volatile computer storage medium having computer instructions stored thereon, which when executed implement an abnormal host detection method based on the AE algorithm as described above.

[0017] Those skilled in the art can understand that the present application has at least the following beneficial effects: The present application provides an abnormal host detection method, system, device and medium based on the AE algorithm, which can select the host features to be processed that are most correlated with the preset abnormal target features from the host data set, ensuring that the features used to train the AE algorithm are highly relevant to the anomaly detection task, thereby improving the efficiency and accuracy of the algorithm. Screening out multicollinear features avoids redundancy and interference between features, allowing the AE algorithm to focus more on learning key features and improving the generalization ability of the model. By training the AE algorithm, the pattern of abnormal host behavior can be learned, and after obtaining the current host data in real time, the anomaly detection results can be output. In addition, the present application generates the current host set to be processed by obtaining the current host data in real time, which reduces the amount of data actually processed by the algorithm, helps to respond in a timely manner in a short time before or after the event occurs, and reduces potential losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Some embodiments of the present disclosure are described below with reference to the accompanying drawings, in which: Figure 1This is a flow chart of an abnormal host detection method based on the AE algorithm provided in an embodiment of the present application.

[0019] Figure 2 It is a schematic diagram of the internal structure of an abnormal host detection system based on the AE algorithm provided in an embodiment of the present application.

[0020] Figure 3 It is a schematic diagram of the internal structure of an abnormal host detection device based on the AE algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be understood by those skilled in the art that the embodiments described below are only preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented through the preferred embodiments. The preferred embodiments are only used to explain the technical principles of the present disclosure, and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present disclosure.

[0022] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0023] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0024] The embodiment provides an abnormal host detection method based on the AE algorithm, such as Figure 1 As shown, the method provided in the embodiment of the present application mainly includes the following steps: Step 110: Obtain host data for a preset historical time period.

[0025] It should be noted that the host data includes at least: host local operations and network behavior data. First, the host operations (host local operations) and network activities (network behavior data) in the organization's internal network environment must be simulated and abstracted into host entities and behavior data. The entity is the identity information used to uniquely identify a host in a network environment. In different contexts, the unique identifier of the host may have different forms. The main ways to uniquely identify the host are: MAC address, IP address, hardware ID, etc. This application uses IP as the host entity identifier. Analysis is carried out around the host IP. Network behavior data is the local operation of the entity and the behavior action log in the network. Network behavior data relies on probe collection. Through the information integration of the host local operation and network behavior data, the user behavior is truly restored and a comprehensive analysis is performed.

[0026] The local operation actions of the host can be collected by the host security product terminal, and the network behavior data depends on the network probe collection. Various data are obtained through syslog, message middleware, HTTP and other channels in an active or passive way.

[0027] Step 120 : calling a preset data unification program corresponding to each type of host data, and processing the corresponding type of host data into a preset unified format based on the preset data unification program to generate a host data set.

[0028] It should be noted that the collected data often has the characteristics of multi-source heterogeneity, large data volume, and different statistical dimensions. Therefore, this application presets a unified program corresponding to each type of host data. It should be supplemented that the unified program can specifically process the data in the current format into data in a preset format, etc. The specific content can be determined by those skilled in the art according to actual needs.

[0029] There is a preset program upload interface, and those skilled in the art can replace the preset data unified program corresponding to each type of host data according to actual needs.

[0030] Step 130: Select a preset number of host features to be processed that have the highest correlation with preset abnormal target features from the host data set; filter out multicollinear features from the host features to be processed to generate a host set to be processed.

[0031] Among them, a preset number of host features to be processed with the highest correlation with the preset abnormal target features are selected from the host data set, which can be specifically: Obtain preset abnormal target parameters; extract target feature data corresponding to the preset abnormal target parameters from the host data set; perform numerical processing on the target feature data and the preset abnormal target features to obtain numerical target feature data and the preset abnormal target features; input the numerical target feature data and the preset abnormal target features into the chi-square verification algorithm to obtain a preset number of host features to be processed that are most highly correlated with the preset abnormal target features.

[0032] It should be noted that the process of obtaining the preset abnormal target parameters can be manually input.

[0033] Among them, multicollinear features are screened out from the host features to be processed to generate a host set to be processed, which can be specifically: According to the similarity between the current host feature to be processed and other host features to be processed, any one of the host features to be processed whose similarity is greater than a preset similarity threshold is deleted until the similarity between any two host features to be processed in the remaining host features to be processed is less than or equal to the preset similarity threshold; the remaining host features to be processed are stored as a host set to be processed.

[0034] It should be noted that the solution of calculating the similarity between the current host feature to be processed and the features of other hosts to be processed can be achieved by calculating the variance inflation factor.

[0035] Step 140: Obtain the labeling results of the host set to be processed, use the labeled host set to be processed as sample data to train the AE algorithm, and obtain a trained AE algorithm.

[0036] Step 150: Acquire current host data in real time to generate a current host set to be processed; input the current host set to be processed into the trained AE algorithm to obtain an output anomaly detection result.

[0037] The current host data is obtained in real time to generate the current host set to be processed, which can be specifically: The current host data is obtained in real time, and the current host data is imported into the corresponding preset data unified program to obtain the current host data set, obtain the current host features to be processed, perform multicollinearity feature screening, and generate the current host set to be processed.

[0038] In addition, this application Figure 2 An abnormal host detection system based on AE algorithm is provided in the embodiment of the present application. Figure 2 As shown, the system provided in the embodiment of the present application mainly includes: The acquisition module 210 is used to acquire the host data of a preset historical time period; wherein the host data at least includes: host local operation and network behavior data.

[0039] The generation module 220 is used to call the preset data unification program corresponding to each type of host data, process the corresponding type of host data into a preset unified format based on the preset data unification program, and generate a host data set; select a preset number of host features to be processed that have the highest correlation with the preset abnormal target features from the host data set; screen out multicollinear features from the host features to be processed, and generate a host set to be processed.

[0040] The generation module 220 includes an acquisition unit, which is used to obtain preset abnormal target parameters; extract target feature data corresponding to the preset abnormal target parameters from the host data set; perform numerical processing on the target feature data and the preset abnormal target features to obtain numerical target feature data and preset abnormal target features; input the numerical target feature data and the preset abnormal target features into the chi-square verification algorithm to obtain a preset number of host features to be processed that have the highest correlation with the preset abnormal target features.

[0041] The generation module 220 includes a generation unit, which is used to delete any one of a pair of host features to be processed whose similarity is greater than a preset similarity threshold according to the similarity between the current host feature to be processed and other host features to be processed, until the similarity between any two of the remaining host features to be processed is less than or equal to the preset similarity threshold; and store the remaining host features to be processed as a host set to be processed.

[0042] The acquisition module 230 is used to obtain the labeling results of the host set to be processed, use the labeled host set to be processed as sample data to train the AE algorithm, and obtain the trained AE algorithm; obtain the current host data in real time to generate the current host set to be processed; input the current host set to be processed into the trained AE algorithm to obtain the output anomaly detection result.

[0043] The acquisition module 230 includes a real-time data processing unit, which is used to obtain the current host data in real time, import the current host data into the corresponding preset data unified program, obtain the current host data set, obtain the current host features to be processed, perform multicollinearity feature screening, and generate the current host set to be processed.

[0044] The above is a method embodiment of the present application. Based on the same inventive concept, the present application embodiment also provides an abnormal host detection device based on the AE algorithm. Figure 3 As shown, the device includes: a processor; and a memory, on which executable codes are stored. When the executable codes are executed, the processor executes an abnormal host detection method based on the AE algorithm as in the above-mentioned embodiment.

[0045] Specifically, the server side obtains host data of a preset historical time period; wherein the host data includes at least: host local operation and network behavior data; calls a preset data unified program corresponding to each type of host data, processes the corresponding type of host data into a preset unified format based on the preset data unified program, and generates a host data set; selects a preset number of host features to be processed that have the highest correlation with preset abnormal target features from the host data set; screens out multicollinear features from the host features to be processed, and generates a host set to be processed; obtains the annotation results of the host set to be processed, uses the annotated host set to be processed as sample data to train the AE algorithm, and obtains a trained AE algorithm; obtains the current host data in real time, and generates the current host set to be processed; inputs the current host set to be processed into the trained AE algorithm, and obtains the output anomaly detection result.

[0046] In addition, an embodiment of the present application further provides a non-volatile computer storage medium on which executable instructions are stored. When the executable instructions are executed, an abnormal host detection method based on the AE algorithm as described above is implemented.

[0047] So far, the technical solutions of the present disclosure have been described in combination with the above multiple embodiments, but it is easy for those skilled in the art to understand that the protection scope of the present disclosure is not limited to these specific embodiments. Without departing from the technical principles of the present disclosure, those skilled in the art can split and combine the technical solutions in the above-mentioned various embodiments, and can also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concept and / or technical principle of the present disclosure will fall within the protection scope of the present disclosure.

Claims

1. An abnormal host detection method based on AE algorithm, characterized in that: The method comprises: Obtaining host data for a preset historical time period; wherein the host data at least includes: host local operation and network behavior data; Calling a preset data unification program corresponding to each type of host data, processing the corresponding type of host data into a preset unified format based on the preset data unification program, and generating a host data set; Select a preset number of host features to be processed that have the highest correlation with the preset abnormal target features from the host data set; filter out multicollinear features from the host features to be processed to generate a host set to be processed; Obtain the labeling results of the host set to be processed, use the labeled host set to be processed as sample data to train the AE algorithm, and obtain a trained AE algorithm; The current host data is obtained in real time to generate the current host set to be processed; the current host set to be processed is input into the trained AE algorithm to obtain the output anomaly detection result.

2. The abnormal host detection method based on AE algorithm according to claim 1 is characterized in that: Select a preset number of host features to be processed that have the highest correlation with the preset abnormal target features from the host data set, including: Obtaining preset abnormal target parameters; extracting target feature data corresponding to the preset abnormal target parameters from the host data set; Numerical processing is performed on the target feature data and the preset abnormal target features to obtain numerical target feature data and the preset abnormal target features; The numerical target feature data and the preset abnormal target feature are input into the chi-square verification algorithm to obtain a preset number of host features to be processed that have the highest correlation with the preset abnormal target feature.

3. The abnormal host detection method based on AE algorithm according to claim 1 is characterized in that: Multicollinear features are screened out from the host features to be processed to generate a host set to be processed, including: According to the similarity between the current host feature to be processed and other host features to be processed, any one of the host features to be processed whose similarity is greater than a preset similarity threshold is deleted, until the similarity between any two host features to be processed in the remaining host features to be processed is less than or equal to the preset similarity threshold; The remaining host characteristics to be processed are stored as a host set to be processed.

4. The abnormal host detection method based on AE algorithm according to claim 1 is characterized in that: Obtain current host data in real time and generate the current host set to be processed, including: The current host data is obtained in real time, and the current host data is imported into the corresponding preset data unified program to obtain the current host data set, obtain the current host features to be processed, perform multicollinearity feature screening, and generate the current host set to be processed.

5. An abnormal host detection system based on AE algorithm, characterized in that: The system comprises: An acquisition module is used to acquire host data of a preset historical time period; wherein the host data at least includes: host local operation and network behavior data; A generation module is used to call a preset data unified program corresponding to each type of host data, process the corresponding type of host data into a preset unified format based on the preset data unified program, and generate a host data set; select a preset number of host features to be processed that have the highest correlation with the preset abnormal target features from the host data set; screen out multicollinear features from the host features to be processed, and generate a host set to be processed; The acquisition module is used to obtain the labeling results of the host set to be processed, use the labeled host set to be processed as sample data to train the AE algorithm, and obtain the trained AE algorithm; obtain the current host data in real time to generate the current host set to be processed; input the current host set to be processed into the trained AE algorithm to obtain the output anomaly detection result.

6. The abnormal host detection system based on AE algorithm according to claim 5 is characterized in that: The generation module includes obtaining units, Used to obtain preset abnormal target parameters; extract target feature data corresponding to the preset abnormal target parameters from the host data set; Numerical processing is performed on the target feature data and the preset abnormal target features to obtain numerical target feature data and the preset abnormal target features; The numerical target feature data and the preset abnormal target feature are input into the chi-square verification algorithm to obtain a preset number of host features to be processed that have the highest correlation with the preset abnormal target feature.

7. The abnormal host detection system based on AE algorithm according to claim 5 is characterized in that: The generation module includes a generation unit, for deleting any one of a pair of host features to be processed whose similarity is greater than a preset similarity threshold according to the similarity between the current host feature to be processed and other host features to be processed, until the similarity between any two host features to be processed in the remaining host features to be processed is less than or equal to the preset similarity threshold; The remaining host characteristics to be processed are stored as a host set to be processed.

8. The abnormal host detection system based on AE algorithm according to claim 5 is characterized in that: The acquisition module includes a real-time data processing unit, It is used to obtain the current host data in real time, import the current host data into the corresponding preset data unified program, obtain the current host data set, obtain the current host features to be processed, perform multicollinearity feature screening, and generate the current host set to be processed.

9. An abnormal host detection device based on AE algorithm, characterized in that: The device comprises: processor; and a memory having executable codes stored thereon, which, when the executable codes are executed, enable the processor to execute an abnormal host detection method based on the AE algorithm as described in any one of claims 1 to 4.

10. A non-volatile computer storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the abnormal host detection method based on the AE algorithm as described in any one of claims 1-4 is implemented.

Images