Network data leakage early warning system based on machine learning and use method thereof

Abstract

The invention relates to the technical field of intelligent early warning, in particular to a network data leakage early warning system based on machine learning and a use method thereof. The network data leakage early warning system based on machine learning comprises a network data unit, a data detection unit and a network early warning unit. According to the method, network flow data, user behavior data and operation log data are integrated, multi-source data are fused, relevance among the data is mined, highly effective features are provided for an early warning model, potential leakage risk capturing capacity is greatly improved, meanwhile, double-stage early warning judgment is adopted, experience judgment and intelligent analysis are combined, false report and missing report are reduced, and the risk of potential leakage risk capturing is improved. In addition, the machine learning model can be updated to adapt to a continuously changing network threat environment.

Description

Technical Field

[0001] The present invention relates to the field of intelligent early warning technology, and in particular to a network data leakage early warning system based on machine learning and a method for using the system. Background Art

[0002] In today's digital age, network data has become one of the core assets of enterprises and organizations, and its security is of vital importance. With the rapid development of network technology, network attack methods are becoming increasingly complex and diverse, and data leakage incidents occur frequently, causing huge economic losses and privacy risks to enterprises and individuals.

[0003] Machine learning technology has been widely used in the field of network security due to its powerful data processing and pattern recognition capabilities. By learning from large amounts of network data, machine learning models can automatically discover patterns and regularities in the data, thereby identifying abnormal behaviors.

[0004] Based on this, the present invention provides a network data leakage early warning system based on machine learning and a method of using the same to solve the technical problems raised above. Summary of the invention

[0005] The purpose of the present invention is to provide a network data leakage warning system based on machine learning and its use method, which integrates network traffic data, user behavior data and operation log data, fuses multi-source data, and mines the correlation between data to provide highly effective features for the warning model, greatly improving the ability to capture potential leakage risks. At the same time, a two-stage warning judgment is adopted, combining experience judgment and intelligent analysis to reduce false alarms and missed alarms. In addition, the machine learning model will be updated to adapt to the ever-changing network threat environment.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The first aspect of the present invention provides a network data leakage early warning system based on machine learning, comprising a network data unit, a data detection unit and a network early warning unit, wherein:

[0008] The network data unit is used to adopt network traffic data information, user behavior data information and operation log data information;

[0009] The data detection unit extracts features from the data according to the received data information, and makes an early warning judgment on network data leakage according to the extracted feature information, and the data detection unit is connected to the network data unit;

[0010] The network early warning unit is used to receive early warning information, as well as the above-mentioned preset threshold information and historical data information, and the network early warning unit is connected to the data detection unit.

[0011] The present invention is further configured as follows: the network data unit includes a network traffic module, a user behavior module, a log data module and a first communication module, wherein:

[0012] The network traffic module is used to use network traffic data information, wherein the network traffic data information includes source IP, destination IP and data packet size;

[0013] The user behavior module is used to use user behavior data information, wherein the user behavior data information includes access file type, access processing method and access times;

[0014] The log data module is used to adopt operation log data information, wherein the operation log data information includes query operation time and data export type;

[0015] The first communication module is used to realize information interaction between the network data unit and the data detection unit. The first communication module is connected to the network traffic module, the user behavior module and the log data module.

[0016] The present invention is further configured as follows: the data detection unit includes a second communication module, a feature extraction module and a database module, wherein:

[0017] The second communication module is used to realize information interaction between the data detection unit and the network data unit and the network early warning unit;

[0018] The feature extraction module is used to extract features from the data, and the feature extraction module is connected to the second communication module, wherein the feature extraction information includes an information entropy data group based on network traffic data information, a comprehensive feature index group based on operation log data information, and a probability array based on operation log data information;

[0019] The database module is used to store the received collected data, the historical data uploaded by the management personnel and the preset threshold information, and the database module is connected to both the second communication module and the feature extraction module.

[0020] The present invention is further configured as follows: the extraction process of the information entropy data group based on the network traffic data information is as follows:

[0021] Using the sliding window method, the network traffic data information with the same source IP and destination IP is divided into one group within the sliding window time;

[0022] Calculate the information entropy of each group within the corresponding sliding window time. In the formula, p(x i ) is the frequency of occurrence of the i-th data packet;

[0023] The information entropy obtained at different sliding window times is combined into an information entropy data group.

[0024] The present invention is further configured as follows: the extraction process of the comprehensive feature index group based on the operation log data information is as follows:

[0025] Assume there are n different access file types T i , m different access processing methods M j , within the sliding window time, the number of times the user accesses the file type is C Ti , the number of times the processing method is used is

[0026] Calculate the corresponding comprehensive characteristic index In the formula, ω ij is the weight coefficient, For file type T k Number of visits;

[0027] The comprehensive feature indexes obtained at different sliding window times are combined into a comprehensive feature index group.

[0028] The present invention is further configured as follows: the extraction process of the probability array based on the operation log data information is as follows:

[0029] Assume that there are a total of N in different sliding window times s Data export operation, where the file type is T i The number of occurrences is n is ;

[0030] Calculate and obtain the corresponding file type T i The probability of occurrence

[0031] Combine the comprehensive probabilities obtained from the query operation time into a probability array.

[0032] The present invention is further configured as follows: the data detection unit further includes an early warning determination module and an information marking module, wherein:

[0033] The early warning determination module performs a first data leakage determination according to the received data information, and performs a second data leakage determination based on the result of the first data leakage determination. The early warning determination module is connected to both the feature extraction module and the database module. The process of the first data leakage determination is as follows:

[0034] Compare the source IP with the pre-stored marked warning IP;

[0035] If the source IP is the same as the pre-stored marked warning IP, a corresponding warning is issued, otherwise, no warning is issued;

[0036] The process of determining the second data leakage is as follows:

[0037] When the first data leakage determination result is no warning, the feature extracted information is used as the input of the trained machine learning model to output the abnormal probability;

[0038] The obtained abnormal probability is compared with the preset threshold value. If the abnormal probability exceeds the preset threshold value, a corresponding warning is issued; otherwise, no warning is issued;

[0039] The information marking module is used to mark the IP that issues the warning and to update the machine learning model. The information marking module is connected to the second communication module, the database module and the warning determination module.

[0040] The present invention is further configured as follows: the network early warning unit includes a third communication module, an early warning display module and an information uploading module, wherein:

[0041] The third communication module is used to realize information interaction between the network early warning unit and the data detection unit;

[0042] The warning display module is used to display the warning information and the uploaded information, and the warning display module is connected to the third communication module;

[0043] The information uploading module is used for the management personnel to upload the preset warning threshold value and the collected historical data information, and the information uploading module is connected to both the third communication module and the warning display module.

[0044] The second aspect of the present invention also provides a method for using the above-mentioned network data leakage early warning system based on machine learning, comprising the following steps:

[0045] Use network traffic data information, user behavior data information and operation log data information;

[0046] Extract features from the collected network traffic data, user behavior data, and operation log data, and make early warning judgments on network data leaks based on the extracted feature information;

[0047] According to the warning judgment results, a warning message is issued.

[0048] The present invention is further configured as follows: before using the network flow data information, user behavior data information and operation log data information, it also includes the above-mentioned preset warning threshold and historically collected data information.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention integrates network traffic data, user behavior data and operation log data, fuses multi-source data, mines the correlation between data, provides highly effective features for the early warning model, and greatly improves the ability to capture potential leakage risks. At the same time, it adopts a two-stage early warning judgment, that is, the first stage quickly compares the source IP and the marked warning IP, and quickly screens based on known risk experience. The second stage uses a machine learning model for judgment, which can slow down the data processing volume and combine experience judgment with intelligent analysis to reduce false alarms and missed alarms. In addition, the machine learning model will be updated, and it can continue to learn and evolve as new risk situations emerge, adjust the early warning strategy in time, and adapt to the ever-changing network threat environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a system diagram of a network data leakage early warning system based on machine learning in the present invention.

[0052] Figure 2 This is a system diagram of a network data unit in a network data leakage early warning system based on machine learning according to the present invention.

[0053] Figure 3 This is a system diagram of a data detection unit in a network data leakage early warning system based on machine learning in the present invention.

[0054] Figure 4 This is a system diagram of a network early warning unit in a network data leakage early warning system based on machine learning in the present invention.

[0055] Description of Figure Numbers:

[0056] 100, network data unit; 110, network traffic module; 120, user behavior module; 130, log data module; 140, first communication module; 200, data detection unit; 210, second communication module; 220, feature extraction module; 230, database module; 240, warning determination module; 250, information marking module; 300, network warning unit; 310, third communication module; 320, warning display module; 330, information upload module. DETAILED DESCRIPTION

[0057] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Example:

[0059] like Figure 1-Figure 4As shown, this embodiment provides a network data leakage warning system based on machine learning, including a network data unit 100, a data detection unit 200 and a network warning unit 300, wherein: the network data unit 100 is used to adopt network traffic data information, user behavior data information and operation log data information; the data detection unit 200 extracts features of the data according to the received data information, and makes a warning judgment on the network data leakage according to the extracted feature information, and the data detection unit 200 is connected to the network data unit 100; the network warning unit 300 is used to receive warning information, as well as the above-mentioned preset threshold information and historical data information, and the network warning unit 300 is connected to the data detection unit 200.

[0060] In the present embodiment, it should be noted that the network early warning unit 300 is used to upload the historically collected data information, which is then uploaded to the data detection unit 200, and the machine learning model is trained in the data detection unit 200 to obtain a trained machine learning model. The network data unit 100 then uses the network traffic data information, user behavior data information, and operation log data information, and uploads the collected data information to the data detection unit 200. The data detection unit 200 will perform feature extraction on the collected information, that is, extract an information entropy data group based on the network traffic data information, a comprehensive feature index group based on the operation log data information, and a probability array based on the operation log data information, and use it as the input of the machine learning model to obtain the output warning probability, and compare it with the preset threshold to determine whether to issue a corresponding warning.

[0061] In the present invention, the network data unit 100 includes a network traffic module 110, a user behavior module 120, a log data module 130 and a first communication module 140, wherein: the network traffic module 110 is used to adopt network traffic data information, wherein the network traffic data information includes source IP, destination IP and data packet size; the user behavior module 120 is used to adopt user behavior data information, wherein the user behavior data information includes access file type, access processing method and access times; the log data module 130 is used to adopt operation log data information, wherein the operation log data information includes query operation time and data export type; the first communication module 140 is used to realize information interaction between the network data unit 100 and the data detection unit 200, and the first communication module 140 is connected to the network traffic module 110, the user behavior module 120 and the log data module 130.

[0062] In this embodiment, it should be noted that network traffic data information, user behavior data information and operation log data information are collected through the network traffic module 110, the user behavior module 120 and the log data module 130, and these data are accurately transmitted to the data detection unit 200 through the first communication module 140, so as to provide basic data support for subsequent data processing and analysis, and ensure that the system can provide network data leakage warning based on rich data sources.

[0063] In the present invention, the data detection unit 200 includes a second communication module 210, a feature extraction module 220 and a database module 230, wherein: the second communication module 210 is used to realize information interaction between the data detection unit 200 and the network data unit 100 and the network early warning unit 300; the feature extraction module 220 is used to extract features from the data, and the feature extraction module 220 is connected to the second communication module 210, wherein the feature extracted information includes an information entropy data group based on network traffic data information, a comprehensive feature index group based on operation log data information and a probability array based on operation log data information; the database module 230 is used to store the received collected data, the historical data uploaded by the management personnel and the preset threshold information, and the database module 230 is connected to both the second communication module 210 and the feature extraction module 220.

[0064] Among them, the process of extracting the information entropy data group based on network traffic data information is as follows:

[0065] Using the sliding window method, the network traffic data information of the same source IP and destination IP is divided into one group within the sliding window time; the information entropy of each group within the corresponding sliding window time is calculated. In the formula, p(x i ) is the frequency of occurrence of the i-th data packet; the information entropy obtained at different sliding window times is combined into an information entropy data group.

[0066] The extraction process of the comprehensive feature index group based on the operation log data information is as follows:

[0067] Assume there are n different access file types T i , m different access processing methods M j , within the sliding window time, the number of times the user accesses the file type is The number of times the treatment method is used is Calculate the corresponding comprehensive characteristic index In the formula, ω ij is the weight coefficient, For file type T k The number of visits of different sliding window times is combined into a comprehensive feature index group.

[0068] The extraction process of the probability array based on the operation log data information is as follows:

[0069] Assume that there are a total of N in different sliding window times s Data export operation, where the file type is T i The number of occurrences is n is ; Calculate and obtain the corresponding file type T i The probability of occurrence Combine the comprehensive probabilities obtained from the query operation time into a probability array.

[0070] In this embodiment, it should be noted that the information uploaded by the network data unit 100, namely, the network flow data information, the user behavior data information and the operation log data information, is received by the second communication module 210, and is transmitted to the feature extraction module 220, which extracts features, namely, an information entropy data group based on the network flow data information, a comprehensive feature index group based on the operation log data information and a probability array based on the operation log data information, and stores them in the database module 230, and converts the original data into representative quantitative features through feature extraction, and then uses the extracted features to input the machine learning model to make early warning judgments, which can identify potential data leakage behaviors, improve the scientificity and accuracy of early warnings, and reduce false alarms and missed reports. For network flow data, the information entropy data group is calculated by a sliding window method, which can effectively capture the dynamic change characteristics of network flow in different time windows, and can reflect the stability and abnormal fluctuation of network flow. In addition, by calculating the comprehensive feature index group and the probability array, key features are extracted from the diversity and frequency of user operations, which is helpful to discover abnormal operation modes.

[0071] In the present invention, the data detection unit 200 also includes an early warning judgment module 240 and an information marking module 250, wherein: the early warning judgment module 240 performs a first data leakage judgment according to the received data information, and performs a second data leakage judgment based on the result of the first data leakage judgment, and the early warning judgment module 240 is connected to both the feature extraction module 220 and the database module 230, wherein the process of the first data leakage judgment is as follows: compare the source IP with the pre-stored marked early warning IP; if the source IP is the same as the pre-stored marked early warning IP, a corresponding early warning is issued, otherwise, no early warning is issued.

[0072] The process of the second data leakage judgment is as follows: when the result of the first data leakage judgment is no warning, the feature extracted information is used as the input of the trained machine learning model to output the abnormality probability; the obtained abnormality probability is compared with the preset threshold, if the abnormality probability exceeds the preset threshold, a corresponding warning is issued, otherwise, no warning is issued.

[0073] The information marking module 250 is used to mark the IP that issues the warning and to update the machine learning model. The information marking module 250 is connected to the second communication module 210 , the database module 230 and the warning determination module 240 .

[0074] In this embodiment, it should be noted that the early warning judgment module 240 will first judge the first data leakage, and then judge the second data leakage. The collected machine learning model uses a neural network model, which is trained by historical data and will not be repeated here. This embodiment adopts a two-time data leakage judgment mechanism. First, the source IP is compared with the pre-stored marked early warning IP, which can quickly discover some known risk sources; if there is no early warning in the first judgment, the extracted feature information is input into the trained machine learning model, and a second judgment is made by calculating the abnormal probability and comparing it with the preset threshold. The multi-level judgment mechanism not only utilizes the traditional experience judgment method, but also combines the intelligent analysis ability of machine learning, greatly improving the accuracy and reliability of the early warning, and reducing false alarms and missed reports. In addition, the information marking module 250 marks the IP after issuing the early warning and updates the machine learning model. This enables the system to continuously learn and evolve as new risk situations emerge, adjust the early warning strategy in time, and adapt to the ever-changing network threat environment.

[0075] In the present invention, the network warning unit 300 includes a third communication module 310, a warning display module 320 and an information upload module 330, wherein: the third communication module 310 is used to realize information interaction between the network warning unit 300 and the data detection unit 200; the warning display module 320 is used to display warning information and uploaded information, and the warning display module 320 is connected to the third communication module 310; the information upload module 330 is used for management personnel to upload preset warning thresholds and collected historical data information, and the information upload module 330 is connected to both the third communication module 310 and the warning display module 320.

[0076] In this embodiment, it should be noted that warning information can be received through the third communication module 310 and transmitted to the warning display module 320, and the warning is displayed through the warning display module 320 for management personnel to view. At the same time, management personnel can also upload preset warning thresholds and collected historical data information through the information upload module 330, and upload them to the data detection unit 200 through the third communication module 310.

[0077] In addition, this embodiment also provides a method for using the above-mentioned network data leakage early warning system based on machine learning, including the following steps:

[0078] Use network traffic data information, user behavior data information and operation log data information.

[0079] Feature extraction is performed on the collected network traffic data information, user behavior data information and operation log data information, and an early warning judgment on network data leakage is made based on the extracted feature information.

[0080] According to the warning judgment results, a warning message is issued.

[0081] Among them, before using the network traffic data information, user behavior data information and operation log data information, the above-mentioned preset warning thresholds and historically collected data information are also included.

[0082] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0083] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A network data leakage early warning system based on machine learning, characterized in that: The invention comprises a network data unit (100), a data detection unit (200) and a network early warning unit (300), wherein: The network data unit (100) is used to use network traffic data information, user behavior data information and operation log data information; The data detection unit (200) extracts features from the data according to the received data information, and makes an early warning judgment on network data leakage according to the extracted feature information. The data detection unit (200) is connected to the network data unit (100); The network early warning unit (300) is used to receive early warning information, as well as the above-mentioned preset threshold information and historical data information. The network early warning unit (300) is connected to the data detection unit (200).

2. According to the machine learning-based network data leakage early warning system of claim 1, it is characterized in that: The network data unit (100) comprises a network traffic module (110), a user behavior module (120), a log data module (130) and a first communication module (140), wherein: The network traffic module (110) is used to use network traffic data information, wherein the network traffic data information includes source IP, destination IP and data packet size; The user behavior module (120) is used to use user behavior data information, wherein the user behavior data information includes access file type, access processing method and access times; The log data module (130) is used to use operation log data information, wherein the operation log data information includes query operation time and data export type; The first communication module (140) is used to realize information interaction between the network data unit (100) and the data detection unit (200), and the first communication module (140) is connected to the network traffic module (110), the user behavior module (120) and the log data module (130).

3. The network data leakage early warning system based on machine learning according to claim 1 is characterized in that: The data detection unit (200) comprises a second communication module (210), a feature extraction module (220) and a database module (230), wherein: The second communication module (210) is used to realize information interaction between the data detection unit (200) and the network data unit (100) and the network early warning unit (300); The feature extraction module (220) is used to extract features from data, and the feature extraction module (220) is connected to the second communication module (210), wherein the feature extraction information includes an information entropy data group based on network traffic data information, a comprehensive feature index group based on operation log data information, and a probability array based on operation log data information; The database module (230) is used to store the received collected data, the historical data uploaded by the management personnel and the preset threshold information, and the database module (230) is connected to both the second communication module (210) and the feature extraction module (220).

4. The network data leakage early warning system based on machine learning according to claim 3 is characterized in that: The extraction process of the information entropy data group based on the network traffic data information is as follows: Using the sliding window method, the network traffic data information with the same source IP and destination IP is divided into one group within the sliding window time; Calculate the information entropy of each group within the corresponding sliding window time. In the formula, p(x i ) is the frequency of occurrence of the i-th data packet; The information entropy obtained at different sliding window times is combined into an information entropy data group.

5. The network data leakage early warning system based on machine learning according to claim 3 is characterized in that: The extraction process of the comprehensive feature index group based on the operation log data information is as follows: Assume there are n different access file types T i , m different access processing methods M j , within the sliding window time, the number of times the user accesses the file type is The number of times the treatment method is used is Calculate the corresponding comprehensive characteristic index In the formula, ω ij is the weight coefficient, For file type T k Number of visits; The comprehensive feature indexes obtained at different sliding window times are combined into a comprehensive feature index group.

6. The network data leakage early warning system based on machine learning according to claim 3 is characterized in that: The extraction process of the probability array based on the operation log data information is as follows: Assume that there are a total of N in different sliding window times s Data export operation, where the file type is T i The number of occurrences is n is ; Calculate and obtain the corresponding file type T i The probability of occurrence Combine the comprehensive probabilities obtained from the query operation time into a probability array.

7. The network data leakage early warning system based on machine learning according to claim 3 is characterized in that: The data detection unit (200) further comprises a warning determination module (240) and an information marking module (250), wherein: The early warning determination module (240) performs a first data leakage determination according to the received data information, and performs a second data leakage determination based on the result of the first data leakage determination. The early warning determination module (240) is connected to both the feature extraction module (220) and the database module (230). The process of the first data leakage determination is as follows: Compare the source IP with the pre-stored marked warning IP; If the source IP is the same as the pre-stored marked warning IP, a corresponding warning is issued, otherwise, no warning is issued; The process of determining the second data leakage is as follows: When the first data leakage determination result is no warning, the feature extracted information is used as the input of the trained machine learning model to output the abnormal probability; The obtained abnormal probability is compared with the preset threshold value. If the abnormal probability exceeds the preset threshold value, a corresponding warning is issued; otherwise, no warning is issued; The information marking module (250) is used to mark the IP that issues the warning and to update the machine learning model. The information marking module (250) is connected to the second communication module (210), the database module (230) and the warning determination module (240).

8. The network data leakage early warning system based on machine learning according to claim 1 is characterized in that: The network warning unit (300) comprises a third communication module (310), a warning display module (320) and an information upload module (330), wherein: The third communication module (310) is used to realize information interaction between the network early warning unit (300) and the data detection unit (200); The warning display module (320) is used to display warning information and uploaded information, and the warning display module (320) is connected to the third communication module (310); The information upload module (330) is used by the management personnel to upload the preset warning threshold and the collected historical data information, and the information upload module (330) is connected to both the third communication module (310) and the warning display module (320).

9. A method for using a network data leakage early warning system based on machine learning according to any one of claims 1 to 8, characterized in that: The following steps are involved: Use network traffic data information, user behavior data information and operation log data information; Extract features from the collected network traffic data, user behavior data, and operation log data, and make early warning judgments on network data leaks based on the extracted feature information; According to the warning judgment results, a warning message is issued.

10. The method for using a network data leakage early warning system based on machine learning according to claim 9, characterized in that: Before using network traffic data information, user behavior data information and operation log data information, the above-mentioned preset warning thresholds and historically collected data information are also included.

Images