Network perception anomaly detection system based on big data

Abstract

The invention relates to the technical field of network detection, and discloses a network perception anomaly detection system based on big data, and the system comprises an acquisition module which is configured to intercept network data in a preset time period, classify the network data, and obtain different types of data; the preliminary analysis module is configured to extract flow characteristics and connection characteristics of the network flow data, preliminarily judge whether the network is abnormal or not according to the flow deviation value and the connection deviation value, and obtain a comprehensive preliminary judgment result of the network by combining two preliminary judgment results; the deep analysis module is configured to calculate an equipment anomaly index through the interface error packet number and the configuration change information; calculating a network anomaly probability value according to the equipment anomaly index and the deviation degree; and the alarm module is configured to determine an alarm level according to the network anomaly probability value. According to the invention, through multi-dimensional data fusion and analysis, a blind area of single-dimensional data monitoring is effectively avoided, so that the detection result is more accurate and reliable.

Description

Technical Field

[0001] The present invention relates to the technical field of network detection, and more particularly, to a network perception anomaly detection system based on big data. Background Art

[0002] With the rapid development of Internet technology and the continuous deepening of enterprise informatization construction, the network has become the core infrastructure for various business activities. However, the complexity and openness of the network also make it face more and more security threats and operation risks.

[0003] Traditional network security monitoring methods mainly rely on static rules, log analysis or signature-based detection methods. These methods are often slow to respond to new attacks and mutated attacks and are difficult to cope with the constantly changing network environment. In addition, with the explosive growth of network traffic and the number of device connections, traditional monitoring methods based on simple threshold detection are difficult to meet the analysis requirements of large-scale and massive data. In this context, big data technology provides new ideas for network perception and anomaly detection. Big data can process a large number of heterogeneous data sources, mine potential associations and abnormal behaviors, thereby improving detection accuracy and response speed. At present, most big data-based network monitoring systems focus on the analysis of a single type of data, such as traffic data or device data, and ignore the comprehensive analysis of multi-dimensional data.

[0004] Therefore, it is necessary to provide a network perception anomaly detection system based on big data to solve the problem that the existing network security monitoring methods lack comprehensive analysis of multi-dimensional data. Summary of the Invention

[0005] In view of this, the present invention proposes a network perception anomaly detection system based on big data, aiming to solve the problem that the existing network security monitoring methods lack comprehensive analysis of multi-dimensional data.

[0006] The present invention proposes a network perception anomaly detection system based on big data, including:

[0007] An acquisition module, configured to intercept network data within a preset time period, classify the network data, and obtain different categories of data; wherein, the different categories of data include network traffic data, device data, and user behavior data; the network traffic data includes traffic characteristics and connection characteristics; the device data includes the number of interface error packets and configuration change information; the user behavior data includes the total number of logins, the number of login errors, and the login frequency;

[0008] The preliminary analysis module is configured to extract the traffic characteristics and connection characteristics of the network traffic data, calculate the traffic stability value according to the traffic characteristics, calculate the connection stability value according to the connection characteristics, calculate the traffic deviation value between the traffic stability value and the preset traffic stability value and the connection deviation value between the connection stability value and the preset connection stability value respectively, preliminarily judge whether the network has an abnormality according to the traffic deviation value and the connection deviation value respectively, and obtain the comprehensive preliminary judgment result of the network by combining the two preliminary judgment results;

[0009] The in-depth analysis module is configured to calculate the device anomaly index through the number of interface error packets and configuration change information if the comprehensive preliminary judgment result is that the network has an abnormality; calculate the deviation degree of the user behavior according to the user behavior data and the historical behavior baseline, and calculate the network anomaly probability value according to the device anomaly index and the deviation degree;

[0010] The alarm module is configured to judge whether to perform an anomaly alarm according to the network anomaly probability value, and determine the alarm level according to the network anomaly probability value.

[0011] Further, the acquisition module is configured to intercept the network data within a preset time period, classify the network data, and when obtaining different types of data, include:

[0012] The preset acquisition time period, clean the acquired network data, and remove duplicate, invalid, and format error data;

[0013] Divide the network data into network traffic data, device data, and user behavior data according to the data type, and store the network data separately according to different categories and acquisition time periods;

[0014] Construct a storage sequence At = [L(l1, l2, l3, …, ln), S(s1, s2, s3, …, sn), Y(y1, y2, y3, …, yn)];

[0015] Where At represents the storage sequence with serial number A and time period t, L represents network traffic data, li represents the i-th network traffic data, i = 1, 2, 3, …, n; S represents device data, sj represents the j-th device data, j = 1, 2, 3, …, n; Y represents user behavior data, yk represents the k-th user behavior data, k = 1, 2, 3, …, n.

[0016] Further, when the preliminary analysis module is configured to extract the traffic characteristics and connection characteristics of the network traffic data, calculate the traffic stability value according to the traffic characteristics, calculate the connection stability value according to the connection characteristics, and calculate the traffic deviation value between the traffic stability value and the preset traffic stability value and the connection deviation value between the connection stability value and the preset connection stability value respectively, it includes:

[0017] The flow characteristics include total flow, flow rate, and flow fluctuation value; the connection characteristics include the number of newly established connections, connection failure rate, and connection duration;

[0018] Calculate the flow stability value based on the total flow, flow rate, and flow fluctuation value, and calculate the connection stability value based on the number of newly established connections, connection failure rate, and connection duration;

[0019] Obtain the average value of the historical normal total flow, the average value of the historical normal flow rate, and the average value of the historical normal flow fluctuation value in the historical period that is the same as the preset collection period, and calculate the flow stability value through the following formula:

[0020] W1 = a1*(T1 / T2) + a2*(R1 / R2) + a3*(V1 / V2);

[0021] In the above formula, W1 represents the flow stability value, T1 represents the total flow, T2 represents the average value of the historical normal total flow, R1 represents the flow rate, R2 represents the average value of the historical normal flow rate, V1 represents the flow fluctuation value, V2 represents the average value of the historical normal flow fluctuation value, and a1, a2, and a3 respectively represent weight coefficients, where the value ranges of a1, a2, and a3 are all [0, 1], and a1 + a2 + a3 = 1;

[0022] Obtain the average value of the historical normal number of newly established connections, the average value of the historical normal connection failure rate, and the average value of the historical normal connection duration in the historical period that is the same as the preset collection period, and calculate the connection stability value through the following formula:

[0023] W2 = b1*(X1 / X2) + b2*(B1 / B2) + b3*(C1 / C2);

[0024] In the above formula, W2 represents the connection stability value, X1 represents the number of newly established connections, X2 represents the average value of the historical normal number of newly established connections, B1 represents the connection failure rate, B2 represents the average value of the historical normal connection failure rate, C1 represents the connection duration, C2 represents the average value of the historical normal connection duration, and b1, b2, and b3 respectively represent weight coefficients, where the value ranges of b1, b2, and b3 are all [0, 1], and b1 + b2 + b3 = 1.

[0025] Further, when the preliminary analysis module is configured to preliminarily determine whether the network is abnormal according to the flow deviation value and the connection deviation value respectively, it includes:

[0026] Preset the maximum flow deviation and the maximum connection deviation;

[0027] Calculate the flow deviation value between the calculated flow stability value and the preset flow stability value, and calculate the connection deviation value between the connection stability value and the preset connection stability value;

[0028] If the flow deviation value is greater than or equal to the maximum flow deviation value, it is preliminarily determined that the network has an abnormality; otherwise, it is preliminarily determined that the network has no abnormality;

[0029] If the connection deviation value is greater than or equal to the preset flow stability value, it is preliminarily determined that the network has an abnormality; otherwise, it is preliminarily determined that the network has no abnormality.

[0030] Furthermore, when the preliminary analysis module is configured to obtain the comprehensive preliminary judgment result of the network by combining the two preliminary judgment results, it includes:

[0031] If both preliminary judgment results are that the network has no abnormality, the comprehensive preliminary judgment result of the network is that the network status is normal, and no abnormality alarm is made;

[0032] If there is an abnormality in the network in one of the two preliminary judgment results, the comprehensive preliminary judgment result of the network is that the network has an abnormality.

[0033] Furthermore, when the in-depth analysis module is configured to calculate the device abnormality index through the interface error packet number and the configuration change information if the comprehensive preliminary judgment result is that the network has an abnormality, it includes:

[0034] Collect the mean value of the historical normal interface error packet number and the mean value of the historical normal configuration change times, and calculate the device abnormality index through the following formula:

[0035]

[0036] In the above formula, E represents the device abnormality index, n represents the time window length, Cb i represents the interface error packet number in the i-th period, Cb represents the mean value of the historical normal interface error packet number, Bc i represents the number of configuration changes in the i-th period, and Bc represents the mean value of the historical normal configuration change times.

[0037] Furthermore, when the in-depth analysis module is configured to calculate the deviation degree of the user behavior according to the user behavior data and the historical behavior baseline, it includes:

[0038] Calculate the mean value of the total number of normal logins, the mean value of the number of normal login errors, and the mean value of the normal login frequency in the historical preset period according to the historical user behavior data, and use the mean value of the total number of normal logins, the mean value of the number of normal login errors, and the mean value of the normal login frequency in the historical preset period as the historical behavior baseline;

[0039] Calculate the deviation degree of the user behavior through the following formula:

[0040]

[0041] In the above formula, PL represents the deviation degree of user behavior, D represents the total number of logins, D0 represents the average value of the total number of normal logins, E represents the number of login errors, E0 represents the average value of the number of normal login errors, P represents the login frequency, and P0 represents the average value of the normal login frequency.

[0042] Furthermore, when the deep analysis module is configured to calculate the network anomaly probability value according to the device anomaly index and the deviation degree, it includes:

[0043] Calculating the sum value of the device anomaly index and the deviation degree of user behavior;

[0044] Setting a first sum value and a second sum value, where the first sum value is less than the second sum value;

[0045] If the sum value is less than the first sum value, the network anomaly probability value is the first probability;

[0046] If the sum value is greater than or equal to the first sum value and less than or equal to the second sum value, the network anomaly probability value is the second probability;

[0047] If the sum value is greater than the second sum value, the network anomaly probability value is the third probability;

[0048] Among them, the value range of the network probability value is 0 < the first probability < the second probability < the third probability < 1.

[0049] Furthermore, when the alarm module is configured to determine whether to perform an anomaly alarm according to the network anomaly probability value, it includes:

[0050] If the network anomaly probability value is the first probability, it is determined not to perform an anomaly alarm;

[0051] If the network anomaly probability value is not the first probability, it is determined to perform an anomaly alarm.

[0052] Furthermore, when the alarm module is configured to determine the alarm level according to the network anomaly probability value, it includes:

[0053] If the network anomaly probability value is the second probability, the alarm level is level one;

[0054] If the network anomaly probability value is the third probability, the alarm level is level two; among them, level one of the alarm levels is lower than level two;

[0055] When the alarm level is level one, increase the duration of the acquisition time period so that the increased duration of the acquisition time period is 1.5 times the current duration of the acquisition time period;

[0056] When the alarm level is level two, increase the duration of the acquisition time period so that the increased duration of the acquisition time period is twice the current duration of the acquisition time period.

[0057] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention can effectively improve network security and stability. By comprehensively analyzing network traffic, device status, and user behavior data, potential anomalies are detected in real time to ensure that problems can be discovered at an early stage and corresponding measures can be taken. The acquisition module ensures that different types of data can be analyzed separately by intercepting and classifying network data in real time, effectively eliminating data redundancy and interference. The preliminary analysis module quickly identifies abnormal fluctuations in network traffic and connections by calculating the traffic stability value and connection stability value and comparing with historical data, thus providing a basis for subsequent in-depth analysis. The in-depth analysis module further accurately identifies the root cause of anomalies by calculating the device anomaly index and the deviation degree of user behavior, and calculates the network anomaly probability value based on this data to achieve precise early warning. The alarm module accurately determines the alarm level according to different anomaly probabilities to ensure that the response to abnormal events is targeted. Overall, the system can detect and early warn of network anomalies in a timely manner, improving network security, stability, and management efficiency. In addition, through multi-dimensional data fusion and analysis, the system effectively avoids the blind spots of single-dimensional data monitoring, making the detection results more accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0059] Figure 1 It is a functional block diagram of the network-aware anomaly detection system based on big data provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0061] In some embodiments of the present application, refer to Figure 1As shown in the figure, this embodiment provides a network perception anomaly detection system based on big data, including:

[0062] A collection module, configured to intercept network data within a preset time period, classify the network data, and obtain data of different categories; wherein, the different categories of data include network traffic data, device data, and user behavior data; the network traffic data includes traffic characteristics and connection characteristics; the device data includes the number of interface error packets and configuration change information; the user behavior data includes the total number of logins, the number of login errors, and the login frequency;

[0063] A preliminary analysis module, configured to extract the traffic characteristics and connection characteristics of the network traffic data, calculate the traffic stability value according to the traffic characteristics, calculate the connection stability value according to the connection characteristics, calculate the traffic deviation value between the traffic stability value and the preset traffic stability value and the connection deviation value between the connection stability value and the preset connection stability value respectively, preliminarily judge whether the network is abnormal according to the traffic deviation value and the connection deviation value respectively, and combine the two preliminary judgment results to obtain the comprehensive preliminary judgment result of the network;

[0064] A deep analysis module, configured to calculate the device anomaly index through the number of interface error packets and configuration change information if the comprehensive preliminary judgment result is that the network is abnormal; calculate the deviation degree of the user behavior according to the user behavior data and the historical behavior baseline, and calculate the network anomaly probability value according to the device anomaly index and the deviation degree;

[0065] An alarm module, configured to judge whether to perform an anomaly alarm according to the network anomaly probability value, and determine the alarm level according to the network anomaly probability value.

[0066] It can be understood that the present invention can effectively improve network security and stability. By comprehensively analyzing network traffic, device status, and user behavior data, potential anomalies can be detected in real time, ensuring that the system can discover problems at an early stage and take corresponding measures. The collection module ensures that different categories of data can be analyzed separately by intercepting and classifying network data in real time, effectively eliminating data redundancy and interference. The preliminary analysis module quickly identifies abnormal fluctuations in network traffic and connections by calculating the traffic stability value and the connection stability value and combining the comparison with historical data, thus providing a basis for subsequent in-depth analysis. The deep analysis module further accurately identifies the root cause of the anomaly by calculating the device anomaly index and the deviation degree of the user behavior, and calculates the network anomaly probability value based on these data, achieving precise early warning. The alarm module accurately determines the alarm level according to different anomaly probabilities, ensuring that the response to anomaly events is targeted. Overall, the system can detect and warn of network anomalies in a timely manner, improving network security, stability, and management efficiency. In addition, through multi-dimensional data fusion and analysis, the system effectively avoids the blind spots of single-dimensional data monitoring, making the detection results more accurate and reliable.

[0067] In some embodiments of the present application, the acquisition module is configured to intercept network data within a preset time period, classify the network data, and when obtaining data of different categories, it includes:

[0068] During the preset acquisition time period, clean the acquired network data to remove duplicate, invalid, and incorrectly formatted data;

[0069] Divide the network data into network traffic data, device data, and user behavior data according to the data type, and store the network data separately according to different categories and acquisition time periods;

[0070] Construct a storage sequence At = [L(l1, l2, l3,..., ln), S(s1, s2, s3,..., sn), Y(y1, y2, y3,..., yn)];

[0071] Where At represents the storage sequence with serial number A and time period t, L represents network traffic data, li represents the i-th network traffic data, i = 1, 2, 3,..., n; S represents device data, sj represents the j-th device data, j = 1, 2, 3,..., n; Y represents user behavior data, yk represents the k-th user behavior data, k = 1, 2, 3,..., n.

[0072] It can be understood that in the embodiments of the present application, by preprocessing, cleaning, and classifying the network data, the acquisition module can not only improve the quality of the data, but also effectively reduce the impact of invalid or incorrect data on subsequent analysis. This process ensures the accuracy and integrity of the network data, greatly improving the efficiency and accuracy of data analysis. By dividing the network data into network traffic data, device data, and user behavior data according to the type and storing them separately according to different time periods, various types of data can be clearly managed and analyzed. This structured storage method enables the system to quickly extract the required data, improving the query efficiency and ensuring rapid response in actual operations. The method of constructing the storage sequence helps to more clearly display the data status of each time period, facilitating the comparison of the change trends in different time periods and thus discovering potential anomalies. Through the classification processing of traffic data, device data, and user behavior data, the system can deeply analyze the correlation between various types of data and identify hidden network security threats or operation problems. In addition, refining the data to the granularity of each time period and category helps to accurately locate the root cause of the problem and improve the accuracy of anomaly detection. In summary, this data acquisition and storage method provides strong data support for subsequent network traffic analysis, anomaly detection, and security monitoring, greatly improving the reliability and response speed of the system.

[0073] In some embodiments of the present application, the preliminary analysis module is configured to extract the traffic characteristics and connection characteristics of network traffic data, calculate the traffic stability value according to the traffic characteristics, and calculate the connection stability value according to the connection characteristics. When calculating the traffic deviation value between the traffic stability value and the preset traffic stability value and the connection deviation value between the connection stability value and the preset connection stability value, it includes:

[0074] The traffic characteristics include total traffic, traffic rate, and traffic fluctuation value; the connection characteristics include the number of newly established connections, connection failure rate, and connection duration.

[0075] Calculate the traffic stability value according to the total traffic, traffic rate, and traffic fluctuation value, and calculate the connection stability value according to the number of newly established connections, connection failure rate, and connection duration.

[0076] Obtain the average value of the historical normal total traffic, the average value of the historical normal traffic rate, and the average value of the historical normal traffic fluctuation value in the historical time period that is the same as the preset collection time period, and calculate the traffic stability value through the following formula:

[0077] W1 = a1*(T1 / T2) + a2*(R1 / R2) + a3*(V1 / V2);

[0078] In the above formula, W1 represents the traffic stability value, T1 represents the total traffic, T2 represents the average value of the historical normal total traffic, R1 represents the traffic rate, R2 represents the average value of the historical normal traffic rate, V1 represents the traffic fluctuation value, V2 represents the average value of the historical normal traffic fluctuation value, and a1, a2, and a3 respectively represent weight coefficients. Among them, the value ranges of a1, a2, and a3 are all [0, 1], and a1 + a2 + a3 = 1;

[0079] Obtain the average value of the historical normal number of newly established connections, the average value of the historical normal connection failure rate, and the average value of the historical normal connection duration in the historical time period that is the same as the preset collection time period, and calculate the connection stability value through the following formula:

[0080] W2 = b1*(X1 / X2) + b2*(B1 / B2) + b3*(C1 / C2);

[0081] In the above formula, W2 represents the connection stability value, X1 represents the number of newly established connections, X2 represents the average value of the historical normal number of newly established connections, B1 represents the connection failure rate, B2 represents the average value of the historical normal connection failure rate, C1 represents the connection duration, C2 represents the average value of the historical normal connection duration, and b1, b2, and b3 respectively represent weight coefficients. Among them, the value ranges of b1, b2, and b3 are all [0, 1], and b1 + b2 + b3 = 1.

[0082] It can be understood that in the embodiments of the present application, the preliminary analysis module provides an important quantitative basis for network anomaly detection by extracting the traffic characteristics and connection characteristics of network traffic data and calculating the traffic stability value and connection stability value based on these characteristics. By calculating the traffic stability value and connection stability value and comparing them with the historical normal values, it is possible to effectively identify situations of traffic fluctuations and connection anomalies, thereby discovering potential network problems or attack behaviors in advance.

[0083] Specifically, the calculation of the traffic stability value takes into account the total traffic, traffic rate, and traffic fluctuation value. By comparing with historical normal data, it can timely capture abnormal fluctuations or overload situations of traffic. The connection stability value focuses on the number of newly established connections, connection failure rate, and connection duration. These indicators directly reflect the health status of network connections and can effectively identify problems such as abnormal connection requests or network interruptions. By comprehensively considering these characteristics, the system can not only judge the stability of the current traffic and connections but also accurately detect possible network failures or malicious attacks. The advantage of this analysis method lies in its high flexibility and accuracy. By dynamically adjusting the calculation weights of traffic and connections, it can be optimized according to different network environments or requirements to adapt to complex and changing network conditions. In addition, the comparative analysis of historical normal data provides a benchmark to help the system better understand the differences between normal and abnormal behaviors, improving the accuracy and efficiency of anomaly detection. This method helps to timely identify and respond to network anomalies, ensuring the stability and security of the network.

[0084] In some embodiments of the present application, when the preliminary analysis module is configured to preliminarily judge whether the network has an anomaly according to the traffic deviation value and the connection deviation value respectively, it includes:

[0085] Presetting the maximum traffic deviation value and the maximum connection deviation value;

[0086] Calculating the traffic deviation value between the traffic stability value and the preset traffic stability value, and calculating the connection deviation value between the connection stability value and the preset connection stability value;

[0087] If the traffic deviation value is greater than or equal to the maximum traffic deviation value, it is preliminarily judged that the network has an anomaly; otherwise, it is preliminarily judged that the network has no anomaly;

[0088] If the connection deviation value is greater than or equal to the preset traffic stability value, it is preliminarily judged that the network has an anomaly; otherwise, it is preliminarily judged that the network has no anomaly.

[0089] In some embodiments of the present application, when the preliminary analysis module is configured to obtain the comprehensive preliminary judgment result of the network by combining the two preliminary judgment results, it includes:

[0090] If both preliminary judgment results indicate that the network is not abnormal, the comprehensive preliminary judgment result of the network is that the network status is normal, and no abnormal alarm is issued.

[0091] If there is a network abnormality in either of the two preliminary judgment results, the comprehensive preliminary judgment result of the network is that the network is abnormal.

[0092] It can be understood that in this application, the preliminary analysis module combines the traffic deviation value and the connection deviation value to make a preliminary judgment on network abnormalities. This method effectively incorporates the stability of the two key factors of traffic and connection into the judgment criteria for anomaly detection, enhancing the comprehensiveness and accuracy of network monitoring. By setting the preset maximum traffic deviation value and maximum connection deviation value, the system can quickly determine abnormal fluctuations in traffic or connection, thereby promptly identifying potential problems. Specifically, if the traffic deviation value or the connection deviation value exceeds the preset maximum threshold, the system immediately triggers an abnormal alarm, indicating that there may be a network abnormality or security threat. First, the deviation values of traffic and connection are calculated independently, avoiding misjudgments that may be caused by a single indicator. Second, by combining the two preliminary judgment results, the system can perform a more accurate comprehensive analysis. For example, if the traffic is abnormal but the connection is normal, or the connection is abnormal but the traffic is normal, the system will combine these two signals to make a more accurate comprehensive judgment, avoiding blind alarms that rely solely on a single factor. This can reduce false alarms, improve the reliability and efficiency of the system, and ensure more accurate network anomaly detection. In addition, this method can be optimized according to different network environments by flexibly adjusting the threshold and deviation judgment criteria, adapting to network architectures of different scales and complexities, and effectively ensuring the stability and security of the network.

[0093] In some embodiments of this application, when the deep analysis module is configured to calculate the device anomaly index through the number of interface error packets and configuration change information if the comprehensive preliminary judgment result is that the network is abnormal, it includes:

[0094] Collect the mean value of the historical normal number of interface error packets and the mean value of the historical normal number of configuration changes, and calculate the device anomaly index through the following formula:

[0095]

[0096] In the above formula, E represents the device anomaly index, n represents the time window length, Cb i represents the number of interface error packets in the i-th period, Cb represents the mean value of the historical normal number of interface error packets, Bc i represents the number of configuration changes in the i-th period, and Bc represents the mean value of the historical normal number of configuration changes.

[0097] It can be understood that in some embodiments of the present application, the configuration of the in-depth analysis module allows for quantifying the degree of device abnormality through specific calculation methods when initially determining that the network is abnormal. Specifically, this module collects the normal means of the number of interface error packets and the number of configuration changes in historical data, and uses this data to calculate the device abnormality index, providing an objective and quantitative means to evaluate the abnormal state of the device. By comparing the differences between the number of interface error packets and the number of configuration changes in the current period with the historical means, abnormal fluctuations during device operation can be effectively identified. This method of calculating the sum of squares of these differences ensures that the impact of abnormal values on the total index is significant, making the anomaly detection more sensitive and accurate. In addition, this method also considers the length of the time window, which means it can adapt to the anomaly detection requirements on different time scales, improving the flexibility and applicability of anomaly detection. Generally speaking, this in-depth analysis module provides a powerful monitoring and early warning mechanism for the health status of network devices through an accurate mathematical model, helping to timely discover and handle potential network problems and ensuring the stable operation of the network.

[0098] In some embodiments of the present application, when the in-depth analysis module is configured to calculate the deviation degree of user behavior based on user behavior data and the historical behavior baseline, it includes:

[0099] Calculating the mean value of the total number of normal logins, the mean value of the number of normal login errors, and the mean value of the normal login frequency within a historical preset period based on historical user behavior data, and using the mean value of the total number of normal logins, the mean value of the number of normal login errors, and the mean value of the normal login frequency within the historical preset period as the historical behavior baseline;

[0100] Calculating the deviation degree of user behavior through the following formula:

[0101]

[0102] In the above formula, PL represents the deviation degree of user behavior, D represents the total number of logins, D0 represents the mean value of the total number of normal logins, E represents the number of login errors, E0 represents the mean value of the number of normal login errors, P represents the login frequency, and P0 represents the mean value of the normal login frequency.

[0103] It can be understood that in some embodiments of the present application, the configuration of the in-depth analysis module is designed to calculate the deviation degree of user behavior by comparing user behavior data with the historical behavior baseline, so as to provide in-depth insights into changes in user behavior patterns. First, the system calculates the average value of the total number of normal logins, the average value of the number of normal login errors, and the average value of the normal login frequency within a preset historical period. These metrics constitute a benchmark for evaluating user behavior. Through these historical data, the system can establish a normal pattern of user behavior, providing a reference for subsequent deviation analysis. Subsequently, the system calculates the deviation degree of user behavior through a specific formula, which comprehensively considers three dimensions: the total number of logins, the number of login errors, and the login frequency. By calculating the difference between the current behavior and the historical baseline, the system can quantify the degree of change in user behavior. This calculation of the deviation degree helps to promptly detect abnormal patterns of user behavior, such as an abnormal increase in the number of login attempts, an abnormal increase in the number of login errors, or a significant change in the login frequency. These may all be signals of security risks or changes in user habits. Generally speaking, through this in-depth analysis, the system can provide a more personalized and secure service experience for users. For service providers, this analysis helps to promptly adjust service strategies, optimize the user experience, and take security measures when necessary to prevent potential risks. For users, this analysis can ensure the security of their accounts while providing services that are more in line with their behavior habits. Therefore, the implementation of this in-depth analysis module not only improves the intelligence level of the system but also brings substantial benefits to users and service providers.

[0104] In some embodiments of the present application, when the in-depth analysis module is configured to calculate the network anomaly probability value based on the device anomaly index and the deviation degree, it includes:

[0105] Calculating the sum value of the device anomaly index and the deviation degree of user behavior;

[0106] Setting a first sum value and a second sum value, where the first sum value is less than the second sum value;

[0107] If the sum value is less than the first sum value, the network anomaly probability value is the first probability;

[0108] If the sum value is greater than or equal to the first sum value and less than or equal to the second sum value, the network anomaly probability value is the second probability;

[0109] If the sum value is greater than the second sum value, the network anomaly probability value is the third probability;

[0110] Wherein, the value range of the network probability value is 0 < the first probability < the second probability < the third probability < 1.

[0111] It is understandable that the in-depth analysis module in this application obtains the network anomaly probability value by comprehensively calculating the device anomaly index and the user behavior deviation degree, providing a refined network anomaly detection method. By setting different value ranges (the first sum value and the second sum value), the system can perform hierarchical probability assessment according to different degrees of anomalies. Specifically, if the sum of the two is less than the first set value, the network anomaly probability value is the lowest first probability, indicating that the network condition is relatively normal; if the sum value is between the first sum value and the second sum value, a medium second probability is given; if the sum value is greater than the second sum value, the probability of network anomaly rises to the higher third probability. First, by hierarchically setting different probability values, the system can dynamically adjust the alarm strategy according to the severity of the anomaly, improving the flexibility and adaptability of detection. Second, with the comprehensive evaluation of the device anomaly index and the user behavior deviation degree, potential network threats or security vulnerabilities can be more comprehensively identified, avoiding the limitations of traditional methods that rely solely on a single indicator. In addition, this hierarchical probability method can effectively reduce false alarms and missed alarms in practical applications, ensuring that the alarm system can respond timely under different abnormal situations. For example, when the network state is relatively stable, the low probability judgment can prevent unnecessary alarms, while in the case of severe anomalies, the higher probability value can trigger high-level alarms in a timely manner to ensure network security.

[0112] In some embodiments of the present application, when the alarm module is configured to determine whether to perform an anomaly alarm according to the network anomaly probability value, it includes:

[0113] If the network anomaly probability value is the first probability, it is determined not to perform an anomaly alarm;

[0114] If the network anomaly probability value is not the first probability, it is determined to perform an anomaly alarm.

[0115] In some embodiments of the present application, when the alarm module is configured to determine the alarm level according to the network anomaly probability value, it includes:

[0116] If the network anomaly probability value is the second probability, the alarm level is level one;

[0117] If the network anomaly probability value is the third probability, the alarm level is level two; where level one in the alarm level is lower than level two;

[0118] When the alarm level is level one, increase the duration of the acquisition time period so that the increased duration of the acquisition time period is 1.5 times the current duration of the acquisition time period;

[0119] When the alarm level is level two, increase the duration of the acquisition time period so that the increased duration of the acquisition time period is 2 times the current duration of the acquisition time period.

[0120] It can be understood that when the probability of network anomalies is low, the system determines not to issue an alarm, thus avoiding false alarms caused by minor anomalies and reducing resource waste and manual intervention. However, as the anomaly probability increases, the system adjusts the alarm level to increase the duration of the acquisition time period, so as to deeply analyze the anomaly phenomenon and timely capture more complex network security events. Specifically, if the alarm level is level one, it indicates that the possibility of network anomalies is low or the anomalies are minor, and the system will extend the acquisition time period to 1.5 times the current one for further confirmation of the anomaly trend and root cause; when the alarm level is level two, it indicates that the anomalies are relatively serious, and the system will increase the data acquisition intensity and extend the acquisition time period to 2 times the current one. This mechanism can provide more accurate and comprehensive anomaly detection results for network monitoring. By extending the acquisition time, the system can accumulate more anomaly data and behavior patterns, reduce the probability of misjudgment, and further improve the efficiency and reliability of network security protection. This intelligent adjustment mechanism not only improves the flexibility of the system, but also ensures that in the face of different levels of anomalies, it can respond in a timely manner according to the actual situation, ensuring that alarm handling is more accurate, fast and efficient.

[0121] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or multiple flows and / or blocks

[0123] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one or more of the flowsFigure 1 The functions specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps of the functions specified in one Figure 1 process or more processes and / or boxes Figure 1 or more boxes.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent replacements can still be made to the specific implementation manners of the present invention. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A network-aware anomaly detection system based on big data, characterized in that: include: The acquisition module is configured to intercept network data within a preset period of time, classify the network data, and obtain data of different categories; wherein the different categories of data include network flow data, device data, and user behavior data; the network flow data includes flow characteristics and connection characteristics; the device data includes the number of interface error packets and configuration change information; the user behavior data includes the total number of logins, the number of login errors, and the login frequency; a preliminary analysis module configured to extract flow characteristics and connection characteristics of the network flow data, calculate a flow stability value according to the flow characteristics, calculate a connection stability value according to the connection characteristics, respectively calculate a flow deviation value between the flow stability value and a preset flow stability value and a connection deviation value between the connection stability value and a preset connection stability value, respectively preliminarily determine whether an abnormality occurs in the network according to the flow deviation value and the connection deviation value, and obtain a comprehensive preliminary determination result of the network by combining the two preliminary determination results; The deep analysis module is configured to calculate the device anomaly index through the number of interface error packets and configuration change information if the comprehensive preliminary judgment result is that the network is abnormal; calculate the deviation of user behavior based on the user behavior data and the historical behavior baseline, and calculate the network anomaly probability value based on the device anomaly index and the deviation; The alarm module is configured to determine whether to issue an abnormality alarm according to the network abnormality probability value, and determine an alarm level according to the network abnormality probability value.

2. The network perception anomaly detection system based on big data according to claim 1 is characterized in that: The acquisition module is configured to intercept network data within a preset period of time, classify the network data, and obtain data of different categories, including: Preset the collection time period to clean the collected network data and remove duplicate, invalid and erroneous data; Dividing the network data into network traffic data, device data and user behavior data according to data types, and storing the network data separately according to different categories and collection time periods; Construct a storage sequence At = [L (l 1, l 2, l 3, ..., ln), S (s1, s2, s3, ..., sn) Y (y1, y2, y3, ..., yn)]; Among them, At represents the storage sequence with serial number A and time period t, L represents network traffic data, li represents the i-th network traffic data, i=1, 2, 3, ..., n; S represents device data, sj represents the j-th device data, j=1, 2, 3, ..., n; Y represents user behavior data, yk represents the k-th user behavior data, k=1, 2, 3, ..., n.

3. The network perception anomaly detection system based on big data according to claim 2 is characterized in that: The preliminary analysis module is configured to extract the flow characteristics and connection characteristics of the network flow data, calculate the flow stability value according to the flow characteristics, calculate the connection stability value according to the connection characteristics, and respectively calculate the flow deviation value between the flow stability value and the preset flow stability value and the connection deviation value between the connection stability value and the preset connection stability value, including: The traffic characteristics include total traffic, traffic rate and traffic fluctuation value; the connection characteristics include the number of new connections, connection failure rate and connection duration; The traffic stability value is calculated based on the total traffic, traffic rate and traffic fluctuation value, and the connection stability value is calculated based on the number of new connections, connection failure rate and connection duration; The average value of the historical normal total flow, the average value of the historical normal flow rate, and the average value of the historical normal flow fluctuation value in the historical time period that is the same as the preset collection time period are obtained, and the flow stability value is calculated by the following formula: W1=a1*(T1 / T2)+a2*(R1 / R2)+a3*(V1 / V2); In the above formula, W1 represents the stable flow value, T1 represents the total flow, T2 represents the historical normal total flow average, R1 represents the flow rate, R2 represents the historical normal flow rate average, V1 represents the flow fluctuation value, V2 represents the historical normal flow fluctuation value average, a1, a2 and a3 represent the weight coefficients respectively, wherein the value ranges of a1, a2 and a3 are all [0, 1], and a1+a2+a3=1; The average value of the number of historical normal new connections, the average value of the historical normal connection failure rate, and the average value of the historical normal connection duration in the historical time period that is the same as the preset collection time period are obtained, and the connection stability value is calculated by the following formula: W2=b1*(X1 / X2)+b2*(B1 / B2)+b3*(C1 / C2); In the above formula, W2 represents the connection stability value, X1 represents the number of new connections, X2 represents the historical average number of normal new connections, B1 represents the connection failure rate, B2 represents the historical average value of normal connection failure rate, C1 represents the connection duration, C2 represents the historical average value of normal connection duration, b1, b2 and b3 represent weight coefficients respectively, where the value ranges of b1, b2 and b3 are all [0, 1], and b1+b2+b3=1.

4. The network perception anomaly detection system based on big data according to claim 3 is characterized in that: The preliminary analysis module is configured to preliminarily determine whether an abnormality occurs in the network according to the flow deviation value and the connection deviation value, including: Preset maximum flow deviation and maximum connection deviation; Calculating a flow deviation value between a flow stability value and a preset flow stability value, and calculating a connection deviation value between a connection stability value and a preset connection stability value; If the traffic deviation value is greater than or equal to the maximum traffic deviation value, it is preliminarily determined that the network is abnormal; otherwise, it is preliminarily determined that the network is not abnormal; If the connection deviation value is greater than or equal to the preset traffic stability value, it is preliminarily determined that the network is abnormal; otherwise, it is preliminarily determined that the network is not abnormal.

5. The network perception anomaly detection system based on big data according to claim 4 is characterized in that: When the preliminary analysis module is configured to combine the two preliminary judgment results to obtain a comprehensive preliminary judgment result of the network, it includes: If both preliminary judgment results show that the network is normal, the comprehensive preliminary judgment result of the network is that the network status is normal, and no abnormal alarm is issued; If one of the two preliminary judgment results indicates that a network abnormality has occurred, the comprehensive preliminary judgment result of the network is that a network abnormality has occurred.

6. The network perception anomaly detection system based on big data according to claim 5 is characterized in that: The in-depth analysis module is configured to, if the comprehensive preliminary judgment result is that the network is abnormal, calculate the device abnormality index through the number of interface error packets and configuration change information, including: Collect the average number of historical normal interface error packets and the average number of historical normal configuration changes, and calculate the device abnormality index using the following formula: In the above formula, E represents the device abnormality index, n represents the time window length, and Cb i represents the number of interface error packets in the i-th period, Cb represents the average number of historical normal interface error packets, Bc i represents the number of configuration changes in the i-th period, and Bc represents the average number of historical normal configuration changes.

7. The network perception anomaly detection system based on big data according to claim 6 is characterized in that: When the in-depth analysis module is configured to calculate the deviation of the user behavior based on the user behavior data and the historical behavior baseline, it includes: Calculate the average total number of normal logins, the average number of normal login errors, and the average normal login frequency within a historical preset period based on historical user behavior data, and use the average total number of normal logins, the average number of normal login errors, and the average normal login frequency within the historical preset period as a historical behavior baseline; The deviation of user behavior is calculated by the following formula: In the above formula, PL represents the deviation of user behavior, D represents the total number of logins, D0 represents the average number of normal logins, E represents the number of login errors, E0 represents the average number of normal login errors, P represents the login frequency, and P0 represents the average number of normal login frequencies.

8. The network perception anomaly detection system based on big data according to claim 7 is characterized in that: When the in-depth analysis module is configured to calculate the network anomaly probability value according to the device anomaly index and the deviation, it includes: Calculate the sum of the device anomaly index and the deviation of user behavior; Setting a first sum value and a second sum value, wherein the first sum value is smaller than the second sum value; If the sum is less than the first sum, the network anomaly probability value is the first probability; If the sum is greater than or equal to the first sum, and less than or equal to the second sum, the network abnormality probability value is the second probability; If the sum is greater than the second sum, the network anomaly probability value is a third probability; The value range of the network probability value is 0<first probability<second probability<third probability<1.

9. The network perception anomaly detection system based on big data according to claim 8 is characterized in that: When the alarm module is configured to determine whether to issue an abnormality alarm according to the network abnormality probability value, it includes: If the network abnormality probability value is the first probability, it is determined that no abnormality alarm is issued; If the network abnormality probability value is not the first probability, it is determined to issue an abnormality alarm.

10. The network perception anomaly detection system based on big data according to claim 9 is characterized in that: When the alarm module is configured to determine the alarm level according to the network abnormality probability value, it includes: If the network abnormality probability value is the second probability, the alarm level is level one; If the network abnormality probability value is the third probability, the alarm level is level 2; wherein level 1 is lower than level 2; When the alarm level is level one, the length of the collection time period is increased so that the length of the increased collection time period is 1.5 times the length of the current collection time period; When the alarm level is level 2, the length of the collection time period is increased so that the length of the increased collection time period is twice the length of the current collection time period.

Images