AI-based network security anomaly detection system and method

Abstract

The present application relates to the field of network security technology, and discloses an AI-based network security anomaly detection system and method, the system comprising an anomaly probability selection module, a dynamic anomaly feature generation module, and a full data anomaly detection module that are sequentially connected in communication. The method corresponds to the system. In the present application, the anomaly probability selection module calculates the anomaly probability for log data and a variety of non-log data respectively and selects the maximum value as the dynamic anomaly probability, thereby realizing the rapid location of the data with the most abnormal probability from multi-dimensional data; the dynamic anomaly feature generation module extracts features based on the data source corresponding to the dynamic anomaly probability, and merges it with the corresponding probability features, effectively integrating real-time data features and historical experience features; the full data anomaly detection module matches time series data and combines it into full data for detection, and uses dynamic anomaly features to output the results of full data detection, thereby comprehensively and accurately realizing network security anomaly detection.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and specifically to an AI-based network security anomaly detection system and method. Background Art

[0002] In the field of network security, it is extremely important to ensure the stability and security of the network. Traditional network security detection methods have gradually exposed many problems in the face of increasingly complex network environments.

[0003] The inventors found in their research on the prior art that the prior art mostly focuses on detecting a single type of data, such as some that rely solely on log data. Chinese Patent No. CN117544391B discloses a network security detection method and system based on log data. Although the invention can process and analyze log data to a certain extent, it lacks effective integration and collaborative detection of non-log data. Its detection accuracy is limited by the update speed and range of log standard information, and it is difficult to cope with new types of attacks. In terms of system performance, efficiency bottlenecks are prone to occur when processing large-scale log data, and it lacks adaptability to different network environments.

[0004] In practical applications, network security anomaly detection requires the integration of multi-dimensional data. Abnormal fluctuations in network traffic data, sudden changes in system performance data, deviations in user behavior data, warnings from security equipment, and changes in network topology data may all indicate security issues. Relying solely on single log data cannot fully capture these potential risks, making it difficult to achieve efficient and accurate network security anomaly detection.

[0005] In summary, there is an urgent need for a new technical solution for network security anomaly detection that integrates multiple data types and is based on AI to solve the above technical problems. Summary of the invention

[0006] The purpose of this application is to provide an AI-based network security anomaly detection system and method to solve the technical problems raised in the above background technology.

[0007] To achieve the above objectives, this application discloses the following technical solutions:

[0008] In a first aspect, the present application discloses an AI-based network security anomaly detection system, characterized in that the system comprises an anomaly probability selection module, a dynamic anomaly feature generation module, and a full data anomaly detection module that are sequentially connected in communication;

[0009] The abnormal probability selection module is configured as follows: using a preset abnormal probability feature database to calculate the abnormal probability of log data and non-log data respectively, generate log abnormal probability and non-log abnormal probability, output the maximum value of the log abnormal probability and the non-log abnormal probability, and define the maximum value as a dynamic abnormal probability; wherein the abnormal probability feature database stores probability features representing the probability of abnormality of log data and the probability of abnormality of non-log data, the probability features are used to represent features corresponding to the probability of abnormality, the log data and the non-log data are operation data of different dimensions for representing network security conditions, the non-log data at least includes network traffic data, system performance data, user behavior data, security equipment alarm data and network topology data, the log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on log data, the non-log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on non-log data, and the network security probability detection is a probabilistic network security detection;

[0010] The dynamic abnormal feature generation module is configured to: use a preset abnormal feature database to extract features from the log data or non-log data corresponding to the dynamic abnormal probability to generate real-time abnormal features, combine the abnormal probability feature database, fuse the real-time abnormal features with the corresponding probability features to generate dynamic abnormal features and output; wherein the abnormal feature database stores abnormal features corresponding to log data and non-log data, and the dynamic abnormal feature is used to characterize the features that fuse the real-time abnormal features and the corresponding probability features;

[0011] The full data anomaly detection module is configured as follows: matching the log data or non-log data corresponding to the dynamic anomaly probability with the log data or non-log data corresponding to the time series, combining the log data and the non-log data to generate full data, performing network security anomaly detection on the full data using the dynamic anomaly feature and outputting the corresponding detection result; wherein the full data is the log data and non-log data corresponding to the dynamic anomaly probability.

[0012] Preferably, the generation of the log anomaly probability includes:

[0013] The log anomaly probability is calculated using a preset log anomaly probability calculation formula, which is:

[0014]

[0015] in, Based on log data The total number of features obtained, For log data The number of features with abnormal probability obtained based on the abnormal probability feature database, For log data Features with abnormal probability , For preset log data Features with abnormal probability The weight parameter, is the calculated log anomaly probability.

[0016] Preferably, the generation of the non-log anomaly probability includes:

[0017] The non-log anomaly probability is calculated using a preset non-log anomaly probability calculation formula, and the non-log anomaly probability calculation formula is:

[0018]

[0019] in, Based on non-log data The total number of features obtained, For non-log data The number of features with abnormal probability obtained based on the abnormal probability feature database, For non-log data Features with abnormal probability , For preset non-log data Features with abnormal probability The weight parameter, is the calculated log anomaly probability.

[0020] Preferably, the generation of the real-time abnormal features includes:

[0021] When the dynamic anomaly probability is the log anomaly probability, extracting features from the log data using the anomaly feature database to generate real-time anomaly features of the log;

[0022] When the dynamic abnormality probability is the non-log abnormality probability, extracting features from the non-log data using the abnormality feature database to generate non-log real-time abnormality features;

[0023] The real-time abnormality feature includes the log real-time abnormality feature and the non-log real-time abnormality feature.

[0024] Preferably, the generation of the dynamic abnormality feature includes:

[0025] When the real-time abnormal feature of the log exists, the probability feature corresponding to the non-log data is matched, and the real-time abnormal feature of the log is fused with the probability feature corresponding to the matched non-log data to generate a dynamic abnormal feature of the log;

[0026] When the non-log real-time abnormal feature exists, the probability feature corresponding to the log data is matched, and the non-log real-time abnormal feature is fused with the probability feature corresponding to the matched log data to generate a non-log dynamic abnormal feature;

[0027] The dynamic abnormality features include the log dynamic abnormality features and the non-log dynamic abnormality features.

[0028] Preferably, the feature fusion includes:

[0029] Corresponding weights are assigned to the real-time abnormal feature and its corresponding probability feature, the weight of the real-time abnormal feature is the maximum value of the log abnormality probability and the non-log abnormality probability, and the weight of the probability feature is the minimum value of the log abnormality probability and the non-log abnormality probability. Feature fusion calculation is performed based on the assigned weights to obtain the dynamic abnormal feature.

[0030] Preferably, the generation of the full data includes:

[0031] When the dynamic abnormal probability is the log abnormal probability, the log data is matched with the non-log data corresponding to the time series, and the log data and the non-log data are combined to generate the full log data;

[0032] When the dynamic abnormal probability is the non-log abnormal probability, the log data corresponding to the time series is matched by using the non-log data, and the log data and the non-log data are combined to generate non-log full data;

[0033] The full data includes the log full data and the non-log full data.

[0034] Preferably, the generation of the full data further comprises:

[0035] When the full data is the log full data, the log full data is combined using the log full data combination formula, and the log full data combination formula is:

[0036]

[0037] in, is the log anomaly probability, For the log data, For the non-log data, The full log data obtained after weighted combination;

[0038] When the full data is the non-log full data, the non-log full data is combined using a non-log full data combination formula, and the non-log full data combination formula is:

[0039]

[0040] in, is the non-log anomaly probability, Non-logged full data after weighted combination.

[0041] Preferably, the system further comprises a data updating module which is in communication connection with the abnormal probability selection module;

[0042] The data update module is configured to calculate the updated probability feature using the initial probability feature, the feature expectation value obtained by statistical analysis based on the log anomaly probability or the non-log anomaly probability, and a preset update ratio.

[0043] In a second aspect, the present application discloses an AI-based network security anomaly detection method, which is applicable to the AI-based network security anomaly detection system as described above, and the method comprises:

[0044] The abnormal probability of log data and non-log data is calculated respectively by using a preset abnormal probability feature database to generate a log abnormal probability and a non-log abnormal probability, and the maximum value of the log abnormal probability and the non-log abnormal probability is output, and the maximum value is defined as a dynamic abnormal probability; wherein the abnormal probability feature database stores probability features representing the probability of abnormality of log data and the probability of abnormality of non-log data, and the probability features are used to represent features corresponding to the probability of abnormality, the log data and the non-log data are operation data of different dimensions for representing network security conditions, and the non-log data at least includes network traffic data, system performance data, user behavior data, security equipment alarm data and network topology data, the log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on log data, and the non-log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on non-log data, and the network security probability detection is a probabilistic network security detection;

[0045] The preset abnormal feature database is used to extract features from the log data or non-log data corresponding to the dynamic abnormal probability to generate real-time abnormal features, and the real-time abnormal features are combined with the corresponding probability features to generate dynamic abnormal features and output; wherein the abnormal feature database stores abnormal features corresponding to the log data and non-log data, and the dynamic abnormal features are used to characterize the features of the fusion of the real-time abnormal features and the corresponding probability features;

[0046] Match the log data or non-log data corresponding to the dynamic anomaly probability with the log data or non-log data corresponding to the time series, combine the log data and the non-log data to generate full data, use the dynamic anomaly feature to perform network security anomaly detection on the full data and output the corresponding detection result; wherein the full data is the log data and non-log data corresponding to the dynamic anomaly probability.

[0047] Beneficial effects: The AI-based network security anomaly detection system and method of the present application calculates the anomaly probability for log data and various non-log data respectively through the anomaly probability selection module and selects the maximum value as the dynamic anomaly probability, thereby realizing the rapid location of the data with the most abnormal possibility from multi-dimensional data; the dynamic anomaly feature generation module extracts features based on the data source corresponding to the dynamic anomaly probability, and merges them with the corresponding probability features, effectively integrating real-time data features and historical experience features; the full data anomaly detection module matches time series data and combines them into full data for detection, and uses dynamic anomaly features to output results for full data detection, thereby comprehensively and accurately realizing network security anomaly detection, greatly improving the accuracy and reliability of detection, and effectively responding to complex and changeable network security threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A structural block diagram of an AI-based network security anomaly detection system provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of an AI-based network security anomaly detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0052] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0053] The first aspect of this embodiment discloses Figure 1 The AI-based network security anomaly detection system shown is characterized in that the system includes an anomaly probability selection module, a dynamic anomaly feature generation module and a full data anomaly detection module that are sequentially connected in communication;

[0054] The abnormal probability selection module is configured as follows: using a preset abnormal probability feature database to calculate the abnormal probability of log data and non-log data respectively, generate log abnormal probability and non-log abnormal probability, output the maximum value of the log abnormal probability and the non-log abnormal probability, and define the maximum value as the dynamic abnormal probability; wherein the abnormal probability feature database stores probability features representing the probability of abnormality of log data and the probability of abnormality of non-log data, the probability features are used to represent the features corresponding to the probability of abnormality, the log data and non-log data are operation data of different dimensions used to represent the network security situation, the non-log data at least includes network traffic data, system performance data, user behavior data, security equipment alarm data and network topology data, the log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on log data, the non-log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on non-log data, and the network security probability detection is to perform probabilistic network security detection;

[0055] The dynamic abnormal feature generation module is configured as follows: using a preset abnormal feature database to extract features from log data or non-log data corresponding to the dynamic abnormal probability to generate real-time abnormal features, combining the abnormal probability feature database, and fusing the real-time abnormal features with the corresponding probability features to generate dynamic abnormal features and output them; wherein the abnormal feature database stores abnormal features corresponding to log data and non-log data, and the dynamic abnormal features are used to characterize the features of the fusion of real-time abnormal features and corresponding probability features;

[0056] The full data anomaly detection module is configured as follows: matching the log data or non-log data corresponding to the dynamic anomaly probability with the log data or non-log data corresponding to the time series, combining the log data and the non-log data to generate full data, using dynamic anomaly features to perform network security anomaly detection on the full data and outputting the corresponding detection results; wherein the full data is the log data and non-log data corresponding to the dynamic anomaly probability.

[0057] Based on the above, this embodiment calculates the abnormal probability of log data and various non-log data respectively and selects the maximum value as the dynamic abnormal probability through the abnormal probability selection module, thereby realizing the rapid location of the data with the highest abnormal probability from multi-dimensional data; the dynamic abnormal feature generation module extracts features according to the data source corresponding to the dynamic abnormal probability, and merges them with the corresponding probability features, effectively integrating real-time data features and historical experience features; the full data anomaly detection module matches time series data and combines them into full data for detection, and uses dynamic abnormal features to output the results of full data detection, thereby comprehensively and accurately realizing network security anomaly detection, greatly improving the accuracy and reliability of detection, and effectively responding to complex and changeable network security threats.

[0058] Specifically, the generation of log anomaly probability includes:

[0059] The log anomaly probability is calculated using the preset log anomaly probability calculation formula. The log anomaly probability calculation formula is:

[0060]

[0061] in, Based on log data The total number of features obtained, For log data The number of features with abnormal probability obtained based on the abnormal probability feature database, For log data Features with abnormal probability , For preset log data Features with abnormal probability The weight parameter, is the calculated log anomaly probability.

[0062] Based on the above, this embodiment uses the log anomaly probability calculation formula to achieve accurate quantitative analysis based on log data features. By setting and calculating the features of the anomaly probability in the log data and its weight parameters, it is possible to systematically sort out clues to security risks in the log data, so that the calculation of the anomaly probability of the log data has a clear mathematical basis and no longer relies on vague empirical judgments, thereby improving the scientificity and accuracy of network security anomaly detection based on log data, providing reliable basic data support for subsequent dynamic anomaly probability determination and overall system detection, and enhancing the system's ability to capture security issues reflected in log data.

[0063] Specifically, the generation of non-log anomaly probability includes:

[0064] The non-log anomaly probability is calculated using the preset non-log anomaly probability calculation formula. The non-log anomaly probability calculation formula is:

[0065]

[0066] in, Based on non-log data The total number of features obtained, For non-log data The number of features with abnormal probability obtained based on the abnormal probability feature database, For non-log data Features with abnormal probability , For preset non-log data Features with abnormal probability The weight parameter, is the calculated log anomaly probability.

[0067] Based on the above, this embodiment uses the non-log anomaly probability calculation formula to achieve effective analysis of non-log data and quantification of anomaly probability. For various non-log data types such as network traffic data and system performance data, calculations are performed based on their respective characteristics and weight parameters, fully mining the abnormal information in different types of non-log data, so that the system can comprehensively consider various security factors in network operation, avoid the limitations of single data type detection, improve the utilization value of non-log data in network security anomaly detection, provide richer and more comprehensive information for accurately judging the network security status, and improve the overall detection capability and adaptability of the system.

[0068] Specifically, the generation of real-time abnormal features includes:

[0069] When the dynamic anomaly probability is the log anomaly probability, the log data is feature extracted using the anomaly feature database to generate real-time anomaly features of the log;

[0070] When the dynamic anomaly probability is a non-log anomaly probability, the anomaly feature database is used to extract features from the non-log data to generate non-log real-time anomaly features;

[0071] Real-time abnormality features include log real-time abnormality features and non-log real-time abnormality features.

[0072] Based on the above, this embodiment uses the abnormal feature database and the judgment of dynamic abnormal probability to achieve targeted real-time abnormal feature extraction. When the dynamic abnormal probability corresponds to log data, the log real-time abnormal feature is extracted; when it corresponds to non-log data, the non-log real-time abnormal feature is extracted. This method ensures that the extracted features are closely related to the data that is most likely to have abnormalities, improves the efficiency and effectiveness of feature extraction, reduces unnecessary waste of computing resources, and lays the foundation for the subsequent accurate generation of dynamic abnormal features, so that the system can quickly focus on potential security threat points and enhance the ability to identify abnormal situations.

[0073] Specifically, the generation of dynamic anomaly features includes:

[0074] When there are real-time abnormal features of logs, the probability features corresponding to non-log data are matched, and the real-time abnormal features of logs are fused with the probability features corresponding to the matched non-log data to generate dynamic abnormal features of logs;

[0075] When there are non-log real-time abnormal features, the probability features corresponding to the log data are matched, and the non-log real-time abnormal features are fused with the probability features corresponding to the matched log data to generate non-log dynamic abnormal features;

[0076] Dynamic anomaly features include log dynamic anomaly features and non-log dynamic anomaly features.

[0077] It should be noted that, different from the existing fixed standard network security detection, this embodiment uses dynamic anomaly features to provide dynamic detection standards for network security detection. Furthermore, different from the existing single-dimensional dynamic detection standards, this embodiment uses log dynamic anomaly features and non-log dynamic anomaly features to provide multi-dimensional dynamic detection standards.

[0078] Based on the above, this embodiment uses feature matching and fusion mechanisms to achieve the optimized generation of dynamic abnormal features. According to the source of real-time abnormal features (log data or non-log data), the corresponding probability features of another type of data are matched for fusion. Through this cross-data type feature fusion, the advantages of different data dimensions are fully combined, and the information content of dynamic abnormal features is enriched, so that it can more accurately reflect the true state of network security. Whether it is log dynamic abnormal features or non-log dynamic abnormal features, they integrate multiple aspects of key information, improve the system's ability to distinguish complex network security anomalies, and effectively reduce the probability of false alarms and missed reports.

[0079] Specifically, feature fusion includes:

[0080] Corresponding weights are assigned to the real-time anomaly features and their corresponding probability features. The weight of the real-time anomaly feature is the maximum value of the log anomaly probability and the non-log anomaly probability, and the weight of the probability feature is the minimum value of the log anomaly probability and the non-log anomaly probability. Feature fusion calculation is performed based on the assigned weights to obtain dynamic anomaly features.

[0081] As a preferred implementation of this embodiment, feature fusion is performed using a preset feature fusion formula, and the feature fusion formula is:

[0082]

[0083] in, is the maximum value of the log anomaly probability and the non-log anomaly probability, For real-time abnormal features, is the minimum value of the log anomaly probability and the non-log anomaly probability, Real-time abnormal features The corresponding probability characteristics are It is a dynamic abnormal feature.

[0084] Based on the above, this embodiment uses the feature fusion formula to achieve efficient fusion of real-time abnormal features and probabilistic features. The formula performs weighted combination according to the size relationship between the log abnormal probability and the non-log abnormal probability and the corresponding features, and reasonably allocates the weights of different features in the dynamic abnormal features. This scientific fusion method enables the generated dynamic abnormal features to reflect the dynamic changes of real-time data while taking into account the probabilistic laws reflected by historical data, improves the representativeness and reliability of dynamic abnormal features, provides a more discriminative feature basis for full data anomaly detection, and enhances the system's detection accuracy of network security anomalies.

[0085] Specifically, the generation of full data includes:

[0086] When the dynamic anomaly probability is a log anomaly probability, the log data is used to match the non-log data corresponding to the time series, and the log data and the non-log data are combined to generate the full log data;

[0087] When the dynamic anomaly probability is a non-log anomaly probability, the non-log data is used to match the log data corresponding to the time series, and the log data and the non-log data are combined to generate non-log full data;

[0088] Full data includes log full data and non-log full data.

[0089] Through the above, this embodiment uses the method of time series matching and data combination to achieve the effective generation of full data. According to the source of dynamic anomaly probability, the log data is accurately combined with the corresponding non-log data to ensure the integrity and relevance of the full data. In this way, the system can comprehensively consider the information of different data dimensions at the same time point or time period, avoiding the one-sidedness caused by isolated data analysis, providing sufficient data support for comprehensive and accurate detection of network security anomalies, improving the system's comprehensive judgment ability on the network security status, and enhancing the credibility of the detection results.

[0090] Specifically, the generation of full data also includes:

[0091] When the full data is log full data, the log full data combination formula is used to combine the log full data. The log full data combination formula is:

[0092]

[0093] in, is the log anomaly probability, For log data, For non-log data, The full log data obtained after weighted combination;

[0094] When the full data is non-log full data, the non-log full data combination formula is used to combine the non-log full data. The non-log full data combination formula is:

[0095]

[0096] in, is the non-log anomaly probability, Non-logged full data after weighted combination.

[0097] Based on the above, this embodiment uses a specific full data combination formula to achieve a reasonable weighted combination of all data. Different formulas are used for log full data and non-log full data, and the data is weighted according to their respective abnormal probabilities, highlighting the role of data that is more likely to have abnormalities in the full data. This weighted combination method makes the full data more targeted when performing anomaly detection, can better reflect the actual situation of network security, improves the effectiveness of the full data in the detection process, and further improves the system's detection performance and accuracy of network security anomalies.

[0098] Specifically, the system also includes a data updating module that is communicatively connected to the abnormal probability selection module;

[0099] The data update module is configured to calculate the updated probability feature using the initial probability feature, the feature expectation value obtained by statistical analysis based on the log anomaly probability or the non-log anomaly probability, and the preset update ratio.

[0100] As a preferred implementation of this embodiment, the probability feature is updated using a probability feature update formula, and the probability feature update formula is:

[0101]

[0102] in, is the initial probability feature, is the expected value of the feature obtained by statistical analysis based on the log anomaly probability or non-log anomaly probability, is the update ratio and , is the updated probability feature.

[0103] Based on the above, this embodiment uses the probability feature update formula and data update module to realize the dynamic update of the probability feature. The feature expectation value is obtained by statistical analysis based on the log abnormality probability or non-log abnormality probability, and the initial probability feature is updated according to a certain update ratio. This enables the system to adapt to changes in the network environment and emerging security threats in a timely manner, ensures that the probability features based on the abnormal probability calculation and detection process are always timely and accurate, continuously improves the system's detection capabilities and adaptability to complex network security situations, and maintains the system's efficiency and reliability in the field of network security detection.

[0104] The second aspect of this embodiment discloses Figure 2 An AI-based network security anomaly detection method is shown, which is applicable to the above AI-based network security anomaly detection system, and the method includes:

[0105] The abnormal probability of log data and non-log data is calculated respectively by using a preset abnormal probability feature database to generate log abnormal probability and non-log abnormal probability, and the maximum value of the log abnormal probability and non-log abnormal probability is output, and the maximum value is defined as the dynamic abnormal probability; wherein the abnormal probability feature database stores probability features representing the probability of abnormality of log data and the probability of abnormality of non-log data, and the probability features are used to represent the features corresponding to the probability of abnormality, and the log data and non-log data are operation data of different dimensions used to represent the network security situation, and the non-log data at least includes network traffic data, system performance data, user behavior data, security equipment alarm data and network topology data, and the log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on log data, and the non-log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on non-log data, and the network security probability detection is to perform probabilistic network security detection;

[0106] The preset abnormal feature database is used to extract features from log data or non-log data corresponding to the dynamic abnormal probability to generate real-time abnormal features. Combined with the abnormal probability feature database, the real-time abnormal features are fused with the corresponding probability features to generate dynamic abnormal features and output them; wherein the abnormal feature database stores abnormal features corresponding to log data and non-log data, and the dynamic abnormal features are used to characterize the features of the fused real-time abnormal features and the corresponding probability features;

[0107] Match the log data or non-log data corresponding to the dynamic anomaly probability with the log data or non-log data corresponding to the time series, combine the log data and the non-log data to generate full data, use the dynamic anomaly features to perform network security anomaly detection on the full data and output the corresponding detection results; wherein the full data is the log data and non-log data corresponding to the dynamic anomaly probability.

[0108] It should be noted that the AI-based network security anomaly detection method of this embodiment corresponds to the aforementioned AI-based network security anomaly detection system. Therefore, the contents not specifically described in the AI-based network security anomaly detection method of this embodiment may include but are not limited to functional definitions, working principles, and technical effects, etc., and may refer to the records of the aforementioned AI-based network security anomaly detection system, and this text will not elaborate on them here.

[0109] In summary, the AI-based network security anomaly detection system and method of this embodiment calculates the anomaly probability for log data and various non-log data respectively through the anomaly probability selection module and selects the maximum value as the dynamic anomaly probability, thereby realizing the rapid location of the data with the highest probability of anomaly from multi-dimensional data; the dynamic anomaly feature generation module extracts features based on the data source corresponding to the dynamic anomaly probability, and integrates them with the corresponding probability features, effectively integrating real-time data features and historical experience features; the full data anomaly detection module matches time series data and combines them into full data for detection, and uses dynamic anomaly features to output results for full data detection, thereby comprehensively and accurately realizing network security anomaly detection, greatly improving the accuracy and reliability of detection, and effectively responding to complex and changeable network security threats.

[0110] In the embodiments provided in the present application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, other electronic units designed to implement the functions described herein, or a combination thereof. For software implementation, part or all of the processes of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a computer. Computer-readable storage media can include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer.

[0111] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An AI-based network security anomaly detection system, characterized in that: The system includes an abnormal probability selection module, a dynamic abnormal feature generation module and a full data abnormality detection module which are sequentially connected in communication; The abnormal probability selection module is configured as follows: using a preset abnormal probability feature database to calculate the abnormal probability of log data and non-log data respectively, generate log abnormal probability and non-log abnormal probability, output the maximum value of the log abnormal probability and the non-log abnormal probability, and define the maximum value as a dynamic abnormal probability; wherein the abnormal probability feature database stores probability features representing the probability of abnormality of log data and the probability of abnormality of non-log data, the probability features are used to represent features corresponding to the probability of abnormality, the log data and the non-log data are operation data of different dimensions for representing network security conditions, the non-log data at least includes network traffic data, system performance data, user behavior data, security equipment alarm data and network topology data, the log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on log data, the non-log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on non-log data, and the network security probability detection is a probabilistic network security detection; The dynamic abnormal feature generation module is configured to: use a preset abnormal feature database to extract features from the log data or non-log data corresponding to the dynamic abnormal probability to generate real-time abnormal features, combine the abnormal probability feature database, fuse the real-time abnormal features with the corresponding probability features to generate dynamic abnormal features and output; wherein the abnormal feature database stores abnormal features corresponding to log data and non-log data, and the dynamic abnormal feature is used to characterize the features that fuse the real-time abnormal features and the corresponding probability features; The full data anomaly detection module is configured to: match the log data or non-log data corresponding to the dynamic anomaly probability with the log data or non-log data corresponding to the time series, combine the log data and the non-log data to generate full data, use the dynamic anomaly feature to perform network security anomaly detection on the full data and output the corresponding detection result; wherein the full data is the log data and non-log data corresponding to the dynamic anomaly probability; The generation of the log anomaly probability includes: The log anomaly probability is calculated using a preset log anomaly probability calculation formula, which is: ； in, Based on log data The total number of features obtained, For log data The number of features with abnormal probability obtained based on the abnormal probability feature database, For log data Features with abnormal probability , For preset log data Features with abnormal probability The weight parameter, is the calculated log anomaly probability; The generation of the non-log anomaly probability includes: The non-log anomaly probability is calculated using a preset non-log anomaly probability calculation formula, and the non-log anomaly probability calculation formula is: ； in, Based on non-log data The total number of features obtained, For non-log data The number of features with abnormal probability obtained based on the abnormal probability feature database, For non-log data Features with abnormal probability , For preset non-log data Features with abnormal probability The weight parameter, is the calculated log anomaly probability.

2. The AI-based network security anomaly detection system according to claim 1 is characterized in that: The generation of the real-time abnormal features includes: When the dynamic anomaly probability is the log anomaly probability, extracting features from the log data using the anomaly feature database to generate real-time anomaly features of the log; When the dynamic abnormality probability is the non-log abnormality probability, extracting features from the non-log data using the abnormality feature database to generate non-log real-time abnormality features; The real-time abnormality feature includes the log real-time abnormality feature and the non-log real-time abnormality feature.

3. The AI-based network security anomaly detection system according to claim 2 is characterized in that: The generation of the dynamic abnormality feature includes: When the real-time abnormal feature of the log exists, the probability feature corresponding to the non-log data is matched, and the real-time abnormal feature of the log is fused with the probability feature corresponding to the matched non-log data to generate a dynamic abnormal feature of the log; When the non-log real-time abnormal feature exists, the probability feature corresponding to the log data is matched, and the non-log real-time abnormal feature is fused with the probability feature corresponding to the matched log data to generate a non-log dynamic abnormal feature; The dynamic abnormality features include the log dynamic abnormality features and the non-log dynamic abnormality features.

4. The AI-based network security anomaly detection system according to claim 1, characterized in that: The feature fusion includes: Corresponding weights are assigned to the real-time abnormal feature and its corresponding probability feature, the weight of the real-time abnormal feature is the maximum value of the log abnormality probability and the non-log abnormality probability, and the weight of the probability feature is the minimum value of the log abnormality probability and the non-log abnormality probability. Feature fusion calculation is performed based on the assigned weights to obtain the dynamic abnormal feature.

5. The AI-based network security anomaly detection system according to claim 1, characterized in that: The generation of the full data includes: When the dynamic abnormal probability is the log abnormal probability, the log data is matched with the non-log data corresponding to the time series, and the log data and the non-log data are combined to generate the full log data; When the dynamic abnormal probability is the non-log abnormal probability, the log data corresponding to the time series is matched by using the non-log data, and the log data and the non-log data are combined to generate non-log full data; The full data includes the log full data and the non-log full data.

6. The AI-based network security anomaly detection system according to claim 5, characterized in that: The generation of the full data also includes: When the full data is the log full data, the log full data is combined using the log full data combination formula, and the log full data combination formula is: ； in, is the log anomaly probability, For the log data, For the non-log data, The full log data obtained after weighted combination; When the full data is the non-log full data, the non-log full data is combined using a non-log full data combination formula, and the non-log full data combination formula is: ； in, is the non-log anomaly probability, Non-logged full data after weighted combination.

7. The AI-based network security anomaly detection system according to claim 1, characterized in that: The system also includes a data updating module in communication connection with the abnormal probability selection module; The data update module is configured to calculate the updated probability feature using the initial probability feature, the feature expectation value obtained by statistical analysis based on the log anomaly probability or the non-log anomaly probability, and a preset update ratio.

8. An AI-based network security anomaly detection method, the method being applicable to the AI-based network security anomaly detection system according to any one of claims 1 to 7, characterized in that: The method includes: The abnormal probability of log data and non-log data is calculated respectively by using a preset abnormal probability feature database to generate a log abnormal probability and a non-log abnormal probability, and the maximum value of the log abnormal probability and the non-log abnormal probability is output, and the maximum value is defined as a dynamic abnormal probability; wherein the abnormal probability feature database stores probability features representing the probability of abnormality of log data and the probability of abnormality of non-log data, and the probability features are used to represent features corresponding to the probability of abnormality, the log data and the non-log data are operation data of different dimensions for representing network security conditions, and the non-log data at least includes network traffic data, system performance data, user behavior data, security equipment alarm data and network topology data, the log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on log data, and the non-log abnormal probability is used to represent the probability of network security abnormality obtained by network security probability detection based on non-log data, and the network security probability detection is a probabilistic network security detection; The preset abnormal feature database is used to extract features from the log data or non-log data corresponding to the dynamic abnormal probability to generate real-time abnormal features, and the real-time abnormal features are combined with the corresponding probability features to generate dynamic abnormal features and output; wherein the abnormal feature database stores abnormal features corresponding to the log data and non-log data, and the dynamic abnormal features are used to characterize the features of the fusion of the real-time abnormal features and the corresponding probability features; Match the log data or non-log data corresponding to the dynamic anomaly probability with the log data or non-log data corresponding to the time series, combine the log data and the non-log data to generate full data, use the dynamic anomaly feature to perform network security anomaly detection on the full data and output the corresponding detection result; wherein the full data is the log data and non-log data corresponding to the dynamic anomaly probability.

Images