An attack frequency classification statistics method and system

Abstract

The application discloses an attack frequency classification and statistics method and system, which comprises the following steps: collecting original logs of each security device and pre-processing the original logs; storing the pre-processed log data into an extensible distributed file system; obtaining the data stored in the extensible distributed file system, combining target dimensions, aggregating the logs, calculating the attack frequency, and performing classification and statistics. With the excellent stream processing capability of Flink, the stability of task operation and the "exactly once" semantics are guaranteed, that is, each log is processed without repetition or omission. The "event timestamp" is added in the normalized log, and the "event timestamp" is used as the aggregation basis when calculating the attack frequency, so that the influence of the out-of-order logs on the final statistical result is eliminated. In combination with the network topology of the security device, the classification standards of "reachable attack" and "block attack" are introduced, so that the attack frequency statistics are more valuable for analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of attack times classification statistics, and particularly relates to an attack times classification statistics method and system. BACKGROUND

[0002] With the rapid development of network technology, security problems are increasingly prominent. In order to cope with a large number of and various hacker attacks, security analysts need to operate various security devices to obtain attack information. However, this analysis method has many limitations: first, various security devices run independently, and the log formats are quite different, which is difficult to analyze uniformly; second, the monitoring capabilities of various security devices have intersections, and a same attack may generate a large number of alarms, and the alarm contents are actually redundant, which greatly increases the workload of monitoring and analysis.

[0003] For the scene of massive log statistics analysis, big data technology has a wide range of applications. Flink is an excellent stream processing framework, which has the characteristics of low delay, high throughput and high performance, and supports "event time" window (in the current scene, it is the time when the log is generated) for real-time calculation. Spark is a fast general-purpose computing engine designed for large-scale data processing, which is based on memory computing and is mainly used for offline analysis.

[0004] The existing attack times statistics scheme has low accuracy, which generally includes the following steps: 1) log collection: collect the logs of security devices and send them to a message queue; 2) take the message queue as a data source, establish a consumption task to store data in a file system, which is generally realized by Storm or Flink; 3) times statistics: count the number of logs in the file system as the attack times. SUMMARY

[0005] This section aims to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification of the present application to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0006] In view of the above existing problems, the present application is proposed.

[0007] Therefore, the present application provides an attack times classification statistics method and system, which can solve the problems mentioned in the background art.

[0008] To solve the above technical problems, the present application provides the following technical scheme, an attack times classification statistics method, comprising:

[0009] Collecting original logs of each security device and preprocessing the original logs;

[0010] storing the preprocessed log data into an extensible distributed file system;

[0011] Obtaining the data stored in the extensible distributed file system, combining the target dimension, aggregating the log, calculating the attack times, and performing classified statistics.

[0012] As a preferred scheme of the attack times classified statistics method, the original log of each security device includes multi-source heterogeneous data generated during the operation of each security device and the receiving time of each log.

[0013] The preprocessing includes deduplication, denoising, strengthening, and normalization of log data.

[0014] If the original log of the security device is in-order data, the deduplication and denoising include taking the original data as a data source, matching the original log content according to a regular, and creating a 5-second Flink time window to deduplicate and denoise the original log in the window.

[0015] If the original log of the security device is out-of-order data, the deduplication and denoising include taking the original data as a data source and adding a "receiving time" field to each log. When processing the log in Flink, an "event time window" is used for processing.

[0016] The strengthening includes associating the original log content with geographic location, assets, attacked business systems, and other missing contents of the original log.

[0017] The normalization includes associating the attack source IP with a geographic location information library to obtain attack source geographic location information, associating the attack destination IP with assets to obtain attacked business system information, mapping the original attack type to a standard attack type according to an alarm specification, and mapping the log content to a prepared unified field in json format and sending it to a message queue.

[0018] As a preferred scheme of the attack times classified statistics method, the storing of the preprocessed log data into an extensible distributed file system includes using a bucketing strategy and a rollback strategy to store the log data,

[0019] The bucketing strategy includes a pre-set number of buckets, and the log model number and time are used as the bucketing basis.

[0020] The rollback strategy includes the following:

[0021] s.t: t >= T h & (S n -S n+i ) = 0 & w >= W y

[0022] Wherein, t is the time that the pre-processed log data file has experienced from acquisition to current, T h is a preset file rollback period, S n is the total data in the scalable distributed file system at n moment, S n+i is the total data in the scalable distributed file system at n+i moment, i is a preset new data discrimination period, w is a current file size value, W y is a preset discrimination file size value, when the above formula condition is satisfied, a new file is generated.

[0023] As a preferred scheme of the attack times classification and statistics method, wherein: the log aggregation includes preliminary aggregation and re-aggregation,

[0024] The preliminary aggregation includes, through a preset fixed scheduling period, acquiring data in the fixed scheduling period on the distributed file system, equally dividing the preset fixed scheduling period according to a first preset time, and naming according to the Mth period-Nth time period;

[0025] When the event time in the pre-processed log is acquired, the acquired event time is mapped with the named time period, and the event time range is obtained according to the mapping result;

[0026] When a certain event time is mapped to the next minute of the time period, the occurrence time of the event time is taken as the starting time, and five minutes are taken as the range, a new event time range is acquired, which is taken as the event time range;

[0027] When the time range is determined, the sending device IP of a certain event is associated in the time range, and the reachable blocking type is acquired;

[0028] According to the event belonging network, geographical position, attack type, attacked business system, reachable blocking type and event time range, the attack times of preliminary aggregation are obtained, and stored in the current day partition of the data warehouse tool partition table.

[0029] As a preferred scheme of the attack times classification and statistics method, wherein: the log aggregation further includes that the re-aggregation includes respectively calculating the attack times of specific dimensions, establishing a partition table in the data warehouse tool, saving the intermediate results of attack times preliminary aggregation, when re-aggregation is performed, the intermediate results of the current day partition are directly calculated by using the data warehouse tool SQL, even if the data is out of order, each attack can also be classified into the correct time range according to the event time.

[0030] As a preferred scheme of the attack frequency classification statistical method, wherein: the target dimensions include geographical location dimensions, attack type dimensions and attacked business system dimensions, each dimension can be further subdivided into network and reachable blocking type;

[0031] The geographical location dimensions include first-class locations and second-class locations, the attack type dimensions include blocking attacks and reachable attacks, and the attacked business system dimensions include first-class businesses and second-class businesses;

[0032] The classification statistics include first, second, third and fourth categories;

[0033] The first category includes: when the sum of blocking attacks and reachable attacks of first-class businesses in first-class locations is greater than a first sum threshold or / and the number of reachable attacks is greater than a first reachable threshold, it is determined that the security device is subjected to first-class attack, at this time, the security analyst is notified to perform business analysis on the first-class businesses in the first-class location, if the security device cannot perform normal operation, it is determined that the first-class attack is a substantial attack, the security analyst is notified to maintain, and the attack is recorded as a first-class substantial attack;

[0034] If the security device operates normally, it is determined that the first-class attack is a non-substantial attack, the security analyst is notified to save the attack data to the cloud, clear the generated attack data in the server, record the attack as a first-class non-substantial attack, and mark the location, continuously obtain the attack type, attack frequency and attack number of the location;

[0035] If the same attack type and attack frequency are still generated, it is determined that the first-class businesses in the first-class location are being attacked, the maintenance personnel are directly notified to maintain, and the attack state is updated to a first-class substantial attack;

[0036] The second category includes: when the sum of blocking attacks and reachable attacks of first-class businesses in second-class locations is greater than a second sum threshold or / and the number of reachable attacks is greater than a second reachable threshold, it is determined that the security device is subjected to second-class attack, and the processing method of the second-class attack is the same as that of the first-class attack.

[0037] As a preferred scheme of the attack frequency classification statistical method, wherein: the classification statistics further include,

[0038] The third type includes, when the blocking attack and the reachable attack of the secondary service of the first-class position are greater than the third sum threshold or / and the reachable attack is greater than the third reachable threshold, then the security device is determined to be attacked by the third attack, at this time, the security analyst is informed to analyze the service of the secondary service of the first-class position, if the security device cannot normally work, then the third attack is determined to be a substantial attack, the security analyst is informed to maintain, and the attack is recorded as a third substantial attack;

[0039] If the security device normally works, then the third attack is determined to be a non-substantial attack, the security analyst is informed to save the attack data to the cloud, clear the generated attack data in the server, and record the attack as a first non-substantial attack, and no other operation is performed;

[0040] The fourth type includes, when the blocking attack and the reachable attack of the secondary service of the second-class position are greater than the fourth sum threshold or / and the reachable attack is greater than the fourth reachable threshold, then the security device is determined to be attacked by the fourth attack, and the processing manner of the fourth attack is the same as that of the third attack.

[0041] An attack frequency classification and statistics system, characterized by comprising a preprocessing module, a storage module and a classification and statistics module,

[0042] The preprocessing module is used for collecting original logs of each security device and pre-processing the original logs;

[0043] The storage module is used for storing the pre-processed log data into an extensible distributed file system;

[0044] The classification and statistics module is used for acquiring data stored in the extensible distributed file system, combining a target dimension, aggregating logs, calculating attack frequencies and performing classification and statistics.

[0045] A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0046] A computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the method.

[0047] The present application has the following advantages: the attack frequency classification and statistics method and system can still have excellent performance in the scene of massive logs, can balance high throughput and low delay, and can guarantee the "exactly once" semantics, that is, each log is processed without duplication or omission.

[0048] The application provides a method for processing multi-source heterogeneous logs in attack frequency statistics process, which can deduplicate redundant logs, and supplement missing information in original logs, such as geographic position and attacked business system, and finally map log content to a unified field set to obtain normalized logs. The normalized logs are transmitted and stored in json format, which is convenient for subsequent analysis and use.

[0049] The application can provide multi-dimensional attack frequency statistics results, which can help to find more useful information when analyzing attack conditions of information systems. Network security operation and maintenance personnel can reduce "reachable attacks" by fine-grained configuration of IPS and WAF interception strategies, reduce manual research and judgment workload, and enable security monitoring personnel to focus on analyzing deeper security events.

[0050] The application solves the influence of out-of-order and delayed logs on attack frequency statistics results. The use of "event time" ensures the correctness of real-time task processing log order, and the use of daily partition data of each data warehouse tool partition table ensures the correctness of attack frequency calculation order in offline task. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0052] Figure 1 A method flow chart of an attack frequency classification and statistics method and system provided for an embodiment of the application;

[0053] Figure 2 An internal structure diagram of a computer device of an attack frequency classification and statistics method and system provided for an embodiment of the application. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the specific embodiments of the application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should be within the protection scope of the application.

[0055] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0056] It should also be noted that, as used in the specification and in the claims, the article "a", "an", or "the" is intended to mean that there is one or more of the features or elements. Unless otherwise indicated, the use of the term "or" in the claims should not be understood as having an exclusive or cumulative meaning. Furthermore, terms such as "first", "second", "third", etc., are used in the description and in the claims for the purpose of clarity only and are not intended to limit the scope of the present application.

[0057] The present application is described in detail below with reference to the accompanying drawings. In describing the present application, the drawings are not necessarily drawn to scale in order to facilitate the understanding of the present application. The drawings are merely examples and should not limit the scope of the present application. In addition, the drawings should include three-dimensional spatial dimensions of length, width, and depth.

[0058] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are merely for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be understood as limiting the present application. In addition, the terms "first", "second", or "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0059] In the present application, unless otherwise specified and limited, the terms "mounting", "connection", and "connection" should be understood broadly, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can also be a mechanical connection, an electrical connection, or a direct connection; it can also be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0060] Embodiment 1

[0061] Reference Figures 1-2 For the first embodiment of the present application, the embodiment provides an attack frequency classification and statistical method and system, comprising:

[0062] Collecting original logs of each security device and pre-processing the original logs;

[0063] The original logs of each security device include multi-source heterogeneous data generated during the operation of each security device and the receiving time of each log.

[0064] It should be noted that the preprocessing includes deduplication, denoising, strengthening and normalization of the log data;

[0065] Further, if the original log of the security device is in order, the deduplication and denoising include taking the original data as the data source, matching the original log content according to the regular, and creating a 5-second Flink time window, and deduplicating and denoising the original log in the window;

[0066] Further, if the original log of the security device is out of order, the deduplication and denoising include taking the original data as the data source, and adding a "receiving time" field to each log. When processing the log in Flink, use "event time window" processing;

[0067] It should be noted that the strengthening includes associating the original log content with the geographical location, asset, attacked business system and other missing contents of the original log;

[0068] It should be noted that the normalization includes associating the attack source IP with the geographical location information library to obtain the attack source geographical location information, associating the attack destination IP with the asset to obtain the attacked business system information, mapping the original attack type to the standard attack type according to the alarm specification, and mapping the log content to the prepared unified field respectively, and sending it to the message queue in json format.

[0069] Further, storing the preprocessed log data into the extensible distributed file system includes using a bucketing strategy and a rollback strategy to realize log data storage,

[0070] Further, the bucketing strategy includes predefining the number of buckets, and taking the log model number and time as the bucketing basis;

[0071] The rollback strategy includes the following:

[0072] s.t: t≥T h & (S n -S n+i ) = 0 & w≥W y

[0073] Where t is the time that the preprocessed log data file has experienced from acquisition to the present, T h is the preset file rollback period, S n is the total data in the extensible distributed file system at time n, S n+i is the total data in the extensible distributed file system at time n+i, i is the preset new data discrimination period, w is the current file size value, W y is the preset file size value. When the above formula conditions are met, a new file is generated.

[0074] The preset file rollback period is preferably 10 minutes, the preset new data discrimination period is preferably set to 5 minutes, and the preset discrimination file size value is preferably 10 MB. The specific setting can be set by the user according to the security requirement. The shorter the period and the smaller the file size, the higher the required security.

[0075] Further, the preprocessed log data is stored in an extensible distributed file system.

[0076] Further, the data stored in the extensible distributed file system is acquired, combined with the target dimension, the log is aggregated, the attack times are calculated, and classified statistics are performed.

[0077] Further, the log aggregation includes preliminary aggregation and re-aggregation.

[0078] It should be noted that the preliminary aggregation includes acquiring data on the distributed file system within a fixed scheduling period according to a first preset time, equally dividing the fixed scheduling period according to the first preset time, and naming according to the Mth period-Nth time period.

[0079] Further, after acquiring the event time in the preprocessed log, the acquired event time is mapped to the named time period, and the event time range is obtained according to the mapping result.

[0080] Further, when a certain event time is mapped to the next minute of the time period, the occurrence time of the event time is taken as the starting time, and five minutes are taken as the range to obtain a new event time range, which is taken as the event time range.

[0081] Further, after determining the time range, the sending device IP of a certain event is associated within the time range to obtain the reachable blocking type.

[0082] Further, according to the event belonging network, geographical location, attack type, attacked business system, reachable blocking type and event time range, the preliminary aggregated attack times are obtained, and stored in the data warehouse tool partition table of the day partition.

[0083] The first preset time is preferably 5 minutes, and the fixed scheduling period is preferably selected as one hour. The accuracy requirement can be set according to the user's accuracy requirement. The higher the accuracy requirement, the shorter the first preset time in the same period.

[0084] It should be noted that the log aggregation also includes re-aggregation of the attack times of specific dimensions, establishment of a partition table in the data warehouse tool, saving of the intermediate results of the preliminary aggregation of the attack times, and direct use of the data warehouse tool SQL to calculate the intermediate results of the day partition when re-aggregation is performed, so that even if the data arrives in disorder, each attack can be classified into the correct time range according to the event time.

[0085] It should be noted that the target dimensions include a geographical location dimension, an attack type dimension, and an attacked business system dimension, and each dimension can be further subdivided into a network and a reachable blocking type;

[0086] It should be noted that the geographical location dimension includes a first-level location and a second-level location, the attack type dimension includes a blocking attack and a reachable attack, and the attacked business system dimension includes a first-level business and a second-level business.

[0087] Further, the classification statistics include a first type, a second type, a third type, and a fourth type.

[0088] Further, the first type includes that when the sum of the blocking attacks and the reachable attacks of the first-level business of the first-level location is greater than a first sum threshold or / and the number of reachable attacks is greater than a first reachable threshold, it is determined that the security device is subjected to a first-level attack, at which time a security analyst is notified to perform business analysis on the first-level business of the first-level location, and if the security device cannot perform normal operation, it is determined that the first-level attack is a substantive attack, a security analyst is notified to perform maintenance, and the attack is recorded as a first-level substantive attack.

[0089] Further, if the security device is in normal operation, it is determined that the first-level attack is a non-substantive attack, a security analyst is notified to save the attack data to the cloud, clear the attack data generated in the server, record the attack as a first-level non-substantive attack, and mark the location, and continuously obtain the attack type, attack frequency, and attack times of the location.

[0090] Further, if the same attack type and attack frequency are still generated, it is determined that the first-level business of the first-level location is being attacked, and a maintenance personnel is directly notified to perform maintenance, and the attack state is updated to a first-level substantive attack.

[0091] Further, the second type includes that when the sum of the blocking attacks and the reachable attacks of the first-level business of the second-level location is greater than a second sum threshold or / and the number of reachable attacks is greater than a second reachable threshold, it is determined that the security device is subjected to a second-level attack, and the processing method of the second-level attack is the same as that of the first-level attack.

[0092] Further, the third type includes, when the sum of the blocking attacks and the reachable attacks of the secondary service of the first position is greater than a third sum threshold or / and the number of reachable attacks is greater than a third reachable threshold, it is determined that the security device is subjected to a third attack, at this time, the security analyst is informed to perform service analysis on the secondary service of the first position, if the security device cannot perform normal operation, it is determined that the third attack is a substantial attack, the security analyst is informed to maintain, and the attack is recorded as a third substantial attack;

[0093] Further, if the security device is normal, it is determined that the third attack is a non-substantial attack, the security analyst is informed to save the attack data to the cloud, clear the generated attack data in the server, and record the attack as a first non-substantial attack, and no other operation is performed;

[0094] Further, the fourth type includes, when the sum of the blocking attacks and the reachable attacks of the secondary service of the second position is greater than a fourth sum threshold or / and the number of reachable attacks is greater than a fourth reachable threshold, it is determined that the security device is subjected to a fourth attack, and the processing manner of the fourth attack is the same as that of the third attack.

[0095] It should be noted that the first position is a position that cannot work normally, so that the whole device cannot work normally, the remaining positions are second positions, the first service is a service that cannot be suspended, and the remaining services are secondary services.

[0096] Preferably, the first sum threshold is 50 times, the first reachable threshold is 20 times, the second sum threshold is 70 times, the second reachable threshold is 35 times, the third sum threshold is 90 times, the third reachable threshold is 40 times, the fourth sum threshold is 110 times, and the fourth reachable threshold is 50 times. The specific setting can be set by the user according to the security requirement, and the fewer the number is, the higher the security requirement is.

[0097] In one embodiment, an attack frequency classification and statistics system includes a preprocessing module, a storage module, and a classification and statistics module,

[0098] The preprocessing module is used to collect original logs of each security device and preprocess the original logs;

[0099] The storage module is used to store the preprocessed log data in an extensible distributed file system;

[0100] The classification and statistics module is used to obtain data stored in the extensible distributed file system, combine target dimensions, aggregate logs, calculate attack frequencies, and perform classification and statistics.

[0101] The above each unit module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls to execute the operation corresponding to each of the above modules.

[0102] In one embodiment, a computer device, which can be a terminal, has an internal structure diagram as shown in Figure 2 The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, a carrier network, NFC (Near Field Communication) or other technologies. The computer program is executed by the processor to implement an attack frequency classification and statistics method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0103] In one embodiment, a computer readable storage medium having a computer program stored thereon is provided. The computer program is executed by a processor to implement the following steps:

[0104] Collecting original logs of each security device and pre-processing the original logs;

[0105] Storing the pre-processed log data into an extensible distributed file system;

[0106] Obtaining the data stored in the extensible distributed file system, combining target dimensions, aggregating the logs, calculating the attack frequency, and performing classification and statistics.

[0107] Embodiment 2

[0108] Referring to Figures 1-2 For one embodiment of the present application, an attack frequency classification and statistics method and system are provided. In order to verify the beneficial effects of the present application, a comparative experiment is performed for scientific demonstration.

[0109] Table 1 Classification and statistics of attack frequency at different positions in a certain fixed period

[0110]

[0111] The application provides an attack frequency classification and statistics method and system, which still has excellent performance in a massive log scenario, can balance high throughput and low delay, and can guarantee the "exactly once" semantics, that is, each log is processed without omission or repetition.

[0112] The application provides a method for processing multi-source heterogeneous logs in attack frequency statistics, which can deduplicate redundant logs and supplement missing information in original logs, such as geographic positions and attacked business systems, and finally maps the log content to a unified field set to obtain normalized logs.

[0113] The application can provide multi-dimensional attack frequency statistics results, which can help to find more useful information when analyzing the attack situation of an information system.

[0114] The application solves the influence of out-of-order and delayed logs on attack frequency statistics results.

[0115] It should be noted that the above examples are only used to illustrate the technical solutions of the application and are not limiting.

[0116] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product.

[0117] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0118] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0119] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0120] While the preferred embodiments of the application have been described, additional variations and modifications can be employed by those skilled in the art. Therefore, the appended claims intend to cover all such modifications and variations as fall within the true spirit and scope of the application.

[0121] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An attack frequency classification statistics method, characterized in that: The application relates to a log data processing method and device. Collecting original logs of each security device and preprocessing the original logs; Storing the preprocessed log data into an extensible distributed file system; Obtaining data stored in the extensible distributed file system, combining target dimensions, aggregating the logs, calculating attack times, and performing classified statistics; The original logs of each security device include multi-source heterogeneous data generated in the running process of each security device and the receiving time of each log; The preprocessing includes deduplication, denoising, strengthening and normalization of log data; If the original logs of the security device are in-order data, the deduplication and denoising include taking the original data as a data source, regularly matching the original log content, and creating a 5-second Flink time window to deduplicate and denoise the original logs in the window; If the original logs of the security device are out-of-order data, the deduplication and denoising include taking the original data as a data source and adding a "receiving time" field to each log. When processing the logs in Flink, an "event time window" is used for processing; The strengthening includes correlating the original log content with geographic location, assets, attacked business systems and other missing contents of the original logs; The normalization includes associating the attack source IP with a geographic location information library to obtain attack source geographic location information, associating the attack destination IP with assets to obtain attacked business system information, mapping the original attack type to a standard attack type according to an alarm specification, and mapping the log content to a prepared unified field respectively, and sending the log content to a message queue in a json format; The storage of the preprocessed log data into the extensible distributed file system includes using a bucketing strategy and a rollback strategy to realize the storage of the log data, The bucketing strategy includes a preset number of buckets, and the log model number and time are used as the bucketing basis; The rollback strategy includes the following: s.t: t ≥ Th & (Sn - Sn+i) = 0 & w ≥ W y Wherein, t is the time that the pre-processed log data file has experienced from acquisition to the present, T h to be a preset file rollback period, S n to be the total data in the scalable distributed file system at time n, S n+i to be the total data in the scalable distributed file system at time n+i, i is a preset new data discrimination period, w is the current file size value, W y to be a preset discrimination file size value, when the above formula condition is met, a new file is generated; The log aggregation includes preliminary aggregation and re-aggregation, The preliminary aggregation includes obtaining data in a fixed scheduling period on the distributed file system through a preset fixed scheduling period, equally dividing the preset fixed scheduling period according to a first preset time, and naming according to the Mth period-Nth time period; When the event time in the preprocessed log is obtained, the obtained event time is mapped with the named time period, and the event time range is obtained according to the mapping result; When a certain event time is mapped to the next minute of a time period, the occurrence time of the event time is taken as the starting time, and a new event time range is obtained as the event time range; After the time range is determined, the sending device IP of a certain event in the time range is associated to obtain an accessible blocking type; According to the network, geographic location, attack type, attacked business system, accessible blocking type and event time range of the event, the preliminary aggregated attack times are obtained and stored in the data warehouse tool partition table of the day. The log aggregation further comprises that the re-aggregation comprises calculating the attack times of specific dimensions respectively, establishing a partition table in a data warehouse tool, saving the intermediate results of the attack times preliminary aggregation, and directly using the data warehouse tool SQL to calculate the intermediate results of the day partition when re-aggregation is performed, so that each attack can be classified into the correct time range according to the event time even if the data arrives in disorder; The target dimensions comprise a geographical position dimension, an attack type dimension and an attacked business system dimension, and each dimension can be further subdivided into a network and a reachable blocking type; The geographical position dimension comprises a first-level position and a second-level position, the attack type dimension comprises a blocking attack and a reachable attack, and the attacked business system dimension comprises a first-level business and a second-level business; The classification statistics comprise a first class, a second class, a third class and a fourth class; The first class comprises that when the sum of the blocking attack and the reachable attack of the first-level business of the first-level position is greater than a first sum threshold or / and the reachable attack times are greater than a first reachable threshold, it is determined that the security device is subjected to a first-level attack, at this time, a security analyst is notified to perform business analysis on the first-level business of the first-level position, if the security device cannot normally work, it is determined that the first-level attack is a substantive attack, the security analyst is notified to maintain, and the attack is recorded as a first-level substantive attack; If the security device normally works, it is determined that the first-level attack is a non-substantive attack, the security analyst is notified to save the attack data to the cloud, clear the generated attack data in the server, record the attack as a first-level non-substantive attack, and mark the position, and continuously acquire the attack type, attack frequency and attack times of the position; If the same attack type and attack frequency are still generated, it is determined that the first-level business of the first-level position is being attacked, the maintenance personnel are directly notified to maintain, and the attack state is updated to a first-level substantive attack; The second class comprises that when the sum of the blocking attack and the reachable attack of the first-level business of the second-level position is greater than a second sum threshold or / and the reachable attack times are greater than a second reachable threshold, it is determined that the security device is subjected to a second-level attack, and the processing mode of the second-level attack is the same as that of the first-level attack; The classification statistics further comprise that The third class comprises that when the sum of the blocking attack and the reachable attack of the second-level business of the first-level position is greater than a third sum threshold or / and the reachable attack times are greater than a third reachable threshold, it is determined that the security device is subjected to a third-level attack, at this time, a security analyst is notified to perform business analysis on the second-level business of the first-level position, if the security device cannot normally work, it is determined that the third-level attack is a substantive attack, the security analyst is notified to maintain, and the attack is recorded as a third-level substantive attack; If the security device normally works, it is determined that the third-level attack is a non-substantive attack, the security analyst is notified to save the attack data to the cloud, clear the generated attack data in the server, and record the attack as a first-level non-substantive attack, without other operations. The fourth type includes, when the blocking attack of the secondary service of the secondary position and the total number of reachable attacks is greater than the fourth total threshold or / and the number of reachable attacks is greater than the fourth reachable threshold, then it is determined that the security device is subjected to a fourth level attack, and the fourth level attack processing manner is the same as the third level attack processing manner.

2. An attack frequency classification statistics system applying the method of claim 1, characterized by: The pre-processing module, the storage module, and the classification and statistics module, The pre-processing module is used for collecting original logs of each security device and pre-processing the original logs; The storage module is used for storing the pre-processed log data into an extensible distributed file system; The classification and statistics module is used for obtaining data stored in the extensible distributed file system, combining target dimensions, aggregating logs, calculating attack times, and performing classification and statistics.

3. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method in claim 1.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in claim 1.

Images