A data processing method and system for reducing alarm noise based on dynamic rules
By standardizing, initially filtering and normalizing a variety of log data, combining the event risk level dimension table, structured event data is generated and dynamic rules are configured, the problem of alarm data filtering is solved, and the alarm noise reduction and operation are achieved Improved dimensional efficiency.
Patent Information
- Application Number
- CN202411268160.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-09-11
AI Technical Summary
In industrial production, how to filter out the required alarm data in real time and accurately from a large amount of alarm data, reduce the work pressure of operation and maintenance management personnel, and improve work quality.
By obtaining raw log data of various log format types, standardized processing, preliminary filtering and normalization processing, building event risk level dimension table, correlating key information of log content, generating structured event data, and configuring dynamic rules based on web pages Generate FlinkSQL calculation tasks to realize noise reduction processing of alarm data.
Effectively extract the alarm data that the operation and maintenance needs are concerned about, reduce the number of alarm data display, improve user work efficiency, and realize flexible configuration of dynamic rules that different operation and maintenance needs are concerned about.
Smart Images

Figure CN118779309B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method and system for reducing alarm noise based on dynamic rules. Background Art
[0002] In the field of industrial production, a large amount of complex alarm data is generated, and the alarm data has different danger levels. How to filter out the required alarm data in real time and accurately from a large amount of alarm data, reduce the work pressure of operation and maintenance management personnel, and improve the quality of work is the key. In view of this, the present invention provides a data processing method and system for reducing alarm noise based on dynamic rules. Summary of the invention
[0003] Based on this, it is necessary to provide a data processing method and system for reducing alarm noise based on dynamic rules to address the above technical problems.
[0004] According to a first aspect of the present invention, a data processing method for reducing alarm noise based on dynamic rules is provided, including: obtaining original log data of multiple log format types; configuring corresponding log parsing rules based on the log format type of the original log data, and standardizing the original log data of the corresponding log format type based on the configured log parsing rules to extract key information of the log content; performing preliminary filtering on the key information of the log content based on predefined preliminary filtering rules to obtain effective key information of the log content; constructing an event risk level dimension table, associating the event risk level dimension table with the effective log content key information, and constructing event detail data containing risk level information; normalizing the event detail data to construct structured event data; outputting all normalized structured event data to the corresponding target topic of the message queue; configuring dynamic rules of operation and maintenance concern based on a web page, and generating corresponding FlinkSQL computing tasks after the dynamic rule configuration is completed, consuming the structured event data under the target topic in the message queue according to the FlinkSQL computing tasks, and obtaining the denoised alarm data.
[0005] In some optional implementations of some embodiments, the original log data includes a first category of original log data and a second category of original log data, wherein the source of the first category of original log data is the original log data reported by the host security program, the source of the second category of original log data is the original log data output by the terminal device, and the second category of original log data is JSON format data.
[0006] In some optional implementations of some embodiments, the corresponding log parsing rules are configured based on the log format type of the original log data, including: for the first type of original log data, Syslog parsing rules are configured as the corresponding log parsing rules; for the second type of original log data, JSON format mapping rules are configured as the corresponding log parsing rules.
[0007] In some optional implementation methods of some embodiments, the original log data of the corresponding log format type is standardized to extract key information of the log content, including: for the first type of original log data, PRI and host name are used as parsing conditions, and Syslog parsing rules are used to parse the original log data of the corresponding log format type to parse the content information of the first type of original log data, and according to the preset mapping rules, the first type of log content key information in JSON format is obtained; for the second type of original log data, the JSON format mapping rules are used to standardize the original log data of the corresponding log format type to obtain the second type of log content key information.
[0008] In some optional implementations of some embodiments, the log content key information is preliminarily filtered based on predefined preliminary filtering rules to obtain valid log content key information, including: based on predefined preliminary filtering rules, the first category of log content key information and the second category of log content key information are preliminarily filtered, and invalid information that does not meet the preliminary filtering rules is filtered out to obtain the first category of valid log content key information and the second category of valid log content key information, respectively.
[0009] In some optional implementations of some embodiments, the event risk level dimension table is constructed, the event risk level dimension table is associated with the effective log content key information, and event detail data containing risk level information is constructed, including: based on the first type of effective log content key information and the second type of effective log content key information, respectively constructing a first type of event risk level dimension table and a second type of event risk level dimension table, the first type of event risk level dimension table includes one or more first type of associated identification table items, the first type of associated identification table items include first type of risk level values and first type of risk level information, the second type of event risk level dimension table includes one or more second type of associated identification table items, the second type of associated identification table items include one or more associated condition information, second type of risk level values and second type of risk level information; based on the first type of effective log content key information, calculating the corresponding first type of log risk level value, the specific calculation formula is as follows: ; Wherein: Level represents the risk level value of the first-class log, Level is encoded in decimal, and the value range of Level is 0-5; PRI represents the priority of the log; Facility represents the type of program or device that identifies the log, and Facility is encoded in decimal, and the value range of Facility is 0-22; Based on the mapping relationship formed by the first-class log risk level value and the first-class risk level value field of the first-class event risk level dimension table, the first-class event risk level dimension table and the first-class valid log content key information are associated to obtain the first-class risk level information corresponding to the first-class risk level value, and the first-class valid log content key information is marked to construct the first-class event detailed data containing the first-class risk level information; Based on the second-class valid log content key information, according to the predefined Multiple association type matching quantity intervals and second-class log risk level values assigned to each association type matching quantity interval, performing type matching comparison on the content information field contained in the second-class valid log content key information and the association condition information field to obtain the type matching quantity result, judging the association type matching quantity interval where the type matching quantity result is located, obtaining the interval judgment result and the second-class log risk level value corresponding to the interval judgment result, based on the mapping relationship formed by the second-class log risk level value and the second-class risk level value field of the second-class event risk level dimension table, associating the second-class event risk level dimension table with the second-class valid log content key information to obtain the second-class risk level information corresponding to the second-class log risk level value, marking the second-class valid log content key information, and constructing the second-class event detailed data containing the second-class risk level information.
[0010] In some optional implementations of some embodiments, the normalization of event detail data to construct structured event data includes: normalizing the first category of event detail data and the second category of event detail data respectively according to the mapping content information field contained in the preset data format, extracting the first content information field and the second content information field required by the corresponding mapping content information field respectively, and constructing the first structured event data and the second structured event data respectively based on the extracted first content information field and the second content information field.
[0011] According to a second aspect of the present invention, there is provided a data processing system for reducing alarm noise based on dynamic rules, comprising: a log acquisition module for acquiring raw log data of various log format types; a first data processing module for configuring corresponding log parsing rules based on the log format type of the raw log data, and performing standardization processing on the raw log data of the corresponding log format type based on the configured log parsing rules to extract key information of the log content; a second data processing module for performing preliminary filtering processing on the key information of the log content based on predefined preliminary filtering rules to obtain effective key information of the log content; and an event detail data construction module for constructing event risk levels. Dimension table, which associates the event risk level dimension table with the key information of the valid log content, and constructs event detail data containing risk level information; structured event data construction module, which is used to normalize the event detail data and construct structured event data; writing module, which is used to output all normalized structured event data to the corresponding target topic of the message queue; denoising module, which is used to configure the dynamic rules of operation and maintenance concerns based on the web page, and after the dynamic rules are configured, generate the corresponding FlinkSQL computing tasks, and consume the structured event data under the target topic in the message queue according to the FlinkSQL computing tasks to obtain the denoised alarm data.
[0012] According to a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0013] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program implements the above method steps when executed by a processor.
[0014] The advantages and beneficial effects of the present invention are as follows: the present invention provides a data processing method and system for reducing alarm noise based on dynamic rules. The method obtains original log data of various log format types, and sequentially performs standardization processing, preliminary filtering processing and normalization processing on the original log data to obtain structured event data, so as to extract common content information data. At the same time, by configuring dynamic rules of operation and maintenance concern on the web page, the purpose of dynamically generating filtering rules can be achieved after the user customizes the operation and maintenance concern requirements. Compared with traditional preset rules, it not only realizes the flexible configuration of different dynamic rules for different operation and maintenance concern requirements, but also can effectively extract alarm data of operation and maintenance concern requirements, and effectively reduce the amount of alarm data displayed, so as to free it from complex alarm data, thereby improving user work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is the overall flow chart of the present invention.
[0016] Figure 2 It is a system principle block diagram of the present invention.
[0017] Figure 3 A schematic diagram of an electronic device. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific implementation methods in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0019] Embodiment 1
[0020] This application proposes a data processing method based on dynamic rules to reduce alarm noise, such as Figure 1 As shown, the following steps are included.
[0021] S1. Obtain raw log data of various log formats.
[0022] In this embodiment, the original log data includes a first category of original log data and a second category of original log data, wherein the source of the first category of original log data is the original log data reported by the host security program, the source of the second category of original log data is the original log data output by the terminal device, and the second category of original log data is JSON format data.
[0023] In this embodiment, the first type of original log data is the original log data reported by the host security program, and the specific log content information includes: PRI, date, host name, log module [process id]: | log type | log module | device id | authorization status | time | processing result, wherein the date refers to the time when the current event occurs, and the date needs to be specific to year, month, day, hour, minute, and second; the process id represents the process id of the current host security program sending the Syslog log; the device id represents the unique code (uuid) of the current host device; the authorization status represents the authorization status of the current host device, and the time represents the time when the current event occurs, and the time needs to be specific to year, month, day, hour, minute, and second; the processing result represents the processing result of the current host device on the current risk event, including alarm, interception, etc.
[0024] In this embodiment, the second category of original log data is the original log data output after passing through the terminal device, and the second category of original log data is JSON format data, wherein the terminal device includes but is not limited to audit, intrusion, firewall equipment, etc., and the specific log content information includes: date, manufacturer / reporting IP, log content, wherein the log content includes log type, specific time of data reporting, and specific message; the specific message includes the IP of the device initiating the attack, the IP of the device being attacked, the MAC of the device initiating the attack, the MAC of the device being attacked, the port of the device initiating the attack, the port of the device being attacked, the protocol type, specific description information of the attack, actions to respond to the attack and other information; the date is the time when a specific event occurs, and the date needs to be specific to different logs of year, month, day, hour, minute, and second.
[0025] In this embodiment, in addition to the above-mentioned common content information, different types of log content information also have differences. For example, the threat log also includes threat level information, the whitelist log also includes whitelist ID, whitelist name and other information, the baseline log also includes baseline ID, baseline name and other information, and the protocol audit log also includes application layer protocol, transport layer protocol, protocol field object, data packet size, data packet content and other information.
[0026] S2. Based on the log format type of the original log data, configure the corresponding log parsing rules, and based on the configured log parsing rules, perform standardization processing on the original log data of the corresponding log format type to extract key information of the log content.
[0027] In this embodiment, based on the log format type of the original log data, corresponding log parsing rules are configured, including: for the first type of original log data, Syslog parsing rules are configured as the corresponding log parsing rules; for the second type of original log data, JSON format mapping rules are configured as the corresponding log parsing rules.
[0028] In this embodiment, the original log data of the corresponding log format type is standardized and the key information of the log content is extracted, including: for the first type of original log data, the original log data of the corresponding log format type is parsed using the Syslog parsing rule with PRI and host name as parsing conditions to parse the content information of the first type of original log data, and according to the preset mapping rule, the key information of the first type of log content in JSON format is obtained; for the second type of original log data, the original log data of the corresponding log format type is standardized using the JSON format mapping rule to obtain the key information of the second type of log content.
[0029] In this embodiment, for the first type of original log data, since this type of original log data adopts the Syslog message format, it is necessary to first use the Syslog parsing rules to parse the first type of original log data. Taking PRI as the parsing condition, the log risk level information can be parsed out. Taking the host name as the parsing condition, the corresponding victim IP, protocol, attack event code, device type, specific alarm content, time and other information can be parsed out. The parsed information can be summarized to obtain the content information of the first type of original log data. Then, a data exchange tool (for example: Nxlog) is used to convert the content information of the first type of original log data into the Syslog message format according to the preset mapping rules. The information is converted into a format. The specific implementation steps are as follows: the content information fields of the first type of original log data are respectively mapped to the attacking device IP (src_ip), the victim device IP (dst_ip), the attacking device port (src_port), the victim device port (dst_port), the protocol type (protocol), the attack event code (primary_code), the device type (device_type_code), the alarm information content (content), and the attack launching time (ts). Then, the key information of the first type of log content in JSON format can be obtained.
[0030] In this embodiment, for the second type of original log data, since this type of original log data is in JSON format, there is no need to perform a format conversion process. According to the preset JSON format mapping rules, the content information fields contained in the original log data are directly mapped to the attacking device IP (src_ip), the victim device IP (dst_ip), the attacking device port (src_port), the victim device port (dst_port), the protocol type (protocol), the attack event code (primary_code), the device type (device_type_code), the alarm information content (content), and the time of launching the attack (ts), and the second type of log content key information can be obtained; by configuring corresponding log parsing rules for different types of original log data, it can be flexibly configured according to the actual log type. Compared with a single Syslog parsing rule, this application adds a JSON format conversion function, which has a wider scope of application and is more flexible to use.
[0031] S3. Based on predefined preliminary filtering rules, perform preliminary filtering on the key information of the log content to obtain valid key information of the log content.
[0032] In this embodiment, based on predefined preliminary filtering rules, preliminary filtering processing is performed on the key information of the log content to obtain valid key information of the log content, including: based on the predefined preliminary filtering rules, preliminary filtering processing is performed on the first category of key information of the log content and the second category of key information of the log content, respectively, and invalid information that does not meet the preliminary filtering rules is filtered out to obtain the first category of valid key information of the log content and the second category of valid key information of the log content, respectively.
[0033] In this embodiment, in order to improve the efficiency of subsequent alarm data screening and reduce the amount of alarm data screened and filtered, the present application adds a process of preliminary filtering of key information in the log content to remove some invalid information, such as data with incorrect IP format, non-alarm events and other information.
[0034] S4. Build an event risk level dimension table, associate the event risk level dimension table with the key information of the valid log content, and construct event detail data containing risk level information.
[0035] In this embodiment, an event risk level dimension table is constructed, the event risk level dimension table is associated with the effective log content key information, and event detail data containing risk level information is constructed, including: based on the first type of effective log content key information and the second type of effective log content key information, a first type of event risk level dimension table and a second type of event risk level dimension table are respectively constructed, the first type of event risk level dimension table includes one or more first type of associated identification table items, the first type of associated identification table items include first type of risk level value and first type of risk level information, the second type of event risk level dimension table includes one or more second type of associated identification table items, the second type of associated identification table items include one or more associated condition information, second type of risk level value and second type of risk level information; based on the first type of effective log content key information, the corresponding first type of log risk level value is calculated, and the specific calculation formula is as follows: ; Wherein: Level represents the risk level value of the first-class log, Level is encoded in decimal, and the value range of Level is 0-5; PRI represents the priority of the log; Facility represents the type of program or device that identifies the log, and Facility is encoded in decimal, and the value range of Facility is 0-22; Based on the mapping relationship formed by the first-class log risk level value and the first-class risk level value field of the first-class event risk level dimension table, the first-class event risk level dimension table and the first-class valid log content key information are associated to obtain the first-class risk level information corresponding to the first-class risk level value, and the first-class valid log content key information is marked to construct the first-class event detailed data containing the first-class risk level information; Based on the second-class valid log content key information, according to the predefined Multiple association type matching quantity intervals and second-class log risk level values assigned to each association type matching quantity interval, performing type matching comparison on the content information field contained in the second-class valid log content key information and the association condition information field to obtain the type matching quantity result, judging the association type matching quantity interval where the type matching quantity result is located, obtaining the interval judgment result and the second-class log risk level value corresponding to the interval judgment result, based on the mapping relationship formed by the second-class log risk level value and the second-class risk level value field of the second-class event risk level dimension table, associating the second-class event risk level dimension table with the second-class valid log content key information to obtain the second-class risk level information corresponding to the second-class log risk level value, marking the second-class valid log content key information, and constructing the second-class event detailed data containing the second-class risk level information.
[0036] In this embodiment, the present application constructs an event risk level dimension table for different types of log data. For example, the system logs of industrial safety audit equipment are divided into four risk levels: general, relatively serious, serious, and extremely serious. Each reported log data is associated with the event risk level dimension table.
[0037] In this embodiment, for the key information of the first type of valid log content, the first type of log risk level value is calculated by PRI, based on the mapping relationship formed by the first type of log risk level value and the field of the first type of risk level value of the first type of event risk level dimension table, when the calculation result of the first type of log risk level value is 0, the first type of risk level value corresponding to the first type of event risk level dimension table is 1, and the corresponding first type of risk level information is particularly serious; when the calculation result of the first type of log risk level value is 1, the first type of risk level value corresponding to the first type of event risk level dimension table is 2, and the corresponding first type of risk level information is serious; when the calculation result of the first type of log risk level value is 2, the first type of risk level value corresponding to the first type of event risk level dimension table is 3, and the corresponding first type of risk level information is relatively serious; when the calculation result of the first type of log risk level value is 3-5, the first type of risk level value corresponding to the first type of event risk level dimension table is 4, and the corresponding first type of risk level information is general; if the calculation result of the first type of log risk level value is greater than or equal to 6, the data is discarded and does not participate in the subsequent statistics.
[0038] In this embodiment, for the second type of effective log content key information, different risk levels are divided in the event risk level dimension table according to different devices and different log types, according to the predefined multiple association type matching quantity intervals and the second type log risk level value assigned to each association type matching quantity interval, for example, for the threat log data reported by the intrusion device, the association type matching quantity interval can be divided into 4, namely [0, 1), [1, 2), [2, 3), [3, 4), when the association type matching quantity interval is [0, 1), the second type log risk level value assigned to the interval is 4, when the association type matching quantity interval is [ 1, 2), the second-class log risk level value assigned to the interval is 3; when the associated type matching number interval is [2, 3), the second-class log risk level value assigned to the interval is 2; when the associated type matching number interval is [3, 4), the second-class log risk level value assigned to the interval is 1. The content information field contained in the second-class valid log content key information is compared with the associated condition information field to obtain the type matching number result, and the associated type matching number interval where the type matching number result is located is determined to obtain the interval judgment result and the second-class log risk level value corresponding to the interval judgment result. For example, the obtained type matching The quantity result is 1, and it is judged that the type matching quantity result belongs to the associated type matching quantity interval [1, 2), and the second type log risk level value corresponding to the associated type matching quantity interval [1, 2) is 3; according to the mapping relationship formed by the second type log risk level value and the second type event risk level value field of the risk level dimension table, the second type event risk level dimension table is associated with the second type valid log content key information to obtain the second type risk level information corresponding to the second type log risk level value. For example, in the second type event risk level dimension table, the second type risk level value can be set to 1, 2, 3, 4. When the second type risk level value When the second category risk level value is 1, the corresponding second category risk level information is extremely serious. When the second category risk level value is 2, the corresponding second category risk level information is serious. When the second category risk level value is 3, the corresponding second category risk level information is relatively serious. When the second category risk level value is 4, the corresponding second category risk level information is general. The aforementioned second category log risk level value of 3 is used as an example. According to the mapping relationship formed by the fields of the second category log risk level value and the second category risk level value of the second category event risk level dimension table, the second category risk level information corresponding to the second category log risk level value of 3 can be obtained, and the second category risk level information is relatively serious.
[0039] In this embodiment, the event risk level dimension table is artificially divided according to data, and different event risk level dimension tables are constructed according to different devices and different log types. For example, the content of the event risk level dimension table includes but is not limited to device type name (device_type), device type code (device_type_code), attack event code (primary_code), attack event sub-code (sub_code), risk level value (level), manufacturer information (manufacturer), log type code (log_type_code), log type (log_type), data enablement status (status), remark information (description), risk level information (event_level), etc. Therefore, the associated condition information may include device type name (device_type), device type code (device_type_code), attack event code (primary_code), attack event sub-code (sub_code), manufacturer information (manufacturer), log type code (log_type_code), log type (log_type), data enablement status (status), remark information (description), etc.
[0040] In this embodiment, each valid log content key information that has been initially filtered is recorded as an event. According to the definition of different events, the risk level information corresponding to different events is obtained by associating the event risk level dimension table with the valid log content key information. On the basis of the valid log content key information, a risk level information field is added to construct event detail data containing risk level information, which is convenient for subsequent alarm data screening.
[0041] S5. Normalize the event detail data to construct structured event data.
[0042] In this embodiment, the event detail data is normalized to construct structured event data, including: normalizing the first category event detail data and the second category event detail data respectively according to the mapping content information field included in the preset data format, extracting the first content information field and the second content information field required for the corresponding mapping content information field respectively, and constructing the first structured event data and the second structured event data respectively based on the extracted first content information field and the second content information field.
[0043] In this embodiment, heterogeneous data (i.e., first-category event details and second-category event details) are normalized to construct a required data format. The specific implementation steps include: extracting content information from the first-category event details and the second-category event details, extracting the value of each content information field, and mapping the extracted value of each content information field to the required fields. For example, extracting the source IP port and mapping it into the ip (src_ip) field of the device initiating the attack, extracting the destination IP port and mapping it into the ip (dst_ip) field of the victim device, extracting the event risk level information and mapping it into the risk level information (event_level) field, extracting the specific event content and mapping it into the alarm information content (content) field, extracting the source port (src_port), the destination port (dst_port), and the time when the event occurred, and mapping them into the time when the attack was initiated (ts) field. It should be noted that if the corresponding content information field in the first-category event details and the second-category event details has a value, mapping processing is performed. If the corresponding content information field has no value, the mapped field is empty.
[0044] S6. Output all normalized structured event data to the corresponding target topic of the message queue.
[0045] In this embodiment, the structured event data after normalization is uniformly output to the message queue of the noise reduction rule statistics program and stored in the corresponding target topic of the message queue, wherein the message queue may adopt kafka.
[0046] S7. Configure the dynamic rules of concern to the operation and maintenance based on the web page. After the dynamic rules are configured, generate the corresponding FlinkSQL computing tasks. According to the structured event data under the target topic in the message queue consumed by the FlinkSQL computing tasks, obtain the denoised alarm data.
[0047] In this embodiment, the user configures dynamic rules through the web page according to the operation and maintenance concerns. Compared with the traditional preset deduplication rules, the user can customize the required rules on the web page and achieve the purpose of dynamic deduplication. For example, the dynamic rule is to filter the events within the last 5 minutes, the asset IP is 10.25.16.150, and the risk level is particularly serious. After the dynamic rule is configured, the system will generate a FlinkSQL computing task according to the edited dynamic rule, and consume the structured event data under the target topic in the message queue to achieve the purpose of executing data analysis and statistics, so as to obtain the denoised alarm data. According to the user's storage requirements, the denoised alarm data can also be output to the business database for subsequent use by the user.
[0048] Embodiment 2
[0049] This embodiment provides a data processing system 100 based on dynamic rules to reduce alarm noise, based on the above embodiment 1. Figure 2 , implementing a data processing method for reducing alarm noise based on dynamic rules as described in Example 1, the system includes a log acquisition module 110, a first data processing module 120, a second data processing module 130, an event detail data construction module 140, a structured event data construction module 150, a writing module 160 and a noise reduction module 170.
[0050] In this embodiment, the log acquisition module 110 is used to acquire original log data of multiple log formats.
[0051] In this embodiment, the first data processing module 120 is used to configure corresponding log parsing rules based on the log format type of the original log data, and based on the configured log parsing rules, perform standardization processing on the original log data of the corresponding log format type to extract key information of the log content.
[0052] In this embodiment, the second data processing module 130 is used to perform preliminary filtering processing on the key information of the log content based on a predefined preliminary filtering rule to obtain valid key information of the log content.
[0053] In this embodiment, the event detail data construction module 140 is used to construct an event risk level dimension table, associate the event risk level dimension table with the effective log content key information, and construct event detail data containing risk level information.
[0054] In this embodiment, the structured event data construction module 150 is used to normalize the event detail data and construct structured event data.
[0055] In this embodiment, the writing module 160 is used to output all normalized structured event data to the corresponding target topic of the message queue.
[0056] In this embodiment, the noise reduction module 170 is used to configure dynamic rules of operation and maintenance concern based on the web page, and after the dynamic rule configuration is completed, generate the corresponding FlinkSQL computing task, and consume the structured event data under the target topic in the message queue of the FlinkSQL computing task to obtain the noise-reduced alarm data.
[0057] Embodiment 3
[0058] This embodiment further provides an electronic device based on the above embodiment 1. Figure 3 , Figure 3The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0059] like Figure 3 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0060] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, etc., an output device 307 including, for example, a liquid crystal display (LCD), a speaker, etc., a storage device 308 including, for example, a magnetic tape, a hard disk, etc., and a communication device 309. The communication device 309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 3 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0061] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.
[0062] Embodiment 4
[0063] Based on the above-mentioned embodiment 1, this embodiment further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned method are implemented.
[0064] It should be noted that the computer-readable medium mentioned above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0065] In this embodiment, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperTextTransferProtocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.
[0066] The computer-readable medium may be included in the device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains training data, converts the training data into initial data; determines an initial rule base based on the initial data, and optimizes the parameters of the initial rule base to obtain a target rule base; calculates the rules in the target rule base according to a preset activation weight calculation formula to obtain activation weights; determines abnormal information according to the test data and the activation weights.
[0067] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0068] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0069] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be provided in a processor, for example, may be described as: a processor including a data acquisition unit, a rule determination unit, a weight calculation unit, and an anomaly determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the data acquisition unit may also be described as a "unit for acquiring training data".
[0070] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used include, without limitation, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.
[0071] Obviously, it should be understood by those skilled in the art that the above-mentioned various steps of the present invention can be performed in a manner different from the present invention, and the simulation method and experimental equipment include but are not limited to the above description. The above-mentioned various steps of the present invention can be performed in an order different from that here in some cases, and the steps shown or described above can be performed separately. Therefore, the present invention is not limited to any specific combination of hardware and software.
[0072] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.
Claims
1. A data processing method for reducing alarm noise based on dynamic rules, characterized in that: include: Get raw log data in various log formats; Based on the log format type of the original log data, configure the corresponding log parsing rules. Based on the configured log parsing rules, standardize the original log data of the corresponding log format type and extract the key information of the log content. Based on the predefined preliminary filtering rules, the key information of the log content is initially filtered to obtain the key information of the valid log content; Construct an event risk level dimension table, associate the event risk level dimension table with key information of valid log content, and construct event detail data containing risk level information; Normalize the event detail data to construct structured event data; Output all normalized structured event data to the corresponding target topic of the message queue; Configure the dynamic rules of operation and maintenance concern based on the web page. After the dynamic rules are configured, generate the corresponding FlinkSQL computing tasks. According to the structured event data under the target topic in the message queue consumed by the FlinkSQL computing tasks, obtain the denoised alarm data. The original log data includes first-category original log data and second-category original log data, wherein the source of the first-category original log data is the original log data reported by the host security program, the source of the second-category original log data is the original log data output by the terminal device, and the second-category original log data is JSON format data; The log format type based on the original log data configures the corresponding log parsing rules, including: For the first type of original log data, configure Syslog parsing rules as the corresponding log parsing rules; For the second type of raw log data, configure the JSON format mapping rule as the corresponding log parsing rule; The standardization process of the original log data of the corresponding log format type is performed to extract key information of the log content, including: For the first type of original log data, the Syslog parsing rules are used to parse the original log data of the corresponding log format type with PRI and host name as parsing conditions, and the content information of the first type of original log data is parsed out. According to the preset mapping rules, the key information of the first type of log content in JSON format is obtained; For the second type of original log data, the JSON format mapping rules are used to standardize the original log data of the corresponding log format type to obtain the key information of the second type of log content.
2. A data processing method for reducing alarm noise based on dynamic rules according to claim 1, characterized in that: The pre-defined preliminary filtering rules are used to perform preliminary filtering on the key information of the log content to obtain the effective key information of the log content, including: Based on the predefined preliminary filtering rules, the first category of log content key information and the second category of log content key information are respectively subjected to preliminary filtering processing to filter out invalid information that does not meet the preliminary filtering rules, thereby obtaining the first category of valid log content key information and the second category of valid log content key information.
3. A data processing method for reducing alarm noise based on dynamic rules according to claim 2, characterized in that: The event risk level dimension table is constructed, the event risk level dimension table is associated with the key information of the effective log content, and event detail data containing risk level information is constructed, including: Based on the key information of the first type of valid log content and the key information of the second type of valid log content, a first type of event risk level dimension table and a second type of event risk level dimension table are respectively constructed, wherein the first type of event risk level dimension table includes one or more first type of associated identification table items, wherein the first type of associated identification table items include first type of risk level values and first type of risk level information, and the second type of event risk level dimension table includes one or more second type of associated identification table items, wherein the second type of associated identification table items include one or more associated condition information, second type of risk level values and second type of risk level information; Based on the key information of the first-category effective log content, the corresponding first-category log risk level value is calculated. The specific calculation formula is as follows: Where: Level represents the risk level value of the first type of log, and Level is encoded in decimal, and the value range of Level is 0-5; PRI represents the priority of the log; Facility represents the type of program or device that generates the log, and Facility is encoded in decimal, and the value range of Facility is 0-22; Based on the mapping relationship formed by the first-class log risk level value and the field of the first-class risk level value of the first-class event risk level dimension table, the first-class event risk level dimension table and the first-class valid log content key information are associated to obtain the first-class risk level information corresponding to the first-class risk level value, and the first-class valid log content key information is annotated to construct the first-class event detailed data containing the first-class risk level information; Based on the second-category effective log content key information, according to a plurality of predefined associated type matching quantity intervals and the second-category log risk level value assigned to each associated type matching quantity interval, the content information field contained in the second-category effective log content key information is compared with the associated condition information field for type matching to obtain the type matching quantity result, the associated type matching quantity interval where the type matching quantity result is located is judged, and the interval judgment result and the second-category log risk level value corresponding to the interval judgment result are obtained, based on the mapping relationship formed between the second-category log risk level value and the field of the second-category risk level value of the second-category event risk level dimension table, the second-category event risk level dimension table is associated with the second-category effective log content key information to obtain the second-category risk level information corresponding to the second-category log risk level value, and the second-category effective log content key information is marked to construct the second-category event detailed data containing the second-category risk level information.
4. The data processing method for reducing alarm noise based on dynamic rules according to claim 3 is characterized in that: The normalization of the event detail data to construct structured event data includes: According to the mapping content information field contained in the preset data format, the first category event detail data and the second category event detail data are normalized respectively, and the first content information field and the second content information field required for the corresponding mapping content information field are extracted respectively; based on the extracted first content information field and the second content information field, the first structured event data and the second structured event data are constructed respectively.
5. A data processing system for reducing alarm noise based on dynamic rules, characterized in that: include: Log acquisition module, used to obtain raw log data of various log formats; A first data processing module is used to configure corresponding log parsing rules based on the log format type of the original log data, and based on the configured log parsing rules, perform standardization processing on the original log data of the corresponding log format type to extract key information of the log content; The second data processing module is used to perform preliminary filtering processing on the key information of the log content based on the predefined preliminary filtering rules to obtain the effective key information of the log content; An event detail data construction module is used to construct an event risk level dimension table, associate the event risk level dimension table with key information of valid log content, and construct event detail data containing risk level information; A structured event data construction module is used to normalize event detail data and construct structured event data; The writing module is used to output all normalized structured event data to the corresponding target topic of the message queue; The noise reduction module is used to configure dynamic rules of operation and maintenance concern based on the web page. After the dynamic rules are configured, the corresponding FlinkSQL computing tasks are generated. According to the structured event data under the target topic in the message queue consumed by the FlinkSQL computing tasks, the noise-reduced alarm data is obtained. The original log data includes first-category original log data and second-category original log data, wherein the source of the first-category original log data is the original log data reported by the host security program, the source of the second-category original log data is the original log data output by the terminal device, and the second-category original log data is JSON format data; The log format type based on the original log data configures the corresponding log parsing rules, including: For the first type of original log data, configure Syslog parsing rules as the corresponding log parsing rules; For the second type of raw log data, configure the JSON format mapping rule as the corresponding log parsing rule; The standardization process of the original log data of the corresponding log format type is performed to extract key information of the log content, including: For the first type of original log data, the Syslog parsing rules are used to parse the original log data of the corresponding log format type with PRI and host name as parsing conditions, and the content information of the first type of original log data is parsed out. According to the preset mapping rules, the key information of the first type of log content in JSON format is obtained; For the second type of original log data, the JSON format mapping rules are used to standardize the original log data of the corresponding log format type to obtain the key information of the second type of log content.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
