An abnormal event log filtering method, device and storage medium

Abstract

The application discloses an abnormal event log filtering method and device and a storage medium, and is applied to a log server. The method comprises the following steps: acquiring a problem type corresponding to an abnormal event; determining subject information of the abnormal event and a transaction corresponding to the subject information according to the problem type; the transaction comprises at least two anchor points; determining an abnormal anchor point based on the at least two anchor points, and determining filtering condition data corresponding to the abnormal event based on abnormal data objects associated with the abnormal anchor point; and filtering log data output by a target anchor point except the abnormal anchor point in the at least two anchor points according to the filtering condition data, so as to obtain target log data, which can avoid that a large amount of logs submerge target information and improve the problem positioning efficiency of program running.

Description

Technical Field

[0001] This application relates to the field of log processing technology, specifically to a log filtering method, device, and storage medium for abnormal events. Background Technology

[0002] A logging system can generally be considered a collection of text-based runtime data output functions designed by software developers to track the actual working process of a program or to locate problems. In related technologies, log output uses a full print mode. However, in scenarios with massive sampling points, random fault objects, and processing links spanning multiple modules, full output leads to critical fault logs being overwhelmed by a large amount of redundant information, making it difficult for operations and maintenance personnel to quickly extract useful information. Furthermore, expired and excessive logs are easily automatically cleaned up by the system, resulting in the loss of fault scene data and making it impossible to trace the root cause of the problem. Summary of the Invention

[0003] This application provides a log filtering method, device, and storage medium for abnormal events, which can filter log data, effectively reduce redundant log output, and improve problem diagnosis efficiency.

[0004] Firstly, this application provides a log filtering method for abnormal events, the method comprising: Obtain the problem type corresponding to the abnormal event; The topic information of the abnormal event and the corresponding transaction are determined based on the problem type; the transaction includes at least two anchor points. An abnormal anchor point is determined based on the at least two anchor points, and the filter condition data corresponding to the abnormal event is determined based on the abnormal data object associated with the abnormal anchor point. Based on the filtering conditions, the log data output by the target anchor points (excluding the abnormal anchor points) from the at least two anchor points is filtered to obtain the target log data.

[0005] Secondly, this application also provides a log filtering device for abnormal events, the device comprising: The acquisition module is used to acquire the problem type corresponding to the abnormal event; The first determining module is used to determine the topic information of the abnormal event and the transaction corresponding to the topic information based on the problem type; the transaction includes at least two anchor points. The second determining module is used to determine an abnormal anchor point based on the at least two anchor points, and to determine the filtering condition data corresponding to the abnormal event based on the abnormal data object associated with the abnormal anchor point. The filtering module is used to filter the log data output by the target anchor points (excluding the abnormal anchor points) among the at least two anchor points according to the filtering condition data, so as to obtain the target log data.

[0006] Thirdly, this application also provides an electronic device, which includes a processor and a memory, wherein a computer program is stored in the memory and the computer program is configured to be executed by the processor to implement any of the methods described above.

[0007] Fourthly, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the method described above.

[0008] This application obtains the problem type corresponding to the abnormal event, matches the log category of the fault to be tracked, and then determines the topic information and the corresponding transaction containing at least two anchor points based on the problem type. This allows logs scattered across multiple modules to be uniformly incorporated into the same transaction link management, ensuring the contextual integrity of log filtering. Based on the abnormal anchor point and the associated abnormal data object, the application determines the filtering condition data, which can generate filtering conditions based on the actual runtime abnormality. Based on the filtering condition data, the application filters the log data of the target anchor point, which can extract the target log directly related to the abnormal event from the full log, avoiding redundant log interference or target information overload, and realizing full-link log filtering. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a log filtering method for abnormal events provided in an embodiment of this application; Figure 2 Another flowchart illustrating log filtering for abnormal events provided in this application embodiment; Figure 3 A timing diagram of anchor points and modules provided for embodiments of this application; Figure 4 A schematic diagram of a log filtering device for abnormal events provided in an embodiment of this application; Figure 5 This is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] Log systems suffer from two major pain points in practical applications: first, critical information requiring troubleshooting is not configured for log output; second, a large amount of irrelevant and redundant logs are output, interfering with troubleshooting. This application addresses the problem of excessive output of redundant logs.

[0012] In power monitoring sites, two typical faults frequently occur: First, some analog sampling points experience sudden, abnormal increases in values ​​at intervals of several days, making it impossible to distinguish whether the anomaly originates from the communication source or the subsequent Supervisory Control and Data Acquisition (SCADA) process. Second, field devices randomly disconnect at night, with the faulty equipment being unpredictable. This leads to unclear fault identification, and the number of similar devices and measurement points is enormous. Printing attribute data for all data objects would overwhelm valid logs with redundant information, making them difficult to view and prone to being deleted by the system due to excessive log volume or retention timeouts. Furthermore, the business logic is scattered across multiple modules, further increasing the difficulty of log tracing and filtering. Therefore, a mechanism is needed that outputs only logs of the objects of interest, enabling problem localization with a small but precise set of logs.

[0013] This embodiment provides a log filtering method for abnormal events, which can be applied to log servers. Figure 1 A flowchart illustrating a log filtering method for abnormal events provided in an embodiment of this application is shown below. Figure 1 As shown, the process includes steps 101 to 104: Step 101: Obtain the problem type corresponding to the abnormal event.

[0014] Abnormal events can be behaviors or states that require diagnosis and localization. Examples include abnormally high data values ​​or abnormally interrupted device connections. Problem types can be manually entered by the system administrator through the configuration interface; or they can be categorized and identified based on the characteristic descriptions of abnormal events using a preset rule base. For example, when a device's data value exceeds a preset threshold, the problem type can be identified as abnormal data fluctuation. This abnormal event is detected at the anomaly judgment point in the business logic, triggering the log filtering process. The log server can belong to a log system; the log system will be referred to as the system below.

[0015] Step 102: Determine the subject information of the abnormal event and the corresponding transaction based on the problem type; the transaction includes at least two anchor points.

[0016] A transaction can be a series of continuous processing steps completed within a specified time. Anchor points are locations in the transaction processing flow used to output log data. Each anchor point can record the data status or processing result at that location. For example, for the problem type of abnormal data fluctuation, the topic information can be identified as data collection and processing, and a topic number can be assigned. A transaction from the start of data collection to the end of data entry can be associated with the topic number.

[0017] Step 103: Determine the abnormal anchor point based on at least two anchor points, and determine the filter condition data corresponding to the abnormal event based on the abnormal data object associated with the abnormal anchor point.

[0018] In this context, an anomaly anchor can be a point among multiple anchors in a transaction where an anomaly event is detected or identified. Filtering condition data can be a set of rules or parameters extracted from the anomaly data object associated with the anomaly anchor. For example, filtering condition data can be the filtering body data object. As an example, at a data validation anchor, if a data value is detected that does not conform to business rules, then that data validation anchor is determined to be an anomaly anchor. The anomaly data object is the raw data output by the anomaly anchor when an anomaly is detected, and the filtering condition data can be fields extracted from this raw data and their corresponding values.

[0019] Step 104: Filter the log data output by the target anchor points (excluding abnormal anchor points) from at least two anchor points according to the filtering conditions to obtain the target log data.

[0020] After receiving log data from each anchor point, the log server categorizes the log data based on transaction identifiers. Filtering the logs output by the target anchor point using filtering conditions effectively reduces log redundancy and identifies log data directly related to abnormal events.

[0021] In some embodiments, at least two anchor points include a starting anchor point; Before filtering the log data output from the target anchor points (excluding anomalous anchor points) based on the filtering criteria data, the process also includes: The sequence number information of a transaction is determined based on the log data of the starting anchor point; Add the sequence number information to the log data output by at least two anchor points other than the starting anchor point to obtain the updated log data; Based on the filtering criteria, the log data output by the target anchor points (excluding abnormal anchor points) from at least two anchor points is filtered to obtain the target log data, including: The updated log data is first filtered based on the filtering conditions to obtain the initial log data. The initial log data is subjected to a second filtering process based on the filtering criteria to obtain the target log data.

[0022] The starting anchor point can be the beginning of a transaction or the log generation point corresponding to the initial execution phase. For example, the starting anchor point can be the transaction initiator. The log data at the starting anchor point can contain the initial context information of the transaction. The sequence number information can be the identifier of the transaction, for example, the transaction number. After obtaining the log data output by the starting anchor point, unique sequence number information can be extracted or generated from the log data.

[0023] To ensure that all logs within the same transaction are traceable, after obtaining the sequence number information, this sequence number information is injected or appended to the log data output by anchors other than the starting anchor point, so that all log data belonging to the same transaction will carry the same sequence number information, forming updated log data.

[0024] The first filtering process can be a preliminary screening of the updated log data after the addition of sequence information. It can use a small part of the information in the filtering condition data for screening. For example, it can perform coarse-grained filtering based on transaction sequence information, problem type, topic information, or abnormal event related fields to quickly exclude transaction logs that are not related to the current abnormal event, or to initially screen the log set that is highly related to the abnormal event topic. This can narrow down the data range for subsequent processing and improve processing efficiency.

[0025] The second filtering process is a finer-grained filtering based on the initial log data obtained from the first filtering process. It can utilize most of the information in the filtering condition data for selection, such as field values ​​or operation types. Through the second filtering process, log data directly related to the anomaly event can be identified and retained, while logs belonging to the same transaction but irrelevant to anomaly diagnosis can be removed, resulting in target log data for anomaly event diagnosis.

[0026] This application, when filtering logs for anomaly events, determines sequence information through a starting anchor point and injects this sequence information into the log data of all other anchor points within the same transaction. This effectively solves the problem of inaccurate correlation of log data due to the lack of a unified identifier in complex distributed environments, ensuring that all relevant logs within the same transaction can be identified. The two-stage filtering mechanism, involving both primary and secondary filtering, allows for coarse-grained screening followed by fine-grained matching, making the filtering process more accurate and efficient. This ensures that the acquired target log data fully reflects the transaction context of the anomaly event, eliminates redundant information, and improves the accuracy of anomaly event diagnosis and the ease of problem localization.

[0027] In some embodiments, determining the filtering condition data corresponding to the abnormal event based on the abnormal data object associated with the abnormal anchor point includes: Retrieve the exception data object associated with the exception anchor point; Extract the target field and its corresponding target value from the abnormal data object; the target field is used to locate the problem of the abnormal event. Determine the type of filtering operation corresponding to the target field based on the diagnostic requirements of abnormal events; Define the target field, target value, and filter operation type as the filter condition data.

[0028] The abnormal data object can be a data object automatically collected by the system or manually entered when or after an abnormal event occurs. It can be a data set containing detailed information about the abnormal event. The target field can be a field (key) in the abnormal data object used to identify or describe the key attributes of the abnormal event. For example, the target field can be the device identifier (ID) or telemetry point ID. The target value can be a specific numerical value or string (value) corresponding to the target field, such as device number A or telemetry point number B. The extraction process can be based on predefined parsing rules, pattern matching, keyword recognition, or structured data query, etc., and is not limited here. The target field and target value can form a key-value pair.

[0029] Diagnostic requirements reflect the desired outcome for users or the system when handling abnormal events. For example, they might want to retain all logs related to the anomaly or remove logs unrelated to the anomaly. The filtering operation type can be the type of processing that involves retaining or removing logs. For example, in a scenario where telemetry point values ​​abnormally spike, the anomaly anchor point could be an abnormal alarm point in the SCADA processing service. The abnormal data object can be the telemetry data entity where the anomaly occurred. The target fields can be the device ID and telemetry point ID. The target value can be the corresponding number between the abnormal device and the abnormal telemetry point. The filtering operation type can be a retention operation type. The device ID, telemetry point ID, the corresponding number between the abnormal device and the abnormal telemetry point, and the retention operation type can be defined as the filtering condition data.

[0030] This application can transform abnormal data objects into filtering condition data, enabling log filtering to directly match the abnormal event itself. It will not retain redundant or irrelevant logs, nor will it lose the log content necessary for troubleshooting, thereby improving the efficiency and accuracy of log filtering and accelerating the diagnosis, location, and handling of abnormal events.

[0031] In some embodiments, the updated log data includes first log data output by the abnormal anchor point and second log data output by the target anchor point; The updated log data is first filtered based on the filtering criteria to obtain initial log data, including: The identification information of the target anchor point is determined based on the first log data; Based on the topic information, sequence number information, and identification information, the target fields extracted from the abnormal data object are matched with the initial fields in the second log data to obtain the first result; If the first result shows that the target field matches the initial field, the initial log data is determined based on the initial field and its corresponding initial value.

[0032] The first log data consists of logs generated by the anomaly anchor point when an anomaly occurs, typically containing contextual information directly related to the anomaly event. Parsing the first log data allows extraction of the target anchor point's identification information, which uniquely identifies or associates with the target anchor point.

[0033] The topic information can be the business context of the abnormal event, ensuring that the match occurs within the correct business domain; the sequence number information is used to distinguish different transaction instances, avoiding confusion between different transactions; the identifier information can narrow the matching scope, processing only the log data of the target anchor directly related to the abnormal anchor. The target field can be a field extracted from the abnormal data object, used to describe the corresponding attributes of the abnormal event. The initial field can be a field that actually exists in the second log data. The matching process can be a direct comparison of field names, or a logical association through a predefined field mapping table to determine whether the target field and the initial field are consistent.

[0034] If the target field is confirmed to match an initial field in the second log data, it indicates that log data related to the abnormal event has been found in the second log data. Based on the successfully matched initial field and its corresponding value in the log data (i.e., the initial value), the initial log data is then filtered from the second log data.

[0035] This application determines the identification information of the target anchor point based on the first log data; according to the topic information, sequence number information and identification information, the target field extracted from the abnormal data is matched with the initial field in the second log data output by the target anchor point, which effectively solves the problem of difficulty in accurately identifying log data related to abnormal events in complex log data.

[0036] In some embodiments, the target field includes a target device identifier field and a target telemetry point identifier field; The target fields extracted from the abnormal data object are matched with the initial fields in the second log data to obtain the first result, including: If the initial field contains both a target device identifier field and a target telemetry point identifier field, then the first result is determined to be a match between the target field and the initial field. If the target device identifier field and the target telemetry point identifier field are not present in the initial fields, then the first result is determined to be that the target field and the initial field do not match.

[0037] The target device identifier field is used to uniquely identify a specific device; for example, the target device identifier field can be a device ID. The target telemetry point identifier field is used to uniquely identify a specific telemetry point or sensor data; for example, the target telemetry point identifier field can be a telemetry point ID.

[0038] If the target device identifier field and the target telemetry point identifier field exist in the initial field for matching, and both exist, then the first result is determined to be a match between the target field and the initial field. Conversely, if neither exists in the initial field, then the first result is determined to be a mismatch between the target field and the initial field.

[0039] This application sets the target fields as the target device identifier field and the target telemetry point identifier field. Only when both of the above identifier fields are present is the match considered successful. This can improve the accuracy of log data matching and avoid judgment errors or omissions due to incomplete information or improper matching.

[0040] In some embodiments, the target field includes a target device identifier field and a target telemetry point identifier field; the initial value includes an initial device identifier value and an initial telemetry point identifier value. The initial log data is determined based on the initial fields and their corresponding initial values, including: The initial device identifier value is determined from the initial log data based on the target device identifier field; The initial telemetry point identifier value is determined in the initial log data based on the target telemetry point identifier field.

[0041] The initial value can be the data content in the log data corresponding to the initial field. When the target field is determined to include both the target device identifier field and the target telemetry point identifier field, the corresponding initial values ​​are determined to be the initial device identifier value and the initial telemetry point identifier value. After receiving the log data, the log server parses the log data and identifies the field representing the device identifier in the log data according to predefined log format or pattern matching rules. After identifying the target device identifier field, the corresponding value is extracted and used as the initial device identifier value. After identifying the target telemetry point identifier field, the corresponding value is extracted and used as the initial telemetry point identifier value.

[0042] This application not only performs field matching, but also obtains the initial device identifier value and the initial telemetry point identifier value, and retrieves the corresponding values ​​from the initial log data, thereby improving the accuracy and effectiveness of log filtering. It can directly locate the device and telemetry point corresponding to the anomaly, providing reliable data support for anomaly investigation and problem tracing.

[0043] In some embodiments, the filtering operation type corresponding to the target field includes a retention operation type and a rejection operation type; The initial log data is subjected to a second filtering process based on the filtering criteria to obtain the target log data, including: The target value corresponding to the target field is matched with the initial value corresponding to the initial field in the initial log data to obtain the second result; If the second result is a successful match and the filtering operation type is a retention operation type, a retention operation is performed on the initial log data to obtain the target log data; If the second result is a successful match and the filtering operation type is a rejection operation, the rejection operation is performed on the initial log data to obtain the target log data; If the target value does not match the initial value in the second result representation, the target log data is obtained by removing data from the initial log data.

[0044] The "Retain" operation type indicator retains the corresponding log data upon successful matching, using it as the target log data. This allows for the differentiation of logs related to abnormal events and the filtering of logs to be analyzed. The "Remove" operation type indicator removes the corresponding log data from the initial log data upon successful matching. This can filter out logs unrelated to abnormal events or logs that hinder the investigation process.

[0045] The target value extracted from the abnormal data object is compared with the actual value in the initial log data. The matching process can be determined based on the actual situation, such as exact string matching, regular expression matching, numerical range matching, or semantic matching based on predefined rules. As an example, if the target value is device number A, and the initial value of an initial field in the initial log data is also device number A, then the match is successful.

[0046] When the target value successfully matches the initial value, and the preset filter operation type is "retain," it indicates that the initial log data is related to an abnormal event. The initial log data is then identified as the target log data, allowing for the collection of log content that meets the requirements and satisfies actual investigation needs. When the target value successfully matches the initial value, but the preset filter operation type is "remove," the initial log data is removed from the log data set to be processed. This filters out logs that match the criteria but do not need to be viewed, reducing interference from irrelevant information. When the target value and initial value do not match, regardless of the set filter operation type, the initial log data does not meet the filtering requirements and is directly removed, improving the accuracy and processing speed of log filtering.

[0047] This application categorizes filtering operations into retention and removal types, and defines log processing rules for different matching results and operation types. By comparing the target value with the initial value, and based on the preset operation type, it determines whether the initial log data should be retained. If the matching result is consistent and the retention operation is selected, the corresponding log is collected. If the matching result is consistent and the removal operation is selected, the corresponding log is removed. If the matching result is inconsistent, the corresponding log is removed. This reduces manual operation, lowers the risk of human error, and improves the speed and accuracy of anomaly analysis.

[0048] In some embodiments, the initial log data is subjected to a second filtering process based on filtering condition data to obtain target log data, including: Retrieve at least two filter condition data corresponding to the abnormal event; If at least one filtering condition corresponds to a filtering operation type of elimination, then the initial log data is eliminated to obtain the target log data. If at least two filter conditions correspond to filter operation types that are both retention operations, then the initial log data is retained to obtain the target log data.

[0049] Each filter condition contains at least one target field, at least one corresponding target value, and a corresponding filter operation type. If any filter condition determines that a log needs to be removed, the initial log data will be removed directly, even if the log meets other retention requirements, allowing for rapid cleanup of useless logs. The initial log data will only be retained if all filter conditions indicate that it should be retained, and no removal conditions are triggered, ensuring that the log content is highly correlated with the abnormal event and that the data results are accurate and reliable.

[0050] This application prioritizes removal operations to quickly filter out irrelevant logs and reduce interference from redundant information. Logs will only be retained if they meet all retention requirements and do not trigger any removal rules. The retained log content is tailored to anomaly investigation needs, improving the efficiency and accuracy of anomaly tracing, problem localization, and log auditing, and avoiding judgment errors and information gaps caused by single conditions or unclear logic.

[0051] In some embodiments, at least two anchor points include a starting anchor point, and the log data output by the starting anchor point carries a preset time; the method further includes: If log data is received from the starting anchor point, the duration of the transaction is determined by timing. If the duration is greater than or equal to the preset time, clean up the log data corresponding to the transaction; If the duration is less than the preset time, the target log data is stored in the preset database based on the cutoff anchor point among at least two anchor points; the preset database is used for tracing abnormal events, locating problems, and auditing logs.

[0052] The log data output by the starting anchor point carries a preset time. This allows the starting anchor point of a transaction to include a pre-defined time threshold in its log data generation. The preset time can be a fixed value configured by the system administrator based on business needs or historical experience, such as a timeout period set through configuration files, database parameters, or the system management interface. Alternatively, the preset time can be dynamically determined based on the transaction type, business scenario, or historical data analysis. For example, a longer preset time can be set for transactions known to be time-consuming, while a shorter preset time can be set for transactions with high real-time requirements.

[0053] When the log server receives log data from the starting anchor, it signifies the start of a new transaction. A timer is started to record the timestamp of the starting anchor log data as the transaction's start time, and the timer continuously calculates the difference between the current time and the start time to determine the transaction's duration. If the transaction's duration exceeds or reaches the preset time carried in the starting anchor log data, a cleanup operation is triggered. Cleanup operations may include deleting all log records related to the transaction from the cache or moving them to the database. Before performing cleanup, the system can selectively generate cleanup reports or alerts based on configuration to notify the administrator. The scope of cleanup is typically limited to all log data generated by the transaction from the starting anchor to the current time to ensure timely resource release.

[0054] This application enables the determination of transaction timeouts by including a preset time in the log data output from the starting anchor point and timing transactions. When the duration of a transaction reaches or exceeds the preset time, the corresponding log data can be cleaned up in a timely manner, effectively avoiding the problem of infinite accumulation of log data due to abnormally long running times or transactions that do not end normally. This reduces the operating pressure on the log server and improves the resource utilization and management efficiency of the log system.

[0055] If the duration of a transaction is less than a preset time, the transaction duration continues to be counted until a cutoff anchor is reached. Based on the cutoff anchor, a storage operation on the target log data is triggered. The cutoff anchor represents the end of the transaction. When log data output at the cutoff anchor is detected, this log data is used as a signal to trigger the storage operation.

[0056] The filtered target log data is persistently stored in a preset database. This database can be a relational database or a time-series database; no specific limitation is made here. During storage, the target log data synchronously saves metadata such as transaction sequence numbers, topic information, and anomaly anchor information. The traceability function is used to reconstruct the complete process and state of the abnormal event by querying the log data in the database, thus determining the root cause of the event. The problem localization function can utilize the stored log data, monitoring, and diagnostic tools to identify the components, lines of code, or configurations that caused the anomaly. The log auditing function can review records of system operations, user behavior, and data access.

[0057] This application stores target log data to a preset database based on the cutoff anchor point when the duration is less than the preset time, enhancing the log server's ability to trace abnormal events and enabling operations and maintenance personnel to perform more comprehensive and accurate problem localization and root cause analysis.

[0058] The following describes the log filtering method for abnormal events provided in the embodiments of this application. As an example, the log filtering method for abnormal events can specifically be a delayed output log method that supports forward and backward filtering based on transactions. Abnormal events may include abnormally high data values, and the problem type belongs to abnormal data fluctuation; topic information may include data collection and processing, and the topic information may correspond to a topic number; transactions may correspond to transaction numbers (i.e., sequence number information).

[0059] In related technologies, the program's processing involves the 61850 client module, the protocol library file (ProtocolLibrary, libprot.so) communication protocol framework library, the front-end agent service, and the SCADA processing service.

[0060] The processing logic for data objects differs in each module. As an example, the 61850 client module performs logic such as determining whether to send data or data from a sudden change or to send data via a general call for data based on the data value and quality code of the sampling point. The libprot.so communication protocol framework library performs unified processing before data is entered into the database, such as comparing the new data value and quality code with the previous data value and quality code in the pre-cached database and performing data entry processing.

[0061] Without modifying the program logic to accommodate logging needs, log calls are added to appropriate locations within the existing logic to perform the logging task. For example, in the previous three modules that print data from all points, the existing data mutation anomaly alarm logic is located in the SCADA processing service program, and log output is added at this location. The input parameters for this log printing include the various fields (data attributes) of the data object. Based on these fields, the log content from the preceding modules is filtered forward, retaining only all logs corresponding to data mutation anomalies.

[0062] To achieve this purpose, the following concepts are proposed: First, log topics and transactions: The log data to be tracked is defined within a topic-based transaction based on the problem type (e.g., telemetry data anomalies). A transaction is a continuous process completed within a specified time. It should be noted that the log filtering process described in this application is executed in the event of a problem.

[0063] Second, the method of transmitting transaction numbers and timeout: Data objects can carry their own transaction numbers. For data objects being imported into the database, the database can be modified to synchronously write the corresponding transaction numbers. Transaction numbers can be set at the data attribute level; updating an attribute or field of a piece of data corresponds to updating a transaction number. Each transaction has a timeout period, which is the maximum lifespan preset by the log server for the transaction. This timeout period is used to prevent log accumulation due to anomalies, ensuring the log server's data cannot be properly cleaned up, or to store it in a file or database. Specifically, the log data at the starting anchor point carries the timeout period. The log server starts timing after receiving the log data from the starting anchor point. If the timeout period exceeds the specified duration, the content is cleaned up.

[0064] Third, log anchors, starting anchors, intermediate anchors, and ending anchors: The position of each log output point is defined within a transaction. Anchors are integers that provide target anchors for log points with given filtering conditions. The starting anchor generates a transaction number and starts the transaction timer. The intermediate anchor outputs regular log data of the business process and generates updated log data. The ending anchor triggers the completion of filtering and executes the storage of target log data into files and databases.

[0065] Fourth, output data objects and filter data objects (filtering subject data objects and filter recipient data objects): Output data objects refer to data objects used only for outputting log content; filter subject data objects refer to the filter condition data thrown by the log output point for filtering target anchor points. The filter condition data is determined based on fields contained in the code context at the program exception point that are sufficient to accurately locate the problem. That is, at the location where the program has an abnormal event (for example, telemetry point data is greater than a preset threshold, causing the telemetry point data to alarm due to being abnormally large), an abnormal anchor point is set up, and the device ID and telemetry point ID related to the abnormal event are used as filter condition data.

[0066] The recipient data object is a data object used for comparison with data objects that are filtered by other anchors. It includes log entries containing recipient data objects. Each log entry pointing to its own anchor (precisely located based on the subject, transaction number, and target anchor) is filtered once (filtered each time it is received). The specific filtering principle is influenced by the filtering condition data (filtering the subject data object) and the type of filtering operation. Each log output point can simultaneously possess both subject filter data objects (filtering other anchors) and recipient filter data objects (filtered by other anchors). All of these are serializable data objects.

[0067] Fifth, filter target anchors: The log entry that outputs the filter conditions requires the transaction's filter target anchors as input, i.e., the list of target anchors explicitly input in the log function's parameters. Target anchors are any anchors other than those where an exception occurred; there can be one or more.

[0068] Sixth, basic filtering logic: At the target anchor point, the filtered receptor data object must completely match multiple key-value pairs with the filtered subject data object. Then, the corresponding operation is performed according to the retention or removal type provided by the filtering subject. When the system receives multiple filtered subject data objects, if the receptor data object completely matches the key-value pairs, and any filtering subject's filtering operation type is removal, the log data corresponding to that receptor object is directly removed. Log data corresponding to objects that do not complete the matching process, as well as objects marked for removal, are also removed.

[0069] Seventh, continuous filtering triggering: Before the log of the cutoff anchor is thrown, each log with a filter recipient object can continuously receive the main filter object issued by multiple abnormal anchors and perform filtering.

[0070] Eighth, upon receiving the log's cutoff anchor point, store the filtering results into the database on the log server.

[0071] By connecting the above concepts, we can obtain a log filtering method for abnormal events. Figure 2 Another flowchart illustrating the log filtering of abnormal events provided in this application embodiment is shown below. Figure 2 As shown, the transaction processing flow is as follows: The transaction begins, sequentially passing through the creation log transaction start anchor point output point, the creation log transaction intermediate anchor point output point, and the creation log transaction end anchor point output point, finally ending the transaction. Specifically, the creation log transaction start anchor point output point corresponds to the first anchor point (start anchor point) of the first transaction, the creation log transaction intermediate anchor point output point corresponds to the second anchor point of the first transaction, and the creation log transaction end anchor point output point corresponds to the end anchor point of the first transaction. Multiple transactions can run concurrently; for example, the first and second transactions in the first topic can be processed in parallel. Multiple subject categories are also supported; for example, it may include topics from the second topic to the nth topic.

[0072] Function prototype: Log(log level, log data, topic, anchor, position, filter target description data, recipient filter data object, lifetime timeout). Anchor positions: start anchor, intermediate anchor, and end anchor.

[0073] Filter target description data: list: {target anchor point, main filter data object, filter operation type}; {Target anchor point, main filtered data object, filter operation type}; {Target anchor point, main filter data object, filter operation type}.

[0074] Filtering operation type: Keep or remove, defines how the log object is output when the key-value pair at the target anchor point is consistent.

[0075] Figure 3 This is a timing diagram of the anchor points and modules provided in the embodiments of this application, such as... Figure 3 As shown, the first module (e.g., the 61850 client module) corresponds to the first anchor point, the second module (e.g., the libprot.so communication protocol framework library) corresponds to the second anchor point, the third module (e.g., the front-end agent service / SCADA processing service) corresponds to the third anchor point, and the nth module (e.g., the SCADA processing service) corresponds to the mth anchor point. The third anchor point acts as the filtering body, filtering logs from the first and second anchor points forward, and also filtering logs from the mth anchor point backward. To track the data processing process in the software, log output points can be set at the locations where abnormal data objects occur (the third anchor point).

[0076] The parameters included in the third anchor (parameters of the log output function) are: 1) topic, transaction number (obtained from the message itself or read from the database); 2) its own anchor; 3) the output log data object; 4) the body filter data object; 5) the target anchor list; 6) the recipient filter data object (if needed); and 7) the merging method of the receiving filter conditions (if needed).

[0077] The first, second, and m-th anchor points are the log output points that accept the filter from the third anchor point. The parameters (parameters of the log output function) include: 1) topic, transaction number (obtained from the message itself or read from the database); 2) its own anchor point; 3) the output log data object (a data object specifically used to output log content that corresponds one-to-one with the filter recipient data object); 4) the recipient filter data object; and 5) the merging method of the accepting filter conditions.

[0078] Here's an example of filtering the subject data object and the recipient data object: At a certain log output anchor, the subject data object is {"device_id":100025, "yc_id":64000078}, which is used as the input parameter of the log function. After receiving it, the log server finds the recipient data object based on the topic, transaction number, and the target anchor it points to. The requirement is to find the corresponding key-value combination in the recipient object (which must contain {"device_id":100025, "yc_id":64000078}; the recipient may have more fields, but not fewer), and the corresponding values ​​must also correspond to the same log entry.

[0079] After receiving logs from log output points, the log server does not immediately write the content to a file. Instead, it caches the content on intermediate storage and categorizes the logs into different job queues based on the topic and transaction number. The log server then filters each log output point within the transaction number that has a filterable data object (the one containing the subject filter object) using all the anchor values ​​it points to. Combining this with filtering and merging conditions, only a portion of the log content is retained. After the log output point outputs the log, the log server directly inputs the filtering results into a separate file, completing the log persistence.

[0080] Figure 4 A schematic diagram of a log filtering device for abnormal events provided in an embodiment of this application is shown below. Figure 4 As shown, this application also provides a log filtering device for abnormal events, which can be applied to a log server. The device 400 includes: The acquisition module 401 is used to acquire the problem type corresponding to the abnormal event; The first determining module 402 is used to determine the topic information of the abnormal event and the transaction corresponding to the topic information according to the problem type; the transaction includes at least two anchor points. The second determining module 403 is used to determine an abnormal anchor point based on the at least two anchor points, and to determine the filtering condition data corresponding to the abnormal event based on the abnormal data object associated with the abnormal anchor point. The filtering module 404 is used to filter the log data output by the target anchor point (excluding the abnormal anchor point) among the at least two anchor points according to the filtering condition data, so as to obtain the target log data.

[0081] To implement the method of the embodiments of this application, Figure 5 A schematic diagram of a hardware structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown in the illustration, this application embodiment also provides an electronic device 50 that may include: a memory 501 for storing a computer program; and a processor 502 for implementing the method described above when executing the computer program. The processor 502 can implement the steps of any of the methods described above, which will not be elaborated further here.

[0082] Of course, in practical applications, such as Figure 5 As shown, the electronic device 50 may further include at least one network interface 503. Various components in the electronic device are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5Various buses are labeled as bus systems 504. The number of processors 502 can be at least one. Network interface 503 is used for wired or wireless communication between electronic devices and other devices. Memory 501 in this embodiment is used to store various types of data to support the operation of the electronic device. The methods disclosed in the above embodiments can be applied to processor 502, or implemented by processor 502. Processor 502 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 502 or by instructions in software form. The processor 502 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 502 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected in the combined execution of hardware and software modules in a microcontroller. The software module may reside in a storage medium located in memory 501. Processor 502 reads information from memory 501 and, in conjunction with its hardware, completes the steps of the aforementioned method. In an exemplary embodiment, electronic device 50 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0083] Specifically, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, such as a memory 501 storing the computer program, which can be executed by a processor 502 to complete the aforementioned method steps. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0084] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0085] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0087] It should be noted that, in the data processing stage, the technical solution of this application has strictly limited the scope of data collection to the minimum necessary to achieve the technical objectives, preventing the acquisition of irrelevant information. For any user information to be collected, the data subject will be clearly informed and their consent obtained. Furthermore, technologies such as encrypted storage and access control are employed to strengthen data security and ensure the security and compliance of the entire data processing process. The technical model and decision-making mechanism are based on objective technical parameters and do not introduce unnecessary parameters such as gender or age that may lead to discrimination, resolutely eliminating algorithmic discrimination and upholding public order and good morals. In addition, the specification fully describes the technical implementation methods, application scenarios, and compliance protection details. The claims are consistent with the content of the specification, key compliance designs are clear and verifiable, and the overall technical design is guided by the protection of public interests and adherence to social ethics, without any circumstances that harm public interests or violate public order and good morals.

[0088] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A log filtering method for abnormal events, characterized in that, The method includes: Obtain the problem type corresponding to the abnormal event; The topic information of the abnormal event and the corresponding transaction are determined based on the problem type; the transaction includes at least two anchor points. An abnormal anchor point is determined based on the at least two anchor points, and the filtering condition data corresponding to the abnormal event is determined based on the abnormal data object associated with the abnormal anchor point; the log data output by the target anchor point other than the abnormal anchor point among the at least two anchor points is filtered according to the filtering condition data to obtain the target log data.

2. The method according to claim 1, characterized in that, The at least two anchor points include the starting anchor point; Before filtering the log data output by the target anchor points (excluding the abnormal anchor points) from the at least two anchor points according to the filtering condition data, the method further includes: The sequence number information of the transaction is determined based on the log data of the starting anchor point; The sequence number information is added to the log data output by the anchor points other than the starting anchor point among the at least two anchor points to obtain the updated log data; The step of filtering the log data output by the target anchor points (excluding the abnormal anchor points) from the at least two anchor points according to the filtering condition data to obtain the target log data includes: Based on the filtering condition data, the updated log data is subjected to a first filtering process to obtain initial log data; The initial log data is subjected to a second filtering process based on the filtering condition data to obtain the target log data.

3. The method according to claim 2, characterized in that, The step of determining the filtering condition data corresponding to the abnormal event based on the abnormal data object associated with the abnormal anchor point includes: Obtain the abnormal data object associated with the abnormal anchor point; Extract the target field and the target value corresponding to the target field from the abnormal data object; the target field is used to locate the problem of the abnormal event. The type of filtering operation corresponding to the target field is determined based on the diagnostic requirements of the abnormal event. The target field, the target value, and the filtering operation type are determined as the filtering condition data.

4. The method according to claim 2, characterized in that, The updated log data includes the first log data output by the abnormal anchor point and the second log data output by the target anchor point; The first filtering process, based on the filtering condition data, is applied to the updated log data to obtain initial log data, including: The identification information of the target anchor point is determined based on the first log data; Based on the topic information, the sequence number information, and the identification information, the target fields extracted from the abnormal data object are matched with the initial fields in the second log data to obtain a first result; If the first result shows that the target field matches the initial field, the initial log data is determined based on the initial field and the initial value corresponding to the initial field.

5. The method according to claim 4, characterized in that, The target fields include the target device identifier field and the target telemetry point identifier field; The step of matching the target field extracted from the abnormal data object with the initial field in the second log data to obtain a first result includes: If the target device identifier field and the target telemetry point identifier field exist in the initial field, then the first result is determined to be that the target field matches the initial field; If the target device identifier field and the target telemetry point identifier field are not present in the initial field, then the first result is determined to be that the target field does not match the initial field.

6. The method according to claim 4, characterized in that, The target field includes a target device identifier field and a target telemetry point identifier field; the initial value includes an initial device identifier value and an initial telemetry point identifier value; The step of determining the initial log data based on the initial field and the initial value corresponding to the initial field includes: The initial device identifier value is determined from the initial log data based on the target device identifier field. The initial telemetry point identifier value is determined in the initial log data based on the target telemetry point identifier field.

7. The method according to claim 4, characterized in that, The filtering operation types corresponding to the target field include retention operation types and removal operation types; The step of performing a second filtering process on the initial log data based on the filtering condition data to obtain the target log data includes: The target value corresponding to the target field is matched with the initial value corresponding to the initial field in the initial log data to obtain the second result; If the second result is a successful match and the filtering operation type is a retention operation type, a retention operation is performed on the initial log data to obtain the target log data; When the second result is a successful match and the filtering operation type is a rejection operation type, a rejection operation is performed on the initial log data to obtain the target log data; If the second result indicates that the target value does not match the initial value, the target log data is obtained by performing a removal operation on the initial log data.

8. The method according to claim 7, characterized in that, The step of performing a second filtering process on the initial log data based on the filtering condition data to obtain the target log data includes: Obtain at least two filter condition data corresponding to the abnormal event; If at least one of the filtering condition data corresponds to a filtering operation type of elimination, then the initial log data is eliminated to obtain the target log data; If the filtering operation type corresponding to at least two of the filtering condition data is a retention operation type, then the initial log data is retained to obtain the target log data.

9. The method according to claim 1, characterized in that, The at least two anchor points include a starting anchor point, and the log data output by the starting anchor point carries a preset time; the method further includes: If log data is received from the starting anchor point, the duration of the transaction is determined by timing. If the duration is greater than or equal to the preset time, the log data corresponding to the transaction is cleared. If the duration is less than the preset time, the target log data is stored in a preset database based on the cutoff anchor point among the at least two anchor points; the preset database is used for tracing the abnormal event, locating the problem, and auditing the logs.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing a computer program configured to be executed by the processor to implement the method of any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, is used to implement the method according to any one of claims 1 to 9.

Images