A method, apparatus, electronic device, and storage medium for filtering log data.

Abstract

This application provides a log data filtering method, apparatus, electronic device, and storage medium, relating to the field of data filtering technology. The method includes: using a SELECTOR parser to perform syntax validation on pre-written filtering rules; after successful validation, creating a filter based on the filtering rules; obtaining logs to be filtered; and using the filter to filter the logs. This method filters logs by setting filtering conditions, thereby saving storage space and reducing query pressure when collecting logs, solving the problem of large amounts of logs occupying storage space and the significant query pressure on security operations personnel when retrieving the required logs from massive amounts of data.

Description

Technical Field

[0001] This application relates to the field of data filtering technology, and more specifically, to a log data filtering method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rise and development of the internet, more and more devices have been connected to the network. These devices are diverse in type and function, and are used in a variety of business scenarios. The log records they generate during operation are also different and enormous. These countless logs occupy a large amount of storage space, and security operations personnel face significant query pressure in retrieving the logs they need from the massive amount of logs. Summary of the Invention

[0003] The purpose of this application is to provide a log data filtering method, device, electronic device, and storage medium. By setting filtering conditions to filter logs, storage space is saved when collecting logs, query pressure is reduced, and the problem of a large number of logs occupying storage space and security operation and maintenance personnel having great query pressure when obtaining the required logs from massive amounts of logs is solved.

[0004] This application provides a log data filtering method, the method comprising:

[0005] Use a SELECTOR parser to perform syntax validation on pre-written filtering rules;

[0006] After successful verification, a filter is created based on the filtering rules;

[0007] Get or receive logs to be filtered;

[0008] The filter is used to filter the logs to be filtered.

[0009] In the above implementation process, logs are filtered through pre-set filtering rules to save storage space and reduce query pressure when collecting logs. Furthermore, the SELECTOR language is used to achieve efficient and fast log filtering, which solves the problem that a large number of logs occupy storage space and that security operations and maintenance personnel face great query pressure when retrieving the required logs from massive amounts of logs.

[0010] Furthermore, the step of using a SELECTOR parser to perform syntax validation on the pre-written filtering rules includes:

[0011] The SELECTOR syntax parser is used to determine whether the filtering rules conform to the SELECTOR language expression, query filtering conditions, logical relational query filtering conditions, nested query filtering conditions, and assertion expression structure.

[0012] In the above implementation process, the SELECTOR syntax parser determines whether the filtering rules conform to the syntax rules of the SELECTOR language. Only SELECTOR language filtering rules with correct syntax can be correctly parsed by the SELECTOR language parser; otherwise, an error will be reported. Therefore, the syntax verification of filtering rules can be achieved.

[0013] Furthermore, before the step of creating a filter based on the filtering rules after the successful verification, the method further includes:

[0014] Store successfully validated filtering rules in a database or XML file.

[0015] In the above implementation process, the successfully verified filtering rules are persisted to prevent loss.

[0016] Further, the step of creating a filter based on the filtering rules includes:

[0017] Create a Selector component;

[0018] The Selector component generates corresponding expression units based on the assertion expressions in the filtering rules;

[0019] Generate a filter from the expression unit.

[0020] In the above implementation process, a corresponding filter is generated through the filtering rules, and the log data is filtered through the filter.

[0021] Furthermore, the step of filtering the logs to be filtered using the filter includes:

[0022] The logs to be filtered are set as a Map object with a key-value structure. The Map object includes the field names of the logs to be filtered and the field values ​​corresponding to the field names.

[0023] The filter performs rule matching on the field name and the field value to obtain log data that conforms to the filtering rules.

[0024] In the above implementation process, the log filtering is achieved by combining SELECTOR and KV object methods. This method is simple and flexible, and enables efficient and fast log data filtering. It solves the problem that existing log data filtering methods require writing complex and cumbersome filtering rules, resulting in poor readability.

[0025] This application embodiment also provides a log data filtering device, the device comprising:

[0026] The syntax validation module is used to perform syntax validation on pre-written filtering rules using the SELECTOR syntax parser.

[0027] The filter creation module is used to create a filter based on the filtering rules after successful verification.

[0028] The log acquisition and reception module is used to acquire or receive logs to be filtered.

[0029] The filtering module is used to filter the logs to be filtered using the filter.

[0030] In the above implementation process, logs are filtered through pre-set filtering rules to save storage space and reduce query pressure when collecting logs. Furthermore, the SELECTOR language is used to achieve efficient and fast log filtering, which solves the problem that a large number of logs occupy storage space and that security operations and maintenance personnel face great query pressure when retrieving the required logs from massive amounts of logs.

[0031] Furthermore, the syntax verification module includes:

[0032] The judgment module is used to determine whether the filtering rules conform to the SELECTOR language expression, query filtering conditions, logical relation query filtering conditions, nested query filtering conditions, and assertion expression structure using the SELECTOR syntax parser.

[0033] In the above implementation process, logs are filtered through pre-set filtering rules to save storage space and reduce query pressure when collecting logs. Furthermore, the SELECTOR language is used to achieve efficient and fast log filtering, which solves the problem that a large number of logs occupy storage space and that security operations and maintenance personnel face great query pressure when retrieving the required logs from massive amounts of logs.

[0034] Furthermore, the filtering module includes:

[0035] The log transformation module is used to set the logs to be filtered as a Map object with a key-value structure. The Map object includes the field names of the logs to be filtered and the field values ​​corresponding to the field names.

[0036] The rule matching module is used by the filter to perform rule matching on the field name and the field value to obtain log data that conforms to the filtering rules.

[0037] In the above implementation process, the log filtering is achieved by combining SELECTOR and KV object methods. This method is simple and flexible, and enables efficient and fast log data filtering. It solves the problem that existing log data filtering methods require writing complex and cumbersome filtering rules, resulting in poor readability.

[0038] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the log data filtering method described in any one of the above-described embodiments.

[0039] This application also provides a readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the log data filtering method described in any of the above-described embodiments. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a log data filtering method provided in this application embodiment;

[0042] Figure 2 A detailed flowchart of the log data filtering method provided in the embodiments of this application;

[0043] Figure 3 A flowchart for generating the filter provided in the embodiments of this application;

[0044] Figure 4 A log filtering flowchart provided for embodiments of this application;

[0045] Figure 5 A schematic diagram illustrating the specific screening process provided in this application embodiment;

[0046] Figure 6 A structural block diagram of the log data filtering device provided in the embodiments of this application;

[0047] Figure 7 This is a structural block diagram of another log data filtering device provided in an embodiment of this application.

[0048] icon:

[0049] 100 - Syntax validation module; 101 - Judgment module; 200 - Filter creation module; 201 - Selector component creation module; 202 - Expression unit generation module; 203 - Filter generation module; 300 - Log acquisition and reception module; 400 - Filtering module; 401 - Log transformation module; 402 - Rule matching module. Detailed Implementation

[0050] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0051] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0052] Example 1

[0053] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a log data filtering method provided in an embodiment of this application. Based on a self-developed SELECTOR query language, this method combines a key-value (KV) object approach to provide a log data filtering method based on the SELECTOR language. Compared to existing log data filtering methods that require writing complex and cumbersome filtering rules with poor readability, this method offers fine-grained filtering, powerful functionality, flexibility, and high efficiency.

[0054] This method uses the SELECTOR language, which is a DFL (Data Filter Language), and provides a framework based on the SELECTOR language. It combines SELECTOR with key-value object methods to achieve efficient and fast log data filtering.

[0055] The method specifically includes the following steps:

[0056] Step S100: Use the SELECTOR syntax parser to perform syntax validation on the pre-written filtering rules;

[0057] like Figure 2 The diagram shows the specific flowchart of the log data filtering method. A filtering rule is written in advance using the SELECTOR syntax, such as SELECTOR(src='127.0.0.1'AND dport='80').

[0058] Before performing log filtering, a filtering rule needs to be created according to the business purpose. This rule is written based on the SELECTOR language and should follow the SELECTOR syntax. In actual implementation, the log data filtering framework provides an editor for quickly writing rules. Meanwhile, when the editor displays existing filtering rules, it uses XML parsing. That is, after converting the filtering rule into an XML string, the expressions are parsed into corresponding XML nodes, and the SELECTOR syntax parser displays the nodes.

[0059] Before saving this filter rule, it should be syntax-checked to ensure its correctness. Syntax check is performed by the SELECTOR parser.

[0060] Step S200: After successful verification, create a filter based on the filtering rules;

[0061] Step S300: Obtain or receive logs to be filtered;

[0062] The logs to be filtered include those actively acquired and those passively received.

[0063] Step S400: Use the filter to filter the logs to be filtered.

[0064] Logs are filtered by pre-defined filtering rules to save storage space and reduce query pressure when collecting logs. The SELECTOR language is used to achieve efficient and fast log filtering, which solves the problem of a large number of logs occupying storage space and the large query pressure for security operations personnel to obtain the required logs from massive amounts of logs.

[0065] Specifically, step S100 includes:

[0066] The SELECTOR syntax parser is used to determine whether the filtering rules conform to the SELECTOR language expression, query filtering conditions, logical relational query filtering conditions, nested query filtering conditions, and assertion expression structure.

[0067] SELECTOR is a data filtering language that is concise, easy to learn and use, and provides a simple interface for operation and writing rules. Compared to common rule writing methods such as XML, SELECTOR offers a higher level of abstraction and richer data structures, such as multiple values ​​and nested rules.

[0068] The syntax rules for the SELECTOR language are as follows:

[0069] expression::=("SELECTOR"|"selector")"("expcondition")":

[0070] The expression represents a SELECTOR language expression. A valid SELECTOR expression must begin with the string "SELECTOR" or "selector", and its filter condition expcondition must be enclosed in parentheses.

[0071] expcondition::=expterm(("OR"|"or")expterm)*:

[0072] The expcondition represents the query filter condition, which consists of one or more expterm query filter units with an 'OR' relationship. Multiple expterms are connected by the string "OR" or "or", indicating that the logical relationship between the expterms is an 'OR' relationship.

[0073] expterm::=expfactor("AND"expfactor)*:

[0074] `expterm` is a unit that constitutes the query filtering conditions. Since multiple `expterm`s form an `OR` relation query filtering statement, it can be called an `OR` relation query filtering unit. It can be composed of `expfactor` query filtering units with higher logical priority, which are `AND` relation query filtering units. Multiple `expfactor`s are connected by the string "AND" or "and", indicating that the logical relationship between `expfactor`s is an `AND` relation.

[0075] expfactor::=[("NOT"|"not")]exppredicate:

[0076] `expfactor` is also a unit of query filtering conditions. Since multiple `expfactor` statements form an AND relationship query filtering statement, it can be called an AND relationship query filtering unit. In logical operations, the precedence of 'AND' is usually higher than that of 'OR'. Similarly, in the SELECTOR syntax specification, `todo:...`. `expfactor` consists of `[("NOT"|"not")]` and `exppredicate`, where `("NOT"|"not")` represents the 'NOT' operator, which can be represented by either "NOT" or "not", meaning the logical relation 'NOT'.

[0077] exppredicate::= <gid>predicate | "(expcondition)":

[0078] exppredicate consists of <gid>predicate or <lparen>expcondition <rparen>composition. <gid>It can be understood as a variable in a programming language; predicate represents the assertion expression supported by the SELECTOR language. <lparen>and <rparen>These represent the left and right parentheses, respectively, and the entire... <lparen>expcondition <rparen>This indicates nested query filtering conditions.

[0079] predicate::=comparisonPredicate

[0080] |betweenPredicate

[0081] |inPredicate

[0082] |likePredicate

[0083] |startwithPredicate

[0084] |nullPredicate;

[0085] A predicate is one of six types of assertion expressions: comparisonPredicate, betweenPredicate, inPredicate, likePredicate, startwithPredicate, and nullPredicate.

[0086] Assertion expressions specifically include the following six types:

[0087] comparisonPredicate::=( <eq> | <gt> | <lt> | <ge> | <le> | <ne> ) <value>;

[0088] comparisonPredicate: Indicates a comparison assertion, supporting equality (or equality). <eq>), greater than ( <gt>), less than ( <lt>), greater than or equal to ( <ge>), less than or equal to ( <le>), not equal to ( <ne>There are 6 comparison operators. The comparison operator is followed by the value to be compared, and then... <value>representation

[0089] betweenPredicate::=[("NOT"|"not")]("BETWEEN"|"between") <value>("AND"|"and") <value>;

[0090] betweenPredicate: This indicates the between...and... assertion, used to select a range of data between two values.

[0091] inPredicate::=[("NOT"|"not")]("IN"|"in")"(" <value> ("," <value>)*")";

[0092] inPredicate: Represents an in assertion, used to determine whether a value is one of a given set of values.

[0093] likePredicate::=[("NOT"|"not")]("LIKE"|"like") <string>;

[0094] ikePredicate: Represents a like assertion, used for fuzzy searches.

[0095] startwithPredicate::=[("NOT"|"not")]("STARTWITH"|"startwith") <string>;

[0096] `startwithPredicate` represents the `startwith` assertion, used to determine whether a string value begins with a specified string.

[0097] nullPredicate::=("IS"|"is")[("NOT"|"not")]("NULL"|"null")

[0098] nullPredicate represents the (is)null assertion, used to determine whether a value is null.

[0099] The filtering rules can be validated based on the above syntax rules. As can be seen from the syntax rules of the SELECTOR language, the SELECTOR language is logically clear, simple to read, and convenient for security operations and maintenance personnel to write and use. Furthermore, because the SELECTOR language allows nested statements, it has great flexibility and powerful filtering capabilities.

[0100] This application also provides a log data filtering framework based on the SELECTOR language, which works by combining SELECTOR with key-value object methods. This framework leverages the simplicity and flexibility of the SELECTOR language, primarily for efficient and rapid filtering of log data. It possesses a highly non-procedural nature, operating at a high-level data structure regardless of the data storage method. Therefore, theoretically, different data storage systems with different underlying structures can use this framework to implement data filtering functions. This not only significantly reduces the burden on security operations personnel but also improves data independence.

[0101] The log data filtering framework based on the SELECTOR language uses JavaCC as the specific implementation of the SELECTOR language. First, this invention provides a JavaCC syntax specification file, which defines the implementation of the SELECTOR language syntax rules, such as keywords and expression structures. Second, a JavaCC parser generator is used to parse the above syntax specification file to generate a SELECTOR language parser. Only syntactically correct SELECTOR language code can be correctly parsed by the SELECTOR language parser; otherwise, an error will occur.

[0102] The SELECTOR language is implemented in Java, and all assertion expressions should be implemented in Java.

[0103] The SELECTOR language in this application comprises two main categories of expression units: CompareExpression and LogicExpression. CompareExpression units are primarily used for comparing numerical values ​​and strings, while LogicExpression units perform other logical operations, such as logical AND and logical OR. In addition to supporting six basic comparison operations (greater than, less than, equal to, greater than or equal to, less than or equal to, and not equal to), the CompareExpression unit expands to include five more expression units, named semantically: BetweenExpression, InExpression, (Is)NullExpression, LikeExpression, and StartWithExpression. CompareExpression units perform value comparisons during data filtering. The LogicExpression unit is further divided into four expression units, named semantically: AndExpression, OrExpression, ParentExpression, and BracketExpression.

[0104] Each expression unit contains four basic operation units: left operand (lOperand), right operand (rOperand), operator, and not operator. The left and right operands can also be expression units. For example, the filtering rule `SELECTOR(a='A' and b>1)` can be constructed as an AND expression. Here, `and` is the operator, `a='A'` is the left operand, which is also a comparison expression, and `b>1` is the right operand, which is also a comparison expression. The left and right operands can be further split until they are reduced to atomic expressions (operations that cannot be further split). Since this invention only involves filtering log data and not querying log data results, when implementing the SELECTOR language, each expression unit only cares about the existence of the queried data and does not concern itself with other data behaviors.

[0105] Therefore, each expression unit provides a `match` method for matching log fields with filter conditions. The specific implementation of the `match` method differs across expression units. Taking the And expression unit (AndExpression) as an example, its `match` method pseudocode is defined as follows:

[0106] oolean match(logData){ / / logData is the log data

[0107] if the expression has not been fully constructed {

[0108] return false;

[0109] }

[0110] if lOperand.match(logData)&&rOperand.match(logData){

[0111] return true;

[0112] }

[0113] return false;

[0114] }

[0115] Prior to step S200, the method further includes:

[0116] Store successfully validated filtering rules in a database or XML file.

[0117] The filtering rules should be persisted to prevent loss. They can be persisted to a database or to an XML file, depending on the specific implementation; no specific limitation is made here.

[0118] like Figure 3 The process of generating a flowchart for the filter, specifically step S200, includes the following steps:

[0119] Step S201: Create the Selector component;

[0120] Step S202: The Selector component generates a corresponding expression unit based on the assertion expression in the filtering rule;

[0121] Step S203: Generate a filter from the expression unit.

[0122] The filter creation steps are as follows: First, create a Selector component to create the filter; then, obtain the filter rules (new settings or persistent reads), and the Selector component calls the SELECTOR syntax parser to verify whether the filter rules are correct; finally, the Selector component generates corresponding expression units for the assertion expressions in the filter rules and combines them into a filter.

[0123] like Figure 4 The diagram shows the log filtering flowchart. Step S400 specifically includes the following steps:

[0124] Step S401: Set the log to be filtered as a KV structure Map object, the Map object including the field name of the log to be filtered and the field value corresponding to the field name;

[0125] Step S402: The filter performs rule matching on the field name and the field value to obtain log data that conforms to the filtering rules.

[0126] In this application, the log data is wrapped into a key-value (KV) Map object using the key-value (KV) object method. If the Map object corresponding to the log matches the expression unit, it returns true; otherwise, it returns false. This allows for a single-time filtering of the log data.

[0127] In actual development, a raw log entry to be standardized is a key-value (KV) structure, reflected in a Map object as a data entity. The key (K) of this object should contain the names of all fields in the raw log, and may also include other fields such as collection type, device address, and log data object type. The value (V) is the actual value corresponding to these field names. All fields in the Map object can be used as log filtering conditions; the filter works by matching these fields and their values ​​according to rules.

[0128] Pass the log Map object to the filter. Logs that meet the filter rules will be returned true by the filter, otherwise false.

[0129] This application, based on the SELECTOR language, effectively combines the KV object method with data persistence methods to achieve fine-grained filtering of massive log data.

[0130] For example, the logs to be filtered include the following fields: device type (devType), time (time), source address (src), destination address (dst), source port (sport), destination port (dport), and protocol type (proto).

[0131] The filtering rules are as follows:

[0132] The `SELECTOR(src='127.0.0.1' AND dport='80')` statement filters log data originating from 127.0.0.1 and destined for port 80, discarding all other log data. After writing the rule, the `SELECTOR` parser checks its syntax; if it doesn't, it's not saved. Once validated, the rule is persisted to the database, and the `Selector` component then creates a filter based on it.

[0133] After the filter is created, the Map object of the log data is passed as a parameter to the match method.

[0134] The following is a sample log entry:

[0135] devType = firewall

[0136] time = "2018-06-05 15:47:48"

[0137] src=127.0.0.1

[0138] dst = 127.0.0.1

[0139] sport=8080

[0140] dport=80

[0141] proto = tcp;

[0142] Construct the corresponding Map object:

[0143] {

[0144] devType:"firewall",

[0145] time: "2018-06-05 15:47:48",

[0146] src:"127.0.0.1",

[0147] dst:”127.0.0.1”,

[0148] sport:8080,

[0149] dport:80,

[0150] proto:tcp

[0151] }

[0152] After the log sample was filtered, its src was '127.0.0.1' and dport was '80', which met the filtering rules, so it returned true.

[0153] The following is a sample log entry:

[0154] devType = firewall

[0155] time = "2018-06-05 15:57:48"

[0156] src=127.0.0.1

[0157] dst = 127.0.0.1

[0158] sport=8080

[0159] dport=90

[0160] proto = udp;

[0161] Construct the corresponding Map object:

[0162] {

[0163] devType:"firewall",

[0164] time: "2018-06-05 15:57:48",

[0165] src:"127.0.0.1",

[0166] dst:"127.0.0.1",

[0167] sport:8080,

[0168] dport:90,

[0169] proto:udp

[0170] }

[0171] After the log sample is matched and filtered, its src is '127.0.0.1' and dport is 90, which does not meet the filtering rules. Therefore, it will return false and the log sample will be discarded.

[0172] By performing the above steps on each log entry with business requirements, the log data will be filtered. Figure 5 The diagram shown illustrates the specific screening process.

[0173] The log query language and log data filtering method described in this application provide security personnel with flexibility in writing filtering rules, simplify the writing tasks for security operations and maintenance personnel, and save on operation and maintenance costs. Moreover, the filtering method has fine granularity and powerful filtering functions. The filtering methods provided are more flexible and universal, without the need for special adaptation to a certain type of log, and can quickly and efficiently filter log data.

[0174] The so-called flexibility means that users are only responsible for the filtering rules, can customize the filtering rules, and can delete or change the filtering rules at any time.

[0175] The term "universality" means that the filtering rules can be applied to all types of log data without limitation, and that the filtering rules are based on log fields rather than log types.

[0176] To effectively address the issue of log data filtering, this application provides a log data filtering language: the SELECTOR language—a DFL (Data Filter Language). It also provides a log data filtering framework based on the SELECTOR language, employing a comprehensive approach combining the SELECTOR language with key-value (KV) partitioning for efficient and rapid log filtering. The SELECTOR filtering framework supports logs from any device. Security operations personnel only need to define filtering conditions and apply them to the corresponding log devices to discard or retain logs that meet the conditions during log retrieval, thus saving storage space and reducing query pressure.

[0177] Example 2

[0178] This application provides a log data filtering device, applied to the log data filtering method described in Embodiment 1, such as... Figure 6 The diagram shown is a structural block diagram of a log data filtering device, which includes:

[0179] The syntax verification module 100 is used to perform syntax verification on pre-written filtering rules using a SELECTOR syntax parser.

[0180] Specifically, the SELECTOR syntax parser is used to determine whether the filtering rules conform to the SELECTOR language expression, query filtering conditions, logical relational query filtering conditions, nested query filtering conditions, and assertion expression structure.

[0181] The filter creation module 200 is used to create a filter based on the filtering rules after successful verification.

[0182] Log acquisition and reception module 300 is used to acquire or receive logs to be filtered.

[0183] The filtering module 400 is used to filter the logs to be filtered using the filter.

[0184] Among them, such as Figure 7 The diagram shown is a structural block diagram of another log data filtering device. Figure 6 Based on this, the syntax verification module 100 includes: a judgment module 101, which uses a SELECTOR syntax parser to determine whether the filtering rules conform to the SELECTOR language expression, query filtering conditions, logical relation query filtering conditions, nested query filtering conditions, and assertion expression structure.

[0185] Filter creation module 200 includes, but is not limited to:

[0186] Selector component creation module 201 is used to create Selector components;

[0187] The expression unit generation module 202 is used by the Selector component to generate corresponding expression units based on the assertion expressions in the filtering rules.

[0188] The filter generation module 203 is used to generate a filter from the expression unit.

[0189] Filtering module 400 includes, but is not limited to:

[0190] Log conversion module 401 is used to set the log to be filtered as a KV structure Map object, wherein the Map object includes the field name of the log to be filtered and the field value corresponding to the field name;

[0191] The rule matching module 402 is used by the filter to perform rule matching on the field name and the field value to obtain log data that conforms to the filtering rules.

[0192] Logs are filtered by pre-defined filtering rules to save storage space and reduce query pressure when collecting logs. The SELECTOR language is used to achieve efficient and fast log filtering, which solves the problem of a large number of logs occupying storage space and the large query pressure for security operations personnel to obtain the required logs from massive amounts of logs.

[0193] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the log data filtering method described in Embodiment 1.

[0194] This application also provides a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the log data filtering method described in embodiment 1.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0196] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0197] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0199] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0200] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.< / string> < / string> < / value> < / value> < / value> < / value> < / value> < / ne> < / le> < / ge> < / lt> < / gt> < / eq> < / value> < / ne> < / le> < / ge> < / lt> < / gt> < / eq> < / rparen> < / lparen> < / rparen> < / lparen> < / gid> < / rparen> < / lparen> < / gid> < / gid>

Claims

1. A method for filtering log data, characterized in that, The method includes: The SELECTOR syntax parser is used to perform syntax validation on the pre-written filtering rules. Specifically, the SELECTOR syntax parser is used to determine whether the filtering rules conform to the SELECTOR language expression, query filtering conditions, logical relational query filtering conditions, nested query filtering conditions, and assertion expression structure. After successful verification, a filter is created based on the filtering rules; Get or receive logs to be filtered; The filter is used to filter the logs to be filtered, specifically including: setting the logs to be filtered as a key-value structured Map object, the Map object including the field names of the logs to be filtered and the field values ​​corresponding to the field names, the field names including the various field names of the original logs, the collection type field, the device address field, and the log data object type field, and the field values ​​are the actual values ​​corresponding to the field names; the filter performs rule matching on the field names and the field values ​​to obtain log data that meets the filtering rules.

2. The log data filtering method according to claim 1, characterized in that, After the verification is successful, but before the step of creating a filter based on the filtering rules, the method further includes: Store successfully validated filtering rules in a database or XML file.

3. The log data filtering method according to claim 1, characterized in that, Creating a filter based on the filtering rules includes: Create a Selector component; The Selector component generates corresponding expression units based on the assertion expressions in the filtering rules; Generate a filter from the expression unit.

4. A log data filtering device, characterized in that, The device includes: The syntax validation module is used to perform syntax validation on pre-written filtering rules using the SELECTOR syntax parser. Specifically, it includes a judgment module, which uses the SELECTOR syntax parser to determine whether the filtering rules conform to the SELECTOR language expression, query filtering conditions, logical relation query filtering conditions, nested query filtering conditions, and assertion expression structure. The filter creation module is used to create a filter based on the filtering rules after successful verification. The log acquisition and reception module is used to acquire or receive logs to be filtered. A filtering module is used to filter the logs to be filtered using the filter. The filtering module includes: a log transformation module, used to set the logs to be filtered as a key-value structured Map object, the Map object including the field names of the logs to be filtered and the field values ​​corresponding to the field names, and the field values ​​being the actual values ​​corresponding to the field names; and a rule matching module, used by the filter to perform rule matching on the field names and the field values ​​to obtain log data that conforms to the filtering rules.

5. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the log data filtering method according to any one of claims 1 to 3.

6. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the log data filtering method according to any one of claims 1 to 3.

Images