Data analysis method, device and equipment based on self-defined strategy and storage medium

Abstract

The application relates to the field of data analysis, in particular to a data analysis method and device based on a self-defined strategy, a computer device and a storage medium.The method comprises the following steps: analyzing a self-defined strategy obtained from a first database to generate a corresponding SQL statement; analyzing target data to obtain a data analysis result; determining a first analysis result based on the SQL statement and the data analysis result; and determining a second analysis result based on the first analysis result and a clue strategy.The application can meet the needs of users for data analysis by using the self-defined strategy of the users, can quickly analyze data by using the SQL statement, and can accurately obtain the data analysis result by further matching the first analysis result with the clue strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data analysis, and in particular to a data analysis method and device based on a custom strategy, a computer device and a storage medium. BACKGROUND

[0002] In the process of network security event analysis, a large amount of multi-dimensional environmental data usually needs to be faced. If only personal experience is used for analysis, the result will depend on the individual level, key information is easy to be missed, and complex correlation analysis is difficult to be carried out.

[0003] The prior art does not have a method suitable for analyzing a large amount of data. SUMMARY

[0004] Therefore, it is necessary to provide a data analysis method and device based on a custom strategy, a computer device and a storage medium in view of the above technical problems.

[0005] In a first aspect, the present application provides a data analysis method based on a custom strategy, which comprises:

[0006] parsing a custom strategy obtained from a first database to generate a corresponding SQL statement; and

[0007] parsing target data to obtain a data parsing result;

[0008] determining a first analysis result based on the SQL statement and the data parsing result;

[0009] determining a second analysis result based on the first analysis result and a clue strategy.

[0010] In an embodiment, the parsing of the custom strategy obtained from the first database to generate the corresponding SQL statement comprises:

[0011] obtaining a custom strategy from a first database, wherein the first database is a MYSQL database;

[0012] mapping the custom strategy to a node of a syntax tree to generate a corresponding SQL statement.

[0013] In an embodiment, the custom strategy comprises a basic condition and an associated condition, and the mapping of the custom strategy to the syntax tree to generate the corresponding SQL statement comprises:

[0014] determining an expression of the basic condition and the associated condition;

[0015] mapping expressions of the basic condition and the associated condition to nodes of a syntax tree respectively to generate corresponding SQL statements, wherein the SQL statement corresponding to the basic condition is taken as a subquery statement.

[0016] In an embodiment, the parsing of the target data to obtain a data parsing result comprises:

[0017] The target data is parsed to obtain collected data and log data.

[0018] The cleaned log data and the collected data are stored in a second database; the second database is a SparkSQL database.

[0019] In an embodiment, the determining of a first analysis result based on the SQL statement and the data parsing result comprises:

[0020] The data parsing result is grouped and aggregated based on the SQL statement to generate a plurality of sub-data sets.

[0021] The first analysis result is generated based on the SQL statement and the plurality of sub-data sets.

[0022] In an embodiment, the determining of a second analysis result based on the first analysis result and a lead strategy comprises:

[0023] The plurality of sub-data sets contained in the first analysis result are matched with the lead strategy.

[0024] The matched sub-data sets are determined as key behavior data, and the unmatched sub-data sets are determined as non-key behavior data.

[0025] In an embodiment, the method further comprises:

[0026] The key behavior data and the non-key behavior data are labeled.

[0027] The labeled key behavior data and non-key behavior data are stored in the first database.

[0028] In a second aspect, an embodiment of the present application provides a data analysis device based on a self-defined strategy, which comprises:

[0029] A first parsing module is configured to parse a self-defined strategy obtained from a first database to generate a corresponding SQL statement; and

[0030] A second parsing module is configured to parse target data to obtain a data parsing result.

[0031] A first determining module is configured to determine a first analysis result based on the SQL statement and the data parsing result.

[0032] The second determining module is configured to determine a second analysis result based on the first analysis result and a clue strategy.

[0033] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor executes the steps of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the processor executes the computer program to implement the steps of the first aspect.

[0035] Compared with the prior art, the above method, device, computer device and storage medium, the custom strategy obtained from the first database is parsed to generate a corresponding SQL statement, the target data is parsed to obtain a data analysis result, the first analysis result is determined based on the SQL statement and the data analysis result, and the second analysis result is determined based on the first analysis result and the clue strategy. The present application can meet the needs of users for data analysis by using the custom strategy of the user, can quickly analyze data by using the SQL statement, and can accurately obtain the data analysis result by further matching the first analysis result with the clue strategy. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 FIG. 1 is a structural schematic diagram of an application environment of a data analysis method based on a custom strategy in an embodiment;

[0037] Figure 2 FIG. 2 is a flowchart of a data analysis method based on a custom strategy in an embodiment;

[0038] Figure 3 FIG. 3 is a flowchart of generating a SQL statement in an embodiment;

[0039] Figure 4 FIG. 4 is a flowchart of parsing target data in an embodiment;

[0040] Figure 5 FIG. 5 is a flowchart of generating a first analysis result in an embodiment;

[0041] Figure 6 FIG. 6 is a flowchart of matching a data set and a clue strategy in an embodiment;

[0042] Figure 7 FIG. 7 is a module connection schematic diagram of a data analysis device based on a custom strategy in an embodiment;

[0043] Figure 8 FIG. 8 is a structural schematic diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given to the drawings needed in the embodiments description. Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can be applied to other similar scenarios without creative labor on the basis of these drawings. Unless the context clearly indicates otherwise or otherwise stated, the same reference numbers in the drawings represent the same structures or operations.

[0045] As shown in the present application and claims, unless the context clearly indicates otherwise or otherwise stated, the words "one", "a", "an", and / or "the" do not specifically refer to the singular, but can also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0046] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the computing device and / or processor. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0047] It should be understood that when a unit or module is described as "connected", "coupled" to other units, modules or blocks, it can mean direct connection or coupling, or communication with other units, modules or blocks, or there can be intermediate units, modules or blocks, unless the context clearly indicates otherwise. The term "and / or" used herein can include any and all combinations of one or more related listed items.

[0048] The data analysis method based on custom strategy provided in the present application can be applied to the application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through the network. The terminal 102 parses the custom strategy obtained from the first database to generate a corresponding SQL statement, and parses the target data to obtain a data parsing result. Based on the SQL statement and the data parsing result, a first analysis result is determined. Based on the first analysis result and the clue strategy, a second analysis result is determined. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0049] In an embodiment, as shown in Figure 2As shown, a data analysis method based on a custom strategy is provided, and the method is applied to Figure 1 an application environment in the first database as an example for illustration, including the following steps:

[0050] S201: parsing the custom strategy obtained from the first database to generate a corresponding SQL statement; and parsing the target data to obtain a data parsing result.

[0051] The user enters the logic of the analysis, and the optional data dimensions and fields are provided by the system, but the association and filtering logic is not limited, and is customized by the user, and only part of the specification is checked and limited. After the custom strategy is entered, it is translated into a specific SQL statement of the SparkSQL library by the parsing engine.

[0052] The target data is, for example, asset data.

[0053] S202: determining a first analysis result based on the SQL statement and the data parsing result;

[0054] S203: determining a second analysis result based on the first analysis result and a lead strategy.

[0055] Based on steps S201-S203, the custom strategy obtained from the first database is parsed to generate a corresponding SQL statement; and the target data is parsed to obtain a data parsing result; a first analysis result is determined based on the SQL statement and the data parsing result; and a second analysis result is determined based on the first analysis result and a lead strategy. The present application utilizes the custom strategy of the user to meet the needs of the user for data analysis, utilizes the SQL statement to quickly analyze the data, and utilizes the further matching of the lead strategy with the first analysis result to accurately obtain the data analysis result.

[0056] In an embodiment, as Figure 3 shown, the parsing of the custom strategy obtained from the first database to generate a corresponding SQL statement includes the following steps:

[0057] S301: obtaining a custom strategy from a first database, wherein the first database is a MYSQL database;

[0058] S302: mapping the custom strategy to a node of a syntax tree to generate a corresponding SQL statement.

[0059] The custom strategy is obtained from the MYSQL database, each custom strategy includes multiple basic conditions and an association condition. Each condition includes multiple expressions. Each expression is composed of a data tuple (second column), an attribute (third column), an operator, a value, and association logic (eighth column).

[0060] Specifically comprising the following steps:

[0061] A, query the custom strategy and the basic conditions and associated conditions contained therein from the MYSQL database;

[0062] B, parse the basic condition expression: first group the expression according to the bracket level, and since the brackets may be nested, the filtering condition needs to be parsed recursively;

[0063] C, build an AST abstract syntax tree: map each part of the expression to a node of the syntax tree, data tuple -> table name, attribute -> field, operator + value -> constraint condition;

[0064] For example: SELECT name, path, createtime, lastvisittime FROM t_file_trace WHERE name IS NOT NULL AND name like '%baidu%'.

[0065] D, parse the associated condition expression, take the basic condition SQL statement as a subquery statement, and translate the associated condition expression into an associated SQL statement.

[0066] In an embodiment, as shown in Figure 4 The parsing of the target data to obtain the data parsing result includes the following steps:

[0067] S401: parsing the target data to obtain the collected data and the log data;

[0068] S402: storing the cleaned log data and the collected data into a second database; the second database is a SparkSQL database.

[0069] It should be noted that the Spark platform has strong scalability. It can be as small as single-machine deployment, running in a single notebook, and submitting tasks to the local node for processing. It can also be expanded to cluster deployment, with multiple servers sharing data processing and computing pressure without modifying the program code.

[0070] It should be further noted that the SparkSQL database is a memory-based computing engine, which is more efficient than file-based computing engines such as Hive, and has more extensive data sources, better compatibility and scalability.

[0071] The fixed format data collected by the data collector is called collected data. The data collector will output the information data of the environment as fixed pattern XML according to the agreement, such as user login records, process status and other information. The collected data is high-quality data that does not need to be cleaned and can be directly stored in the database.

[0072] Various log raw files in the environment are called log data. The log data is classified by source, such as Tomcat, Weblogic, etc., and then is cleaned, filtered and formatted. Qualified data will enter a SparkSQL database for storage. Taking Tomcat logs as an example, the logs will be regularly matched according to the standard log format of a certain version of Tomcat. Rows that cannot be matched will be discarded, and rows that can be matched will be divided and useful data will be extracted.

[0073] Both log data and collected data will be converted into corresponding entity objects to form fixed table structures for query. For example, collected data will be converted into objects and tables with fields such as name, localgroup, domain, state, des, hide, risk, etc.

[0074] In an embodiment, as shown in Figure 5 determining a first analysis result based on the SQL statement and the data analysis result includes:

[0075] S501: grouping and aggregating the data analysis result based on the SQL statement to generate a plurality of sub-data sets;

[0076] S502: generating a first analysis result based on the SQL statement and the plurality of sub-data sets.

[0077] The data analysis result is grouped and aggregated according to the fields used in the association condition to form a plurality of sub-data sets. Each sub-data set forms a suspicious behavior, and each row of the sub-data set is a behavior trace under the behavior. The placeholders in the behavior description rule template and the disposition suggestion template are replaced with the data of the sub-data set, and then backfilled into the behavior description rule and the disposition suggestion.

[0078] In an embodiment, as shown in Figure 6 determining a second analysis result based on the first analysis result and a clue strategy includes:

[0079] S601: matching a plurality of sub-data sets contained in the first analysis result with the clue strategy;

[0080] S602: determining the matched sub-data sets as key behavior data; and determining the unmatched sub-data sets as non-key behavior data.

[0081] The sub-data sets are associated with the clue strategy according to the association condition. If a certain row or several rows hit the clue, the corresponding behavior is marked as key behavior data.

[0082] After the first analysis result is processed, a series of suspicious behaviors are formed, and specific disposal suggestions are given for each suspicious behavior. The suspicious behaviors include all original data forming the result (referred to as behavior traces), and the user can view the corresponding source to verify the second analysis result.

[0083] In an embodiment, after determining the key behavior data and the non-key behavior data, the key behavior data and the non-key behavior data are marked, and the marked key behavior data and the non-key behavior data are stored in the first database.

[0084] Based on the above data analysis, the user can view the first analysis result, the second analysis result, and the marking result.

[0085] If the analysis result does not match the expectation, the data analysis can be performed again after adjusting the custom policy.

[0086] It should be understood that although each step in the above flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0087] In an embodiment, as shown in FIG. 7, the present application provides a data analysis device based on a custom policy, which comprises: Figure 7

[0088] A first analysis module 702 is configured to analyze the custom policy obtained from the first database to generate a corresponding SQL statement; and

[0089] A second analysis module 704 is configured to analyze the target data to obtain a data analysis result.

[0090] A first determination module 706 is configured to determine a first analysis result based on the SQL statement and the data analysis result.

[0091] A second determination module 708 is configured to determine a second analysis result based on the first analysis result and the clue policy.

[0092] In an embodiment, the first analysis module comprises:

[0093] ​The acquisition module is configured to acquire the customized strategy from a first database, which is a MYSQL database.

[0094] The mapping module is configured to map the customized strategy to a node of a syntax tree to generate a corresponding SQL statement.

[0095] In an embodiment, the customized strategy includes a basic condition and an associated condition, and the mapping module includes:

[0096] The third determination module is configured to determine expressions of the basic condition and the associated condition.

[0097] The mapping submodule is configured to map the expressions of the basic condition and the associated condition to nodes of a syntax tree respectively to generate corresponding SQL statements, wherein the SQL statement corresponding to the basic condition is used as a subquery statement.

[0098] In an embodiment, the second parsing module includes:

[0099] The second parsing submodule is configured to parse the target data to obtain collected data and log data.

[0100] The storage module is configured to store the cleaned log data and the collected data into a second database, which is a SparkSQL database.

[0101] In an embodiment, the first determination module includes:

[0102] The grouping and aggregation module is configured to group and aggregate the data parsing results based on the SQL statement to generate a plurality of sub-data sets.

[0103] The first determination submodule is configured to generate a first analysis result based on the SQL statement and the plurality of sub-data sets.

[0104] In an embodiment, the second determination module includes:

[0105] The matching module is configured to match a plurality of sub-data sets contained in the first analysis result with the lead strategy.

[0106] The second determination submodule is configured to determine the matched sub-data sets as key behavior data and determine the unmatched sub-data sets as non-key behavior data.

[0107] In an embodiment, the method further includes:

[0108] The marking module is configured to mark the key behavior data and the non-key behavior data, and store the marked key behavior data and non-key behavior data into the first database.

[0109] The specific definition of the data analysis device based on the custom strategy can refer to the definition of the analysis method in the above, which will not be repeated here. Each module in the above data analysis device based on the custom strategy can be realized by software, hardware and their combination in whole or in part. The above each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0110] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 8 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store action detection data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the steps in any one of the above data analysis methods based on the custom strategy.

[0111] Those skilled in the art can understand that Figure 8 The structure shown in the above

[0112] In one embodiment, a computer device is provided, which includes a memory and a processor, and the memory stores a computer program. The processor executes the computer program to implement the steps in any one of the above data analysis methods based on the custom strategy.

[0113] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by the processor to implement the steps in any one of the above data analysis methods based on the custom strategy.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, the RAM can be in a variety of forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0115] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0116] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for data analysis based on a self-defined policy, characterized in that, The method includes: The custom strategy retrieved from the first database is parsed to generate the corresponding SQL statement; and The target data is parsed to obtain the data parsing results; Based on the SQL statement and the data parsing results, the first analysis result is determined; Based on the first analysis result and the clue strategy, the second analysis result is determined; The step of parsing the custom strategy obtained from the first database and generating the corresponding SQL statement includes: obtaining the custom strategy from the first database, where the first database is a MySQL database; mapping the custom strategy to nodes in the syntax tree and generating the corresponding SQL statement; The custom strategy includes basic conditions and associated conditions. Mapping the custom strategy to a syntax tree to generate corresponding SQL statements includes: determining the expressions for the basic conditions and the associated conditions; mapping the expressions for the basic conditions and the associated conditions to nodes in the syntax tree to generate corresponding SQL statements, wherein the SQL statement corresponding to the basic conditions is a subquery statement.

2. The method according to claim 1, characterized in that, The parsing of the target data to obtain the data parsing results includes: The target data is parsed to obtain the collected data and log data; The cleaned log data and the collected data are stored in a second database; the second database is a SparkSQL database.

3. The method according to claim 1, characterized in that, The determination of the first analysis result based on the SQL statement and the data parsing result includes: Based on the SQL statement, the data parsing results are grouped and aggregated to generate multiple sub-datasets; Based on the SQL statement and the multiple sub-datasets, a first analysis result is generated.

4. The method according to claim 3, characterized in that, The determination of the second analysis result based on the first analysis result and the clue strategy includes: Match the multiple subsets of data contained in the first analysis results with the clue strategy; The matching subsets are identified as key behavioral data; the non-matching subsets are identified as non-key behavioral data.

5. The method according to claim 4, characterized in that, The method further includes: The key behavioral data and the non-key behavioral data are marked; The tagged key behavior data and non-key behavior data are stored in the first database.

6. A data analysis device based on a custom strategy, characterized in that, The device includes: The first parsing module is used to parse the custom strategy obtained from the first database and generate the corresponding SQL statement; The step of parsing the custom strategy obtained from the first database and generating the corresponding SQL statement includes: obtaining the custom strategy from the first database, where the first database is a MySQL database; mapping the custom strategy to nodes in the syntax tree and generating the corresponding SQL statement; The custom strategy includes basic conditions and associated conditions. Mapping the custom strategy to a syntax tree to generate corresponding SQL statements includes: determining the expressions of the basic conditions and the associated conditions; mapping the expressions of the basic conditions and the associated conditions to nodes of the syntax tree respectively to generate corresponding SQL statements, wherein the SQL statement corresponding to the basic conditions is a subquery statement. The second parsing module is used to parse the target data and obtain the data parsing results; The first determining module is used to determine the first analysis result based on the SQL statement and the data parsing result; The second determining module is used to determine the second analysis result based on the first analysis result and the clue strategy.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images