Rule model engine-based big data platform abnormal access behavior identification method and system

Abstract

The application discloses a big data platform abnormal access behavior identification method and system based on a rule model engine, collects query logs, physical table metadata and view information on a data platform, extracts query statements and query account information, performs blood analysis on the query statements, constructs a blood atlas of query links, and determines sensitive field information involved in the query, obtains a data portrait of a query object, a user portrait of a query person and a behavior portrait of a query statement, configures an abnormal access behavior evaluation model based on field sensitivity features on a blood link, user portrait features and behavior portrait features, performs an evaluation task on a batch of self-defined query object entities in an audit range, and outputs an evaluation result, and according to a risk classification rule of the evaluation model, outputs the evaluation result as a query statement list of different risk levels. The application more accurately identifies whether user access to sensitive information is reasonable, and provides some judgment basis for compliance and data security departments.

Description

Technical Field

[0001] This invention relates to the field of data risk analysis technology, specifically to a method and system for identifying abnormal access behavior on a big data platform based on a rule model engine. Background Technology

[0002] In the application of big data platforms, data security and compliance auditing are crucial aspects. To identify abnormal user access behavior to sensitive data, existing technologies have proposed several security auditing methods based on user behavior analysis and knowledge graphs.

[0003] For example, an existing data security auditing method intercepts user-obtained data content through an API gateway and constructs a user profile based on characteristics such as the time, location, and IP address of the user's access to the content. This method further utilizes knowledge graph technology to extract security element entities from the data, such as account lists, operation logs, system permissions, data permissions, and data classification and grading. These entities are then labeled and associated, ultimately constructing a knowledge graph and inputting it into the data security auditing model to determine whether a user has passed the security audit.

[0004] However, the aforementioned existing technologies have the following technical drawbacks in practical applications:

[0005] 1. Existing technologies only audit based on the "current query log" and do not perform in-depth lineage analysis and upstream tracing of the tables and fields involved in the query statement (such as SQL statement). This makes it impossible to fully determine whether the fields involved in the current query are truly sensitive fields, nor can it assess the true compliance of the query behavior, which may lead to the omission of risks caused by the transmission of sensitive fields from upstream.

[0006] 2. Existing technologies fail to cover query scenarios where sensitive data has been processed through complex logic such as SQL function concatenation, aggregation calculations, field mapping, and transformation. In such scenarios, sensitive data may be "disguised," and traditional methods struggle to restore its true nature, leading to a failure in identification capabilities.

[0007] 3. Existing user profiles do not include crucial information such as whether individuals have correctly obtained data permissions, their department and team characteristics, etc. This information is essential for determining whether users are querying within their authorized scope and whether there are any instances of overstepping their authority.

[0008] 4. Existing audit models do not consider whether a user is a frequent visitor. Frequent query behavior can be an anomaly, such as data operations personnel frequently exporting large amounts of sensitive data. The lack of this dimension reduces the sensitivity to identifying potential risky behavior.

[0009] 5. Existing technologies rely solely on knowledge graph technology to analyze the relationships between different entities (such as accounts, permissions, and data). Due to the lack of tracing the lineage of query fields, it is impossible to accurately obtain the true dependencies between different entities in the data flow chain, which affects the accuracy of audit results.

[0010] Therefore, how to provide a method and system that can perform in-depth lineage analysis and comprehensive profile construction of query behavior on big data platforms, and accurately identify abnormal access behavior accordingly, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0011] The purpose of this invention is to provide a method and system for identifying abnormal access behavior in big data platforms based on a rule model engine. For query statements written in database query dialects, the method identifies the current sensitive field information used in the query statement or the sensitive field information traced back to the upstream to construct a data profile. Combined with the user profile features and behavioral profile features of the queryer, an abnormal query access identification model is constructed. Through timed scheduling, abnormal query behavior is detected periodically and corresponding reports are generated.

[0012] To achieve the above functions, this invention designs a method for identifying abnormal access behavior in a big data platform based on a rule model engine, including the following steps S1-S6, to complete the identification of abnormal access behavior by the queryer:

[0013] Step S1: Collect query logs, physical table metadata and view information from the data platform, extract query statements and query account information, and clean and standardize the query logs;

[0014] Step S2: Perform lineage analysis on each query statement, construct a lineage graph of the query chain, and determine the sensitive field information involved in the query, wherein the sensitive field information includes the sensitive identifier of the current query field and the sensitive identifier of the upstream field determined by tracing upwards;

[0015] Step S3: Obtain the data profile of the queried object, the user profile of the querier, and the behavioral profile of the query statement;

[0016] Step S4: Based on the rule engine, use the configuration file to configure the abnormal access behavior identification rules and risk classification rules for the field sensitivity features, user profile features and behavioral profile features on the lineage link, and build an abnormal access behavior evaluation model.

[0017] Step S5: According to the risk classification rules of the assessment model, the assessment tasks are performed in batches on the custom query object entities within the audit scope through timed scheduling, and the assessment results are output.

[0018] Step S6: Based on the risk classification rules of the assessment model, the assessment results are generated into a list of query statements for different risk levels.

[0019] As a preferred technical solution of the present invention: the lineage analysis of the query statement in step S2 includes: parsing the SQL statement, tracing the fields involved in the query upstream along the lineage, and determining whether the fields are marked as sensitive fields and whether they have been manually confirmed.

[0020] As a preferred technical solution of the present invention: the data profile of the query object in step S3 includes the classification label, security level, rating status and department of the query object table or field;

[0021] The user profile of the inquirer includes the inquirer's job position, department or team, scope of authorization, and scope of business responsibilities.

[0022] The behavioral profile of the query statement includes the query filtering conditions, field processing logic, field transformation mapping rules, result export time, query time period, and query frequency of each segment.

[0023] As a preferred technical solution of the present invention: the abnormal access behavior evaluation model in step S4 is configured through DSL language, which supports monitoring and operation of rule execution status and viewing of rule execution results.

[0024] As a preferred technical solution of the present invention: the evaluation task in step S5 is configured to be executed in batches, and report instances are generated for the execution results, supporting the download of report results.

[0025] As a preferred technical solution of the present invention: the behavioral profile of the query statement in step S4 includes dimension information on whether the user is a frequent visitor, which is used to identify frequent query behavior.

[0026] This invention also designs an abnormal access behavior identification system for a big data platform based on a rule model engine, which implements the aforementioned abnormal access behavior identification method for a big data platform based on a rule model engine based on the following modules:

[0027] Lineage Analysis Function Module: It uses code parsing technology to parse SQL statements, traces the sensitive field information involved in the query upwards along the lineage and determines its sensitivity level; and obtains the data profile of the query object involved in the query statement, the user profile of the queryer, and the behavioral profile of the query statement through the data retrieval interface.

[0028] Data service management module: Configures data retrieval interfaces for data profiles of query objects, user profiles of queryers, and behavioral profiles of query statements, including the management of query log retrieval interfaces;

[0029] Rule Management Module: Utilizes the rule engine to configure rules for identifying abnormal access behavior, builds an evaluation model for abnormal access behavior, and supports monitoring and operating the rule execution status and viewing the rule execution results;

[0030] Report Management Module: Manages report instances generated by management rules and supports downloading report results.

[0031] The present invention also designs an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for identifying abnormal access behavior of a big data platform based on a rule model engine.

[0032] The present invention also designs a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the above-described method for identifying abnormal access behavior of a big data platform based on a rule model engine.

[0033] Beneficial effects: Compared with existing technologies, this invention changes the previous model that relied on manual analysis and expert decision-making, including the following advantages:

[0034] 1. It incorporates the experience of security experts and industry policy requirements into the risk assessment model, and provides flexible and configurable modification capabilities for the assessment model, making it easy to empower the risk identification capabilities to security risk control activities in all departments of the company.

[0035] 2. In accordance with policy and regulatory requirements and company control requirements, we utilize metadata automation management capabilities and combine them with processing characteristics to customize the dissemination of sensitive labels and determine security levels, overcoming the shortcomings of slow timeliness and high cost of manual labeling.

[0036] 3. By leveraging lineage analysis capabilities to construct a query lineage graph and tracing query logs over a period of time, abnormal data access behaviors can be identified. Furthermore, as long as complete view, table, and statement metadata is collected and the corresponding sensitive information is correctly labeled, abnormal access queries can be identified, ultimately achieving "no omissions or errors" in risk exposure.

[0037] 4. By leveraging the inference capabilities of operator lineage codes and merging multiple layers of code, it can identify whether the queried data has undergone data transformation, overcoming the drawback of easily losing key information when relying solely on lineage inference. Furthermore, after merging multiple layers of code and performing caliber trimming, and then analyzing whether the conditions and processing logic used in the simplified caliber can be used to reconstruct the final sensitive fields, it solves the problem of previous unrecognizable operations caused by "field transformation" or "repackaging." Attached Figure Description

[0038] Figure 1This is a schematic diagram of a method for identifying abnormal access behavior of a big data platform based on a rule model engine, according to an embodiment of the present invention.

[0039] Figure 2 This is a flowchart of a method for identifying abnormal access behavior in a big data platform based on a rule model engine, according to an embodiment of the present invention.

[0040] Figure 3 This is an example diagram of a rule engine configuration provided according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of an abnormal access behavior identification system for a big data platform based on a rule model engine, provided according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0043] This invention is applied to a sensitive data leakage risk analysis scenario in a big data platform for the securities industry. It audits the logs returned after the data query process to identify whether users exhibit abnormal query behavior. This behavior has the following three characteristics:

[0044] 1. Unauthorized Access: This refers to a query behavior that exceeds the scope of authorized duties. Authorization here, from a job responsibility perspective, is defined as the range of data that can be accessed and the query behavior itself. The data range refers to the clearly defined dataset range that can be queried, permissions that are not accessible to the database account. Query behaviors include: whether the query is de-identified, whether it triggers a firewall query, whether it focuses on specific user queries, and whether it is a high-frequency query. After defining the rules for anomaly identification, the comparison between the user profile, data profile, and behavior profile and the "correctly authorized scope" is used to determine whether the behavior is unauthorized. If a user's authorized query behavior includes querying specific personal information using a de-identified method (i.e., querying without specifically specifying the individual's PII identifier), even if the returned results contain sensitive data, it is not considered an abnormal data access behavior. For example, an administrator, to check data availability, uses an account with operation and maintenance permissions to add `limit 1` to the query statement.

[0045] 2. The user initiated a query. This is specifically reflected in the SQL audit log, where the user must have at least one DQL query statement. The characteristics of the query statement should be analyzed, such as whether the returned results contain sensitive data and whether the query used a de-identified query method for further refinement. If the user only creates or updates a table or view containing sensitive data without initiating a query on that table or view, it is not considered abnormal query behavior. If the user uses a de-identified method to query specific personal information, but the returned results contain sensitive data, it is necessary to consider whether the user's job responsibilities authorize such query behavior. For example, if data operations personnel frequently query sensitive data, this behavior needs to be identified as suspected abnormal access.

[0046] 3. The query object is sensitive data. Sensitive data refers to field data that has been manually identified and confirmed as "sensitive level" based on the PII sensitive information classification and grading standard. In this invention, data that has not been manually confirmed is identified by the system scanning tool and then marked as suspected sensitive data.

[0047] The method for identifying abnormal access behavior of a big data platform based on a rule model engine provided in this embodiment of the invention refers to... Figure 1 , Figure 2 The process includes the following steps S1-S6 to identify abnormal access behavior by the inquirer:

[0048] Step S1: Collect query logs, physical table metadata and view information from the data platform, extract query statements and query account information, and clean and standardize the query logs;

[0049] Step S2: Perform lineage analysis on each query statement, construct a lineage graph of the query chain, and determine the sensitive field information involved in the query, wherein the sensitive field information includes the sensitive identifier of the current query field and the sensitive identifier of the upstream field determined by tracing upwards;

[0050] Each query statement is associated with a custom query object entity, which carries the relevant attribute information of the query statement, such as account, time, frequency, etc.

[0051] The lineage analysis of the query statement includes: parsing the SQL statement, tracing the fields involved in the query upstream along the lineage, and determining whether the fields are marked as sensitive fields and whether they have been manually confirmed.

[0052] When the fields in the current query have not been manually confirmed, it is necessary to perform lineage analysis and reasoning on the query statement to construct a lineage graph of the query chain.

[0053] Step S3: Obtain the data profile of the queried object, the user profile of the querier, and the behavioral profile of the query statement;

[0054] The data profile of the query object includes the category label, security level, classification status, and department of the query object table or field;

[0055] The user profile of the inquirer includes the inquirer's job position, department or team, scope of authorization, and scope of business responsibilities (stored in a structured manner according to tags).

[0056] The behavioral profile of the query statement includes the query filtering conditions, field processing logic, field transformation mapping rules, result export time, query time period, and query frequency of each segment.

[0057] Step S4: Refer to Figure 3 Based on the rule engine, the abnormal access behavior identification rules and risk classification rules are configured by using configuration files to configure the field sensitivity features, user profile features and behavioral profile features on the lineage link, and an abnormal access behavior evaluation model is constructed.

[0058] The abnormal access behavior evaluation model is configured using the DSL language, supports monitoring and manipulating the rule execution status, and allows viewing the rule execution results.

[0059] The behavioral profile of the query statement includes information on whether the user is a frequent visitor, which is used to identify high-frequency query behavior.

[0060] Step S5: According to the risk classification rules of the assessment model, the assessment tasks are performed in batches on the custom query object entities within the audit scope through timed scheduling, and the assessment results are output; in the embodiment, the assessment results are in the form of a list of query statements identified as "high risk", "suspected risk", etc.

[0061] The evaluation task is configured to be executed in batches, and report instances are generated for the execution results, supporting report result download.

[0062] Step S6: Based on the risk classification rules of the assessment model, the assessment results are generated into a list of query statements for different risk levels.

[0063] A big data platform abnormal access behavior identification system based on a rule model engine, referring to... Figure 4 The following modules are used to implement the method for identifying abnormal access behavior in a big data platform based on a rule model engine:

[0064] Lineage analysis module: It uses code parsing technology to parse SQL statements and traces the sensitive field information involved in the query upwards along the lineage to determine its sensitivity level; in the embodiment, it traces upwards up to 10 levels.

[0065] And through the data retrieval interface, we can obtain the data profile of the query object involved in the query statement, the user profile of the queryer, and the behavioral profile of the query statement;

[0066] Data service management module: Configures data retrieval interfaces for data profiles of query objects, user profiles of queryers, and behavioral profiles of query statements, including the management of query log retrieval interfaces;

[0067] The methods for configuring each data retrieval interface are as follows:

[0068] Create new profile API information, enter the profile API name, parameters, etc.;

[0069] Enter the URL address to debug the profiling API;

[0070] After successful debugging, submit the generated image API interface;

[0071] Rule Management Module: Utilizes the rule engine to configure rules for identifying abnormal access behavior, builds an evaluation model for abnormal access behavior, and supports monitoring and operating the rule execution status and viewing the rule execution results;

[0072] The method for configuring abnormal access behavior identification rules and building an abnormal access behavior evaluation model is as follows:

[0073] Add a new model on the designated page and edit the corresponding rule information;

[0074] Based on the scenario, generate various data access behavior recognition models, such as monitoring high-frequency access, bypassing firewalls, focusing on key accounts, and focusing on key users;

[0075] Report Management Module: Manages report instances generated by management rules and supports downloading report results.

[0076] The method for automating the execution of the model and generating a visual recognition result report is as follows:

[0077] Automated execution of recognition models, generating various recognition model report information;

[0078] The identification model report displays a report overview, analysis results, and execution log information;

[0079] The report overview displays the total number of daily queries, the number of suspected abnormal queries, and the month-on-month comparison. It also highlights key tables, key targets, and the source information of suspected abnormal queries to facilitate analysis by system administrators.

[0080] For suspected abnormal access, display whether the current query involves sensitive information or the link tracing involves sensitive information.

[0081] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for identifying abnormal access behavior of a big data platform based on a rule model engine.

[0082] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned method for identifying abnormal access behavior of a big data platform based on a rule model engine.

[0083] In summary, the embodiments of the present invention construct a systematic, automated, and structured processing method that can automatically generate key query information, including query user, query time, and query object, without manual intervention;

[0084] Constructing a data lineage map supports the interpretability analysis of the abnormal access behavior assessment model, providing data tracing basis for the identification results and inference of the impact path.

[0085] The data profiles of the query objects, the user profiles of the queryers, and the behavioral profiles of the query statements are associated with the access objects. Combined with the node relationships in the lineage graph, cross-system and cross-level feature enhancements are achieved, and data access behavior recognition models in different scenarios can be created through permutations and combinations.

[0086] The abnormal access behavior evaluation model is applied to scenarios such as anomaly detection, risk identification, user segmentation, or intelligent decision-making to achieve accurate identification and prediction of target objects.

[0087] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for identifying abnormal access behavior in a big data platform based on a rule model engine, characterized in that, The process includes the following steps S1-S6 to identify abnormal access behavior by the inquirer: Step S1: Collect query logs, physical table metadata and view information from the data platform, extract query statements and query account information, and clean and standardize the query logs; Step S2: Perform lineage analysis on each query statement, construct a lineage graph of the query chain, and determine the sensitive field information involved in the query, wherein the sensitive field information includes the sensitive identifier of the current query field and the sensitive identifier of the upstream field determined by tracing upwards; Step S3: Obtain the data profile of the queried object, the user profile of the querier, and the behavioral profile of the query statement; Step S4: Based on the rule engine, use the configuration file to configure the abnormal access behavior identification rules and risk classification rules for the field sensitivity features, user profile features and behavioral profile features on the lineage link, and build an abnormal access behavior evaluation model. Step S5: According to the risk classification rules of the assessment model, the assessment tasks are performed in batches on the custom query object entities within the audit scope through timed scheduling, and the assessment results are output. Step S6: Based on the risk classification rules of the assessment model, the assessment results are generated into a list of query statements for different risk levels.

2. The method for identifying abnormal access behavior of a big data platform based on a rule model engine according to claim 1, characterized in that, The lineage analysis of the query statement in step S2 includes: parsing the SQL statement, tracing the fields involved in the query upstream along the lineage, and determining whether the fields are marked as sensitive fields and whether they have been manually confirmed.

3. The method for identifying abnormal access behavior of a big data platform based on a rule model engine according to claim 1, characterized in that, The data profile of the query object in step S3 includes the category label, security level, classification status, and department of the query object table or field; The user profile of the inquirer includes the inquirer's job position, department or team, scope of authorization, and scope of business responsibilities. The behavioral profile of the query statement includes the query filtering conditions, field processing logic, field transformation mapping rules, result export time, query time period, and query frequency of each segment.

4. The method for identifying abnormal access behavior of a big data platform based on a rule model engine according to claim 1, characterized in that, The abnormal access behavior evaluation model described in step S4 is configured using the DSL language, which supports monitoring and operating the rule execution status and viewing the rule execution results.

5. The method for identifying abnormal access behavior of a big data platform based on a rule model engine according to claim 1, characterized in that, The evaluation task described in step S5 is configured to be executed in batches, and report instances are generated for the execution results, supporting report result download.

6. The method for identifying abnormal access behavior of a big data platform based on a rule model engine according to claim 1, characterized in that, The behavioral profile of the query statement in step S4 includes dimension information on whether the user is a frequent visitor, which is used to identify high-frequency query behavior.

7. A big data platform abnormal access behavior identification system based on a rule model engine, characterized in that, The following modules are used to implement the method for identifying abnormal access behavior in a big data platform based on a rule model engine, as described in any one of claims 1-6: Lineage analysis module: It uses code parsing technology to parse SQL statements and traces the sensitive field information involved in the query upwards along the lineage to determine its sensitivity level; And through the data retrieval interface, we can obtain the data profile of the query object involved in the query statement, the user profile of the queryer, and the behavioral profile of the query statement; Data service management module: Configures data retrieval interfaces for data profiles of query objects, user profiles of queryers, and behavioral profiles of query statements, including the management of query log retrieval interfaces; Rule Management Module: Utilizes the rule engine to configure rules for identifying abnormal access behavior, builds an evaluation model for abnormal access behavior, and supports monitoring and operating the rule execution status and viewing the rule execution results; Report Management Module: Manages report instances generated by management rules and supports downloading report results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for identifying abnormal access behavior of a big data platform based on a rule model engine as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for identifying abnormal access behavior of a big data platform based on a rule model engine as described in any one of claims 1-6.

Images