A data management method, device, apparatus and storage medium

Abstract

The application provides a data management method, device, equipment and storage medium. The method comprises: collecting a plurality of data and determining a data label corresponding to each data; performing syntax analysis on an SQL statement related to a data scheduling task to construct an abstract syntax tree, determining a data routing relationship between each data based on the abstract syntax tree, and establishing a data blood relationship graph; in response to a label change event, determining a plurality of second data having a data routing relationship with the first data based on the data blood relationship graph; updating the data label of each second data according to the data label of the first data after the change, and adjusting the access control policy corresponding to each second data. The application realizes dynamic updating of the access control policy, without manual modification one by one, improves the data management efficiency, and avoids data asset leakage.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data management method, apparatus, device, and storage medium. Background Technology

[0002] As enterprises deepen their digital transformation, the amount of data generated and accumulated by various business systems is growing explosively. As a core production factor, data management is increasingly valued by enterprises.

[0003] In existing data access control systems, data access control policies are typically predefined statically. When data labels need to be adjusted due to changes in business requirements, data administrators need to manually identify the affected data and modify the access control policies for each data item.

[0004] The aforementioned method of managing data manually is inefficient, and manual intervention can easily overlook data assets that need to be adjusted. This may create security vulnerabilities during the policy change window, leading to unauthorized access to sensitive data and resulting in data leakage. Summary of the Invention

[0005] The main objective of this invention is to provide a data management method, apparatus, device, and storage medium. It aims to address the technical problems of low efficiency and data leakage in existing data management methods.

[0006] To achieve the above objectives, the present invention provides a data management method, the method comprising: Collect multiple data points; Perform data analysis on each data point to determine the corresponding data label for each data point; Based on the data routing relationships between each data point, a data lineage diagram is established. Syntax analysis is performed on the SQL statements related to the data scheduling task to construct an abstract syntax tree. The correspondence between the source fields and the target fields extracted from the abstract syntax tree is used to determine the data routing relationship between each data. Based on the data lineage graph, the first data is used as the starting point to traverse along the directed edges to determine multiple second data that have a data routing relationship with the first data. Based on the data tags after the first data is changed, the highest confidentiality level among the sensitivity level tags of the first data and the current sensitivity level tags of each second data is determined as the target sensitivity level tag, and the sensitivity level tag corresponding to each second data is updated to the target sensitivity level tag; Based on the updated data tag of each second data, adjust the access control policy corresponding to each second data.

[0007] Optionally, the traversal along the directed edges starting from the first data includes: Starting from the first data, a breadth-first traversal or a depth-first traversal is performed along the directed edges in the data lineage graph, and the maximum depth of the traversal is configurable.

[0008] Optionally, establishing a data lineage graph based on the data routing relationships between each data item includes: Perform syntax analysis on SQL statements related to data scheduling tasks to determine the data routing relationship between each piece of data, and / or parse event logs to determine the data routing relationship between each piece of data; The data routing relationships between each data point are mapped to a preset graph database to obtain the data lineage graph; The data lineage diagram includes a data version identifier.

[0009] Optionally, the method further includes: Receive a policy update request, the policy update request being used to update the access control policy, and the policy update request carrying the updated policy version; In response to the policy update request, calculate the version difference between the updated policy version and the current policy version; Update the access control policy based on the version difference set.

[0010] Optionally, the method further includes: Upon receiving a data access request from the first user, the system obtains the data tag corresponding to the third data, the identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user. The data access request is used to represent access to the third data. Based on the data tags corresponding to the third data, multiple first access control policies are determined; The identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user are matched with each first access control policy to obtain a second access control policy. Based on the second access control policy, an access operation is performed on the third data.

[0011] Optionally, the method further includes at least one of the following: The system receives multiple operations input by a second user on the policy construction interface and generates an access control policy based on these operations. These operations include subject type selection, object type selection, access type selection, and environmental condition selection. Receive policy description language input from a second user, and generate access control policies based on the policy description language; Generate access control policies based on historical access logs.

[0012] To address the aforementioned technical problems, this application also provides a data management device, the device comprising: The data acquisition module is used to collect multiple data points. The first determining module is used to perform data analysis on each data and determine the data label corresponding to each data. A module is established to create a data lineage graph based on the data routing relationships between each data item. The second determining module is used to perform syntax analysis on the SQL statements related to the data scheduling task to construct an abstract syntax tree, determine the data routing relationship between each data based on the correspondence between the source fields and the target fields extracted from the abstract syntax tree, and determine multiple second data that have a data routing relationship with the first data based on the data lineage graph, starting from the first data and traversing along the directed edges. The first update module is used to determine the highest confidentiality level among the sensitivity level tags of the first data and the current sensitivity level tags of each second data as the target sensitivity level tag based on the data tags after the first data is changed, and to update the sensitivity level tag corresponding to each second data to the target sensitivity level tag. The adjustment module is used to adjust the access control policy corresponding to each second data according to the updated data tag of each second data.

[0013] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution: The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the data management method described above.

[0014] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data management method described above.

[0015] To address the aforementioned technical problems, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data management method described above.

[0016] This application provides a data management method, apparatus, device, and storage medium. The method includes: collecting multiple data sets; performing data analysis on each data set to determine the corresponding data tag; establishing a data lineage graph based on the data routing relationships between each data set; performing syntax analysis on SQL statements related to data scheduling tasks to construct an abstract syntax tree; determining the data routing relationship between each data set based on the correspondence between source fields and target fields extracted from the abstract syntax tree; and based on the data lineage graph, traversing along directed edges starting from the first data set to determine multiple second data sets that have a data routing relationship with the first data set; determining the target sensitivity level tag based on the changed data tag of the first data set and the highest confidentiality level among the changed sensitivity level tags of the first data set and the current sensitivity level tags of each second data set; and updating the sensitivity level tag corresponding to each second data set to the target sensitivity level tag; and adjusting the access control policy corresponding to each second data set based on the updated data tag of each second data set. This application achieves dynamic updates to the access control policy, eliminating the need for manual modification of the access control policy for each data set, improving the efficiency of data management, and avoiding data asset leakage due to manual management. Attached Figure Description

[0017] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the data management method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the metadata collection and classification process provided in this application embodiment; Figure 3 This is a flowchart illustrating the dynamic strategy propagation mechanism provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the incremental strategy synchronization mechanism provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the multi-dimensional context-aware decision mechanism provided in the embodiments of this application; Figure 6 This is a schematic diagram of the application architecture of the data management method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the data management device provided in the embodiments of this application; Figure 8 This is a basic structural block diagram of a computer device provided in the embodiments of this application. Detailed Implementation

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the data management method provided in this application embodiment. The data management method provided in this application embodiment is applied to a data management device, and the method includes the following steps: Step 101: Collect multiple data points.

[0022] In this step, multiple metadata can be collected through a preset metadata collection module. It should be understood that the above metadata can be considered as the data in the embodiments of this application.

[0023] One optional implementation is that the metadata collection module runs as a daemon on the data nodes of the data platform. Another optional implementation is that the metadata collection module remotely connects to the data source via a standard interface in an agentless manner.

[0024] The metadata acquisition module includes an extensible acquisition adapter framework, with one adapter plugin corresponding to each data source type. These adapter plugins include, but are not limited to, adapters for relational databases such as MySQL / PostgreSQL / Oracle, distributed file system adapters, message queue adapters such as Kafka / Pulsar, and search engine adapters such as Elasticsearch.

[0025] Optionally, the process for collecting multiple data points is as follows: (a) The metadata collection module triggers full or incremental collection tasks according to a configurable scheduling cycle; (b) Each adapter plugin calls the system table or management API of the corresponding data source to obtain metadata such as data table list, field definition, partition information, and storage statistics; (c) The collected raw metadata is standardized and written to the metadata repository in JSON format; (d) For scenarios with high real-time requirements, the metadata collection module also supports near real-time incremental collection based on data source event notifications.

[0026] Step 102: Perform data analysis on each data point to determine the corresponding data label for each data point.

[0027] In this step, after collecting multiple data points, content analysis and semantic understanding can be performed on the collected metadata based on predefined sensitive data identification rules and natural language processing models. Each piece of collected metadata is assigned a classification label and a sensitivity level label, i.e., a data label.

[0028] The aforementioned category labels include, but are not limited to: personal identification information, financial data, business analysis data, and technology research and development data. Sensitivity level labels include, but are not limited to, four levels: public, internal, confidential, and top secret.

[0029] Please see Figure 2 , Figure 2 This is a flowchart illustrating the metadata collection and classification process provided in this application embodiment. The specific scheme for metadata collection has been described in the foregoing embodiments and will not be repeated here.

[0030] exist Figure 2 In the process example shown, semantic analysis is first performed on the data content of each data point to determine the corresponding category label for each data point.

[0031] Specifically, the data can first be matched against rules, such as field names, field comments, and table names, against a predefined sensitive data identification rule base. This rule base contains hundreds of predefined rules.

[0032] Optionally, data with field names containing keywords such as "id_card", "ID card", and "social_security" can be classified as personal identification information. Data with field names containing keywords such as "salary", "bonus", and "revenue" can be classified as financial data. Custom rules can be flexibly extended using regular expressions.

[0033] For fields where rule matching cannot determine the category, content sampling analysis can be performed. Optionally, the first 1000 rows of fields can be sampled by default. Analysis is performed using the built-in pattern matcher, for example, to identify data patterns that conform to the format of ID card numbers.

[0034] After determining the category label for each data point, a sensitivity level label is determined based on the data category represented by the category label. The mapping relationship between data categories and sensitivity level labels can be customized.

[0035] For example, if the category label corresponding to the data represents personal identity information, then the confidentiality level represented by the sensitivity level label corresponding to the data can be determined to be confidential.

[0036] For example, if the category label corresponding to the data represents financial data, then the confidentiality level represented by the sensitivity label corresponding to the data can be determined to be confidential.

[0037] For example, if the category label corresponding to the data represents business analysis data, then the confidentiality level represented by the sensitivity level label corresponding to the data can be determined to be internal.

[0038] For example, if the data corresponds to the research and development data of the classification label characterization technology, then the confidentiality level of the sensitivity label characterization corresponding to the data can be determined to be internal.

[0039] For example, if the classification label corresponding to the data represents public information, then the confidentiality level represented by the sensitivity level label corresponding to the data can be determined to be public.

[0040] Optionally, data tags are bound to metadata assets, and information such as timestamps for classification and grading, rules used, and confidence scores are recorded and written to a classification tag repository.

[0041] Step 103: Based on the data routing relationship between each data, establish a data lineage diagram.

[0042] In this step, the execution logs, SQL statements, and data flow records of the data processing tasks are parsed in a non-intrusive manner, and the dependencies between data, i.e., routing relationships, are automatically extracted to construct a data lineage graph at the field level.

[0043] Optionally, the above data lineage graph is stored in the form of a directed acyclic graph, where each node represents a data field and each directed edge represents the flow of data from the source field to the target field.

[0044] Step 104: Perform syntax analysis on the SQL statements related to the data scheduling task to construct an abstract syntax tree. Determine the data routing relationship between each piece of data based on the correspondence between the source fields and the target fields extracted from the abstract syntax tree. Based on the data lineage graph, traverse along the directed edges starting from the first piece of data to determine multiple pieces of second data that have a data routing relationship with the first piece of data.

[0045] Optionally, the traversal along the directed edges starting from the first data includes: Starting from the first data, a breadth-first traversal or a depth-first traversal is performed along the directed edges in the data lineage graph, and the maximum depth of the traversal is configurable.

[0046] In this step, the SQL statements related to the data scheduling task are parsed to construct an abstract syntax tree. From this abstract syntax tree, the correspondence between the source table and source fields in the SELECT clause and the target table and target fields in the INSERT / CREATE clause is extracted to determine the data routing relationship between each piece of data. Then, the position of the first piece of data in the data lineage graph is located. Starting from the first piece of data, a breadth-first or depth-first traversal is performed along the directed edges of the lineage graph to determine multiple pieces of second data that have a data routing relationship with the first piece of data. Optionally, to avoid an excessively large traversal range, the maximum traversal depth can be configured, for example, limiting the traversal level to 10 levels.

[0047] Step 105: Based on the data tags after the first data is changed, determine the highest confidentiality level among the sensitivity level tags after the first data is changed and the current sensitivity level tags of each second data as the target sensitivity level tag, and update the sensitivity level tag corresponding to each second data to the target sensitivity level tag.

[0048] In this step, after identifying multiple pieces of second data that have a data routing relationship with the first data, the data label corresponding to each piece of second data is updated. Optionally, according to the least upper bound propagation principle, the sensitivity level label of the first data after its modification and the sensitivity level label representing the highest confidentiality level among the current sensitivity level labels of each piece of second data are determined as the target sensitivity level label, and the sensitivity level label corresponding to each piece of second data is updated to the target sensitivity level label.

[0049] Specifically, please refer to Figure 3 , Figure 3This is a flowchart illustrating the dynamic policy propagation mechanism provided in an embodiment of this application. Figure 3 In the illustrated process example, after receiving a label change event, starting with the data field of the changed asset (i.e., the first data), all direct and indirect downstream nodes (i.e., the second data) are traversed along directed edges in the data lineage graph. For each downstream node traversed, the currently stored sensitivity level label is read as the second sensitivity level label, and the sensitivity level label after the first data change carried in the label change event is also obtained as the first sensitivity level label. This yields the first sensitivity level label and multiple second sensitivity level labels. The target sensitivity level label is determined according to the "least upper bound propagation principle," and the calculated target sensitivity level label is used as the updated sensitivity level label for each second data. The aforementioned "least upper bound propagation principle" can be understood as: target sensitivity level = max(first sensitivity level, second sensitivity level 1, second sensitivity level 2, ..., second sensitivity level). n ), where the max operation means taking the highest level of confidentiality among all the comparison objects.

[0050] Optionally, the sensitivity levels are ordered from low to high as follows: Public < Internal < Confidential < Top Secret.

[0051] For example, if the sensitivity level of the first data is upgraded from "Internal" to "Confidential," and the original sensitivity level of the second data is "Internal," then the target sensitivity level is max(Confidential, Internal) = Confidential; if the second data also has other upstream nodes with a sensitivity level of "Top Secret," then the target sensitivity level is max(Confidential, Internal, Top Secret) = Top Secret. In this embodiment, by introducing the above-mentioned dynamic policy propagation mechanism, the problems of delayed policy updates and easy omission of the second data under the traditional manual method are effectively solved. Through automated lineage traversal and level propagation calculation, it is ensured that the data security policy and the actual data security level are always consistent, eliminating the security control vacuum period caused by untimely manual operation.

[0052] Step 106: Adjust the access control policy corresponding to each second data according to the updated data tag of each second data.

[0053] In this step, after updating the data label corresponding to each second piece of data, the access control policy is adjusted for any given second piece of data based on the updated data label. Optionally, the access control policy that matches the updated data label can be determined as the adjusted access control policy, thereby updating the access control policy corresponding to the second piece of data.

[0054] In this embodiment, when the data label corresponding to the first data changes, multiple second data are determined based on a pre-established data lineage diagram representing the data routing relationship between each data. The data label corresponding to each second data is updated according to the changed data label of the first data, thereby adjusting the access control policy corresponding to each second data. This achieves dynamic updating of the access control policy without the need for manual modification of the access control policy for each data, improving the efficiency of data management and avoiding data asset leakage due to manual management.

[0055] Optionally, establishing a data lineage graph based on the data routing relationships between each data item includes: Perform syntax analysis on SQL statements related to data scheduling tasks to determine the data routing relationship between each piece of data, and / or parse event logs to determine the data routing relationship between each piece of data; The data routing relationships between each data point are mapped to a preset graph database to obtain the data lineage graph; The data lineage diagram includes a data version identifier.

[0056] In this embodiment, a non-intrusive method can be used to construct the data lineage graph, as follows: (a) SQL parsing: Obtain the SQL statement of the data scheduling task, use the SQL parser to perform syntax analysis on the SQL statement, and construct an abstract syntax tree (AST). Extract the correspondence between the source table and source fields in the SELECT clause and the target table and target fields in the INSERT / CREATE clause from the AST, and then determine the data routing relationship between each piece of data.

[0057] (b) Data flow tracing: For non-SQL data flow, the data flow path is restored by parsing the event log.

[0058] An alternative implementation involves determining the data routing relationship between each piece of data using the SQL parsing method described above.

[0059] Another alternative implementation is to determine the data routing relationship between each piece of data through the data flow tracing method described above.

[0060] Another alternative implementation is to determine the data routing relationship between each piece of data through the above-described SQL parsing and data flow tracing methods.

[0061] (c) Lineage Graph Construction: The extracted data routing relationships are stored in the graph database in the form of triples (fully qualified name of the source field, fully qualified name of the target field, and processing logic summary). The graph database uses a distributed graph storage engine that supports massive nodes and efficient graph traversal. The lineage graph supports two query directions: downstream tracing (finding all derived data from the source data) and upstream tracing (finding all original data sources from the report).

[0062] (d) Lineage version management: Since the data processing logic may change over time, a data version identifier is assigned to each lineage analysis result, that is, a data version identifier is assigned to the data lineage relationship diagram, which supports viewing historical lineage relationship versions and comparing the differences between different versions.

[0063] Optionally, the method further includes: Receive a policy update request, the policy update request being used to update the access control policy, and the policy update request carrying the updated policy version; In response to the policy update request, calculate the version difference between the updated policy version and the current policy version; Update the access control policy based on the version difference set.

[0064] Please see Figure 4 , Figure 4 This is a flowchart illustrating the incremental strategy synchronization mechanism provided in an embodiment of this application. Figure 4 In the illustrated process example, the incremental policy synchronization mechanism is executed by the policy incremental synchronization module.

[0065] The policy management center maintains a global policy metadata table. Each record in this table corresponds to a policy rule and includes a unique policy identifier (UUID), a policy content hash value, the current version number, a creation timestamp, and a last modification timestamp. Each time a policy is created, modified, or deleted, the version number of the corresponding record is incremented by 1.

[0066] Each policy execution node maintains its own set of loaded policy versions locally. When initiating an incremental synchronization request to the policy management center, the node compresses and encodes all local policy identifiers and their version numbers using a Bloom filter, transmitting the encoded binary data in the request body. It should be understood that the advantage of using a Bloom filter is that it can efficiently represent the current policy version state held by the node with minimal space overhead.

[0067] After receiving the incremental synchronization request, the policy management center decodes the Bloom Filter and performs a membership query to quickly identify missing or outdated policy rules and form a version difference set.

[0068] The policy management center serializes the policy rules in the version difference set into binary data packets, compresses them using gzip, and sends them to the requesting nodes via an HTTP / 2 persistent connection. To ensure transmission reliability, each data packet includes a CRC32 checksum.

[0069] After receiving incremental data packets, the policy execution node first verifies the CRC32, then decompresses and merges each policy rule into the local memory cache and persistent storage. The merging logic supports three operation types: INSERT, UPDATE, and logical deletion (marked as inactive).

[0070] Once completed, the policy execution node updates its local version number record and sends an ACK confirmation command to the policy management center. If the management center does not receive the ACK confirmation command within the timeout period (30 seconds by default), it will retransmit the incremental data packet the next time the node initiates a synchronization request.

[0071] In this embodiment, the amount of data synchronized in a single instance is reduced from the entire policy file to only the incremental policy that has been changed, thereby shortening the policy synchronization latency from minutes to seconds. At the same time, it significantly reduces the consumption of network bandwidth and system CPU, effectively solving the technical problems of high policy synchronization latency and high resource consumption in a large-scale distributed environment.

[0072] Optionally, the method further includes: Upon receiving a data access request from the first user, the system obtains the data tag corresponding to the third data, the identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user. The data access request is used to represent access to the third data. Based on the data tags corresponding to the third data, multiple first access control policies are determined; The identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user are matched with each first access control policy to obtain a second access control policy. Based on the second access control policy, an access operation is performed on the third data.

[0073] Please see Figure 5 , Figure 5 This is a flowchart illustrating the multi-dimensional context-aware decision-making mechanism provided in an embodiment of this application. Figure 5 In the illustrated process example, the multi-dimensional context-aware decision mechanism is executed by the policy execution engine.

[0074] The policy execution engine is deployed at the data access gateway layer as an embedded SDK or a standalone agent process, performing real-time analysis on each data access request. The specific execution flow is as follows: (a) When the first user initiates a data access request through an application or data query tool, the SQL parser built into the strategy execution engine performs lexical and syntactic analysis on the SQL statement related to the third data, and extracts the target data table name, the target field list and the requested operation type.

[0075] (b) The policy execution engine obtains the context information of the data access request. This context information includes: extracting user identity information from the authentication token carried in the data access request, i.e., the identity information of the first user; the operation type, data volume, and operation frequency of the data access request, i.e., the operation information corresponding to the data access request; querying the classification label and sensitivity level of the target data from the metadata label library, i.e., the data label corresponding to the third data; obtaining the current timestamp from the system clock and calculating the time window to which it belongs, i.e., the time window to which the data access request belongs; obtaining the target terminal IP address and network area corresponding to the first user from the network layer information of the data access request; and determining the client device type from the device fingerprint reported by the data access request, i.e., the network environment in which the target terminal corresponding to the first user is located.

[0076] (c) The policy execution engine uses the data tag corresponding to the third data as an index to retrieve multiple first access control policies that match the data tag in the local memory cache.

[0077] (d) The policy execution engine matches and evaluates the collected context information against the multiple retrieved first access control policies one by one to determine the second access control policy. Optionally, an evaluation strategy based on expression compilation optimization is adopted, that is, the conditional expressions in the access control policy are pre-compiled into opcode sequences, and the opcode sequences are directly executed during actual matching to avoid repeated parsing of conditional expressions. The evaluation proceeds in descending order of access control policy priority, using a short-circuit strategy, that is, once all conditions of a policy are met, the matching result is returned immediately.

[0078] (e) The second access control policy includes three types: "Allow," "Deny," and "Requires Additional Approval." For the second access control policy indicating "Allow," the third data is sent to the target terminal corresponding to the first user. For the second access control policy indicating "Deny," an error response indicating access denial is returned to the target terminal corresponding to the first user, and a security alarm event is recorded. For the second access control policy indicating "Requires Additional Approval," the data access request is temporarily suspended, and an approval task is sent; access is granted only after approval.

[0079] In this embodiment, by introducing a multi-dimensional context-aware decision mechanism, it is possible to identify access requests that appear to be the same but have significant differences in context and make differentiated decisions, thereby effectively preventing the risk of data leakage while ensuring normal business efficiency.

[0080] Optionally, the method further includes at least one of the following: The system receives multiple operations input by a second user on the policy construction interface and generates an access control policy based on these operations. These operations include subject type selection, object type selection, access type selection, and environmental condition selection. Receive policy description language input from a second user, and generate access control policies based on the policy description language; Generate access control policies based on historical access logs.

[0081] In one optional implementation, the data management platform provides a drag-and-drop policy building interface, where a second user can input multiple operations to generate access control policies. These operations include subject type selection, object type selection, access type selection, and environmental condition selection. The second user can be understood as a data management personnel.

[0082] For example, from the policy building interface panel, select the subject type (user / user group / role), object type (data category label / sensitivity level / specific data table), operation type (read / write / delete / export / share), and environmental conditions (time window / network zone / device type), and then combine the above elements in the central canvas area of ​​the policy building interface to form a complete access control policy.

[0083] Another alternative implementation involves receiving a policy description language input from a second user and generating an access control policy based on the policy description language.

[0084] For example, the policy description language might be: "ALLOW read ON TABLE classified:PII WHEREuser.role=FINANCE_ANALYST AND network.zone=OFFICE_INTRANET AND time.hourBETWEEN 9 AND 18". This would generate an access control policy allowing the financial analyst role to access a data table classified as personally identifiable information via the office intranet between 9:00 AM and 6:00 PM on weekdays. It should be understood that the above policy description language supports nested conditions, regular expression matching, and user-defined function extensions.

[0085] Another alternative implementation is to set up a policy adaptive recommendation mechanism based on historical access patterns, thereby reducing the burden of manual policy configuration. By analyzing historical access logs and using statistical analysis techniques, typical data access patterns of users in different departments and roles can be identified, and access control policies can be generated.

[0086] For example, if analysis of historical access logs reveals that analysts in the finance department access financial statement data intensively from the 1st to the 5th of each month, an access control policy can be automatically created that allows financial analysts to read financial statements on weekdays from the 1st to the 5th of each month.

[0087] Optionally, before the access control policy takes effect, a simulation test can be performed on the access control policy. A second user inputs virtual access request parameters, which simulates the execution of the access control policy, and the execution result is returned to help the second user verify the accuracy of the policy.

[0088] Optionally, each creation, modification, or deletion of an access control policy is recorded as a new policy version, with the version number automatically incrementing. Secondary users can view the policy change history and roll back to any historical version as needed.

[0089] In other embodiments, such as Figure 6 As shown, an audit monitoring module can be set up for audit tracing. Specifically, it includes the following steps: (a) Audit Event Collection: After completing each access decision, the policy execution engine constructs an audit event object containing complete request information and sends it asynchronously to the audit collector via a high-performance lock-free queue. The asynchronous sending method ensures that the audit record will not become a critical path bottleneck in the access request processing.

[0090] (b) Audit event standardization: The audit collector formats the received audit events according to a unified data model, including fields such as event type (access request / policy change / classification change), timestamp, requesting user identifier, target data asset identifier, operation type, judgment result, matching policy identifier list, client IP, network area, device type, and requested data volume.

[0091] (c) Persistent Storage: Standardized audit events are written in batches to a time-series optimized columnar storage engine. Leveraging the compression and indexing features of columnar storage, efficient writing and querying of massive amounts of audit data is supported.

[0092] (d) Queries and Reports: The audit monitoring module provides a web-based visual interface, supporting combined queries by time range, user, data asset, operation type, network region, and other dimensions. The built-in reporting engine can automatically generate data security compliance reports on a daily / weekly / monthly basis. The report content includes: sensitive data access volume trends, access denial event statistics, abnormal access analysis outside of working hours, and identification of high-frequency data export behavior.

[0093] In addition, it supports setting access behavior baselines for specific data assets or users, and automatically triggers alarm notifications when the actual access pattern deviates from the baseline.

[0094] Please see Figure 6 , Figure 6 This is a schematic diagram of the application architecture of the data management method provided in the embodiments of this application. Figure 6 The application architecture shown includes a metadata acquisition module, a data classification and grading engine, a data lineage analysis engine, a policy management center module, a policy propagation engine, a policy synchronization module, a policy execution engine, and an audit and monitoring module. Through the collaborative work of these modules, the data management method described in the above embodiment is realized.

[0095] Further reference Figure 7 As a response to the above Figure 1 To implement the method shown, this application provides an embodiment of a data management device 700, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0096] An embodiment of the present invention provides a data management device 700 comprising: Acquisition module 701 is used to acquire multiple data points; The first determining module 702 is used to perform data analysis on each data and determine the data label corresponding to each data. Module 703 is used to establish a data lineage diagram based on the data routing relationship between each data item; The second determining module 704 is used to perform syntax analysis on the SQL statements related to the data scheduling task to construct an abstract syntax tree, determine the data routing relationship between each data according to the correspondence between the source field and the target field extracted from the abstract syntax tree, and determine multiple second data that have a data routing relationship with the first data by traversing along the directed edge starting from the first data based on the data lineage graph. The first update module 705 is used to determine the highest confidentiality level among the sensitivity level tags of the first data and the current sensitivity level tags of each second data according to the data tags after the first data is changed, and to update the sensitivity level tag corresponding to each second data to the target sensitivity level tag. The adjustment module 706 is used to adjust the access control policy corresponding to each second data according to the updated data tag of each second data.

[0097] Optionally, the second determining module 704 is specifically used for: Starting from the first data, a breadth-first traversal or a depth-first traversal is performed along the directed edges in the data lineage graph, and the maximum depth of the traversal is configurable.

[0098] Optionally, the establishment module 703 is specifically used for: Perform syntax analysis on SQL statements related to data scheduling tasks to determine the data routing relationship between each piece of data, and / or parse event logs to determine the data routing relationship between each piece of data; The data routing relationships between each data point are mapped to a preset graph database to obtain the data lineage graph; The data lineage diagram includes a data version identifier.

[0099] Optionally, the data management device 700 further includes: A receiving module is used to receive a policy update request, the policy update request being used to update the access control policy, and the policy update request carrying the updated policy version; The calculation module is used to respond to the policy update request and calculate the version difference between the updated policy version and the current policy version; The second update module is used to update the access control policy based on the version difference set.

[0100] Optionally, the data management device 700 further includes: The acquisition module is used to acquire, upon receiving a data access request sent by a first user, the data tag corresponding to the third data, the identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user. The data access request is used to represent access to the third data. The third determining module is used to determine multiple first access control policies based on the data tags corresponding to the third data; The matching module is used to match the identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user with each first access control policy to obtain a second access control policy. The execution module is used to perform access operations on the third data based on the second access control policy.

[0101] Optionally, the data management device 700 further includes: The first generation module is used to receive multiple operations input by the second user on the policy construction interface, and generate access control policies based on the multiple operations. The multiple operations include subject type selection operation, object type selection operation, access type selection operation and environment condition selection operation. The second generation module is used to receive the policy description language input by the second user and generate an access control policy based on the policy description language. The third generation module is used to generate access control policies based on historical access logs.

[0102] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0103] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that only the computer device 8 with components 81-83 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0104] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0105] The memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 8. Of course, the memory 81 may also include both the internal storage unit and its external storage device of the computer device 8. In this embodiment, the memory 81 is typically used to store the operating platform and various application software installed on the computer device 8, such as program code for data management methods. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or will be output.

[0106] In some embodiments, the processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run program code stored in the memory 81 or process data, for example, to run program code for the data management method.

[0107] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 8 and other electronic devices.

[0108] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described data management method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware online platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0110] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0111] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A data management method, characterized in that, The method includes: Collect multiple data points; Perform data analysis on each data point to determine the corresponding data label for each data point; Based on the data routing relationships between each data point, a data lineage diagram is established. Syntax analysis is performed on the SQL statements related to the data scheduling task to construct an abstract syntax tree. The correspondence between the source fields and the target fields extracted from the abstract syntax tree is used to determine the data routing relationship between each data. Based on the data lineage graph, the first data is used as the starting point to traverse along the directed edges to determine multiple second data that have a data routing relationship with the first data. Based on the data tags after the first data is changed, the highest confidentiality level among the sensitivity level tags of the first data and the current sensitivity level tags of each second data is determined as the target sensitivity level tag, and the sensitivity level tag corresponding to each second data is updated to the target sensitivity level tag; Based on the updated data tag of each second data, adjust the access control policy corresponding to each second data.

2. The method according to claim 1, characterized in that, The traversal along the directed edges starting from the first data includes: Starting from the first data, a breadth-first traversal or a depth-first traversal is performed along the directed edges in the data lineage graph, and the maximum depth of the traversal is configurable.

3. The method according to claim 1, characterized in that, The step of establishing a data lineage diagram based on the data routing relationships between each data item includes: Perform syntax analysis on SQL statements related to data scheduling tasks to determine the data routing relationship between each piece of data, and / or parse event logs to determine the data routing relationship between each piece of data; The data routing relationships between each data point are mapped to a preset graph database to obtain the data lineage graph; The data lineage diagram includes a data version identifier.

4. The method according to claim 1, characterized in that, The method further includes: Receive a policy update request, the policy update request being used to update the access control policy, and the policy update request carrying the updated policy version; In response to the policy update request, calculate the version difference between the updated policy version and the current policy version; Update the access control policy based on the version difference set.

5. The method according to claim 1, characterized in that, The method further includes: Upon receiving a data access request from the first user, the system obtains the data tag corresponding to the third data, the identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user. The data access request is used to represent access to the third data. Based on the data tags corresponding to the third data, multiple first access control policies are determined; The identity information of the first user, the operation information corresponding to the data access request, the time window to which the data access request belongs, and the network environment of the target terminal corresponding to the first user are matched with each first access control policy to obtain a second access control policy. Based on the second access control policy, an access operation is performed on the third data.

6. The method according to any one of claims 1-4, characterized in that, The method further includes at least one of the following: The system receives multiple operations input by a second user on the policy construction interface and generates an access control policy based on these operations. The multiple operations include subject type selection, object type selection, access type selection, and environmental condition selection. Receive policy description language input from a second user, and generate access control policies based on the policy description language; Generate access control policies based on historical access logs.

7. A data management device, characterized in that, The device includes: The data acquisition module is used to collect multiple data points. The first determining module is used to perform data analysis on each data and determine the data label corresponding to each data. A module is established to create a data lineage graph based on the data routing relationships between each data item. The second determining module is used to perform syntax analysis on the SQL statements related to the data scheduling task to construct an abstract syntax tree, determine the data routing relationship between each data based on the correspondence between the source fields and the target fields extracted from the abstract syntax tree, and determine multiple second data that have a data routing relationship with the first data based on the data lineage graph, starting from the first data and traversing along the directed edges. The first update module is used to determine the highest confidentiality level among the sensitivity level tags of the first data and the current sensitivity level tags of each second data as the target sensitivity level tag based on the data tags of the first data after the data change, and update the sensitivity level tag corresponding to each second data to the target sensitivity level tag. The adjustment module is used to adjust the access control policy corresponding to each second data according to the updated data tag of each second data.

8. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data management method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data management method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data management method as described in any one of claims 1 to 6.

Images