Data management method, system and device and storage medium

Abstract

The invention relates to the technical field of data governance, in particular to a data governance method, system and device and a storage medium. The data management method comprises the following steps: acquiring original business data; storing the original business data to a specified position of a database in a distributed manner, and converting the data format of the original business data into a uniform structure model to obtain integrated business data; monitoring the operation of the database management system on the database, and when it is monitored that the database management system performs a table structure change operation on the database, collecting graph demand information of business data in the database, and constructing a data blood relationship graph of the business data according to the graph demand information; and performing encryption processing on the integrated business data and the data blood relationship map to complete processing of the original business data. According to the invention, the convenience of analyzing and managing the business data by the user can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a data governance method, system, device and storage medium. Background Art

[0002] With the deepening development of the big data era, data governance technology has become the core support for enterprises to achieve data asset management. Current mainstream data governance systems generally adopt a layered architecture design, exemplified by commercial solutions such as IBM InfoSphere and Informatica MDM, as well as standardized frameworks based on the DAMA-DMBOK theory. A typical data governance system architecture typically includes the following core modules: a data source layer (integrating structured and unstructured multi-source data), a data acquisition / extraction module (including format preprocessing), a data storage layer (data warehouse / lake), a metadata management and quality monitoring module (implementing integrity and accuracy verification), a security and permissions module (including encryption and access control), and upper-layer data application services.

[0003] Although the existing architecture has formed a mature paradigm at the basic data processing level, there are still significant technical bottlenecks in practical applications: First, the rigid interface design between modules leads to excessive architectural coupling, making it difficult to adapt to the dynamically expanding heterogeneous data sources (such as streaming data and edge node data) in the era of the Internet of Things; second, data quality monitoring mostly adopts a T+1 batch processing mode, which cannot meet the real-time anomaly detection requirements of scenarios such as financial risk control; finally, the security control module and the application layer permission system lack a coordination mechanism, making it difficult to build a cross-level data security closed loop.

[0004] The rigid design of existing architectures can no longer meet the agility, real-time, and intelligent requirements of data governance systems driven by digital transformation. Especially when faced with new business scenarios such as massive real-time data stream processing, dynamic metadata correlation analysis, and cross-domain security policy collaboration, traditional technology systems expose systemic flaws such as response delays and dramatically increased maintenance costs, hindering user management of business information. Summary of the Invention

[0005] This application aims to at least solve the technical problems existing in the prior art and provide a data governance method, system, device and storage medium.

[0006] In a first aspect, the present invention provides a data governance method, comprising:

[0007] Obtain original business data;

[0008] Distribute and store the original business data in a designated location in the database, and convert the data format of the original business data into a unified structure model to obtain integrated business data;

[0009] Monitor the database management system's operations on the database. When the database management system is detected performing table structure changes on the database, the system collects graph demand information of the business data in the database and constructs a data lineage graph of the business data based on the graph demand information. The data lineage graph includes several nodes and several edges connecting the nodes. Each node represents a data object, and the edges represent the relationship between data objects.

[0010] The integrated business data and data lineage graph are encrypted to complete the processing of the original business data.

[0011] In a second aspect, the present invention provides a data governance system, comprising:

[0012] Data acquisition module, used to obtain original business data;

[0013] The data storage module is used to store the original business data in a distributed manner in a designated location of the database, and convert the data format of the original business data into a unified structure model to obtain the integrated business data;

[0014] The data management module is used to monitor the database management system's operations on the database. When it detects that the database management system is performing a table structure change operation on the database, it collects graph demand information of the business data in the database and constructs a data lineage graph of the business data based on the graph demand information. The data lineage graph includes several nodes and several edges used to connect the nodes. Each node represents a data object, and the edges represent the relationship between data objects.

[0015] The data security control module is used to encrypt the integrated business data and data lineage graph to complete the processing of the original business data.

[0016] In a third aspect, the present invention provides an electronic device, comprising:

[0017] at least one processor; and,

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data governance method described above.

[0020] In summary, this application has the following beneficial technical effects:

[0021] The collected original business data is stored in the database using distributed storage technology, and the original business data is converted into a unified data model to ensure data consistency and high availability, so as to facilitate users' management of heterogeneous data and to facilitate users' large-scale data integration and complex data analysis of business data in the database; by constructing a data lineage graph, the relationship between data objects can be visualized, and users can clearly view the relationship between data objects through a graphical interface; the nodes of the data lineage graph contain attribute information of the data objects, and users can query according to the name, type and other conditions of the data object to quickly locate the data object and its upstream and downstream relationships, so as to facilitate users' analysis and management of business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a data governance method provided by one embodiment of the present invention;

[0023] Figure 2 A data lineage graph according to an embodiment of the present invention;

[0024] Figure 3 A functional module diagram of a data governance system provided by one embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of an electronic device for implementing the data governance method provided in one embodiment of the present invention.

[0026] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0027] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0028] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0029] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0030] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0031] Reference Figure 1 FIG. 1 is a flow chart of a data governance method according to an embodiment of the present invention. In this embodiment, the data governance method includes:

[0032] S1. Obtain original business data.

[0033] Specifically, the original business data is at least one of structured data, semi-structured data and unstructured data.

[0034] Among them, structured data refers to data that can be represented and stored using a relational database, usually in the form of a two-dimensional table; structured data includes data in a relational database (such as customer data, order data, product data, etc.) and tabular data (such as transcripts, financial statements, etc.). Each row of structured data represents the information of an entity, and each column of data has strict homogeneity.

[0035] Semi-structured data does not completely follow a fixed architectural pattern, but contains self-describing tags or labels to reflect the hierarchical relationship of data; common semi-structured data includes: HTML documents, JSON, XML, and data in NoSQL databases.

[0036] Unstructured data refers to data without a fixed structure, usually including documents, pictures, videos, audio, etc. in various formats; unstructured data has no fixed fields and types.

[0037] Specifically, the steps to obtain original business data are:

[0038] S11. Monitoring business data collection instructions.

[0039] S12: When a business data collection instruction is detected, respond to the business data collection instruction to determine the business scenario and data format of the data source corresponding to the original business data to be collected.

[0040] S13. Filter out data acquisition rules that match the data source from the data acquisition rule library according to the business scenario and data format of the data source as target data acquisition rules, and generate control instructions based on the target data acquisition rules to control the acquisition plug-in to extract data from the data source according to the control instructions to obtain original business data.

[0041] The collection rule library pre-stores a variety of communication protocols related to data collection, such as HTTP API, JDBC, FTP, Kafka, MQTT, and WebSocket. Multiple collection rules are formulated based on multiple communication protocols and stored in the database:

[0042] HTTP API stands for Hypertext Transfer Protocol Application Programming Interface. HTTP is an application layer protocol used to transmit information over the internet. HTTP uses a request / response model, with clients sending requests to servers. It's well-suited for data exchange based on web services and can efficiently collect data from systems that provide HTTP interfaces.

[0043] JDBC stands for Java Database Connectivity. JDBC is an API and protocol used for interaction between Java applications and databases. It is used to connect to various databases, realize real-time data collection in the database, and support mainstream database management systems such as MySQL, Oracle, SQL Server, etc.

[0044] FTP stands for File Transfer Protocol. It is a protocol used to transfer files over the Internet. It allows users to transfer files between different computer systems. FTP supports downloading files from remote servers through the FTP protocol and is suitable for collecting file storage data sources.

[0045] MQTT stands for Message Queuing Telemetry Transport. It is a lightweight publish / subscribe messaging protocol suitable for device communication in low-bandwidth, high-latency, or unstable network environments. It is particularly well-suited for Internet of Things (IoT) scenarios. MQTT enables data transmission between devices and servers through the lightweight MQTT protocol, supporting low-bandwidth, high-latency network environments.

[0046] Kafka is a distributed stream processing platform for distributed message queue systems. It can efficiently read data streams from Kafka topics and is suitable for large-scale distributed data collection scenarios.

[0047] WebSocket is a full-duplex communication protocol based on TCP that supports real-time two-way communication. The client and server can actively send data at the same time without waiting for each other's request. The WebSocket protocol can receive data in real time from servers that support the WebSocket protocol, making it suitable for data collection scenarios that require real-time interaction.

[0048] After receiving the business data collection instruction, the system will automatically determine the communication protocol that matches the original business data to be collected based on the business scenario and data format of the data source corresponding to the original business data to be collected, and then determine the data collection rules that are compatible with the data source corresponding to the original business data to be collected as the target data collection rules; based on the target data collection rules, control instructions are generated to control the collection plug-in to extract data from the data source according to preset rules.

[0049] In the preferred implementation of this embodiment, the collection plug-in can be set to multiple types to meet the data collection needs in different scenarios; specifically, the collection plug-in can select at least one of Apache Flume and Logstash. Apache Flume is specifically used for log collection and can efficiently collect, aggregate and transmit large amounts of log data, and is suitable for large-scale log data collection scenarios; while Logstash is suitable for the collection and parsing of complex log structures, has powerful data processing capabilities, can filter, convert and format log data, supports multiple input and output plug-ins, and can flexibly configure the data collection process. In addition, the collection plug-in can also use the collection plug-in self-developed module to extract and collect business data from the data source. The collection plug-in self-developed module can be designed for specific application scenarios such as government platforms, supports automatic structure recognition of common data formats such as XML, JSON, Excel, etc., and can automatically parse and extract data without manual intervention to improve the efficiency and accuracy of data collection.

[0050] In a preferred implementation of this embodiment, after the business data of the data source is captured using the acquisition plug-in, the captured business data is cleaned and converted (such as null value processing and outlier filtering) to improve the data access speed and quality and obtain the original business data.

[0051] By setting up a collection rule library, the system will automatically match and apply corresponding rules during the data collection process, eliminating the need for users to manually write complex rule scripts. This further simplifies the data collection configuration process and improves the efficiency and accuracy of data collection. Users can quickly complete data source access and data collection task configuration without professional development capabilities, greatly improving the ease and flexibility of data collection and reducing the overall cost of data collection.

[0052] S2. Distribute and store the original business data in a designated location of the database, and convert the data format of the original business data into a unified structure model to obtain integrated business data.

[0053] In this embodiment, distributed storage technology is used to build a unified data lake architecture, integrating formats such as HDFS (distributed file system - underlying storage), Hive (data warehouse tool - metadata management), and Parquet (column storage format - efficient analysis), synchronously archiving structured and unstructured data, uniformly converting heterogeneous interfaces, and providing a unified schema to solve the architectural fragmentation problem of "separation of warehouse and lake" in traditional systems.

[0054] Heterogeneous data mainly includes structured data such as Oracle, MySQL, and PostgreSQL data tables, semi-structured data such as XML, JSON, YAML files, and REST API responses, as well as unstructured data such as Office documents, images, audio and video, scans, time series data, and graph data. The time series data in the database comes from sensors or log streams (IoT), and the graph data in the database comes from social networks or data lineage analysis (Neo4j).

[0055] Since different data sources (such as Kafka streaming data, MySQL relational tables, and S3 files) have different access interfaces, after the original business data is distributed and stored in the designated location of the database, the data format of the original business data is converted into a unified structural model, and heterogeneous interfaces are uniformly converted; for unstructured data without a schema (such as CSV), the schema is dynamically mapped on demand; for structured data with a strong schema (such as Hive tables), the original schema constraints are retained.

[0056] By abstracting underlying differences through a unified metadata layer (such as Hive Metastore), a standard SQL interface or API (such as Spark SQL) is provided to the outside world. Through schema mapping, users can use the same set of SQL to simultaneously query heterogeneous data within the data lake.

[0057] By integrating the original business data, it is possible to quickly integrate, archive and interact with heterogeneous data to ensure data consistency and high availability.

[0058] S3. Monitor the operations of the database management system on the database. When it is detected that the database management system performs table structure change operations on the database, the graph requirement information of the business data in the database is collected, and a data lineage graph of the business data is constructed based on the graph requirement information.

[0059] Graph requirement information includes operational log events and data flow paths for business data. In this embodiment, graph requirement information is recorded as metadata for business data within the database. Metadata is structured data that describes the attributes, structure, and relationships of business data. In this embodiment, metadata (such as field changes and table relationships) is automatically collected based on data event triggers, and a metadata warehouse (supporting time version management) is constructed to obtain graph requirement information. A data lineage graph (GraphDB visualization) is automatically constructed, significantly reducing the cost of manual data registration and maintenance.

[0060] During the execution of data processing tasks, a hook mechanism captures and writes log events in real time. The hook mechanism inserts event records at key operation points (such as data reading, conversion, and writing), ensuring that every step of the data processing process is accurately recorded. These log events contain key information such as the data processing timestamp, operation type, and the data objects involved, providing foundational data for subsequent metadata management and lineage graph construction.

[0061] Log events use a standardized format to record key information including timestamp, operation type, data object, and context information: the timestamp is used to record the specific time when the event occurred, accurate to the millisecond level; operation types include read, write, conversion, etc.; data objects include table names, field names, file paths, etc.; context information includes task ID, execution node, etc., which is used to associate and track data processing processes and other content.

[0062] When the database management system is detected to be performing a table structure change operation on the database, the graph requirement information of the business data in the database is collected, and a data lineage graph is constructed based on the graph requirement information. In this embodiment, a graph database such as Apache Atlas or Neo4j can be used to construct a metadata graph model. Apache Atlas is an open source metadata management and governance tool that supports a rich set of metadata management functions; Neo4j is a high-performance graph database that can efficiently store and query complex relational data. Users can choose a suitable graph database to store and manage metadata according to actual needs to obtain a data lineage graph for business data.

[0063] In the preferred implementation of this embodiment, a lightweight graph database (such as Neo4j) is used to construct a data lineage graph. The data content of the lightweight graph database (such as Neo4j) mainly includes nodes, relationships, attributes, labels, indexes and constraints. Among them, nodes are basic entities with attributes and labels; relationships are connections between nodes with types and attributes; attributes are represented in the form of key-value pairs to describe nodes and relationships; labels are used to classify nodes; indexes are used to speed up queries; and constraints are used to ensure data integrity. Compared with traditional relational databases or other complex data storage solutions, it has higher performance and flexibility. Lightweight graph databases can efficiently store and query complex relational data, support rapid processing and analysis of large-scale data, and meet the high performance requirements of modern data processing systems for metadata management and lineage graph construction.

[0064] The data lineage graph includes several nodes and several edges used to connect the nodes. Each node represents a data object (such as a table or field in a database, a file in a file system, etc.), and the edge represents the relationship between data objects.

[0065] Node attributes include at least one of the following: table name, field name, file path, data type, data source, and data format. Edges represent dependencies or call relationships between data objects, such as associations between tables or mappings between fields. Specifically, edge types include dependency and call relationships. A dependency relationship indicates that one data object depends on another, such as the data in Table A originating from Table B. A call relationship indicates that one data object is called by another, such as a stored procedure calling data in a table.

[0066] Through graph databases (such as Neo4j), the relationship between nodes and edges can be visualized, so that the data lineage graph can be displayed in an intuitive graphical form. The visualized data lineage graph is as follows: Figure 2 As shown; Figure 2 In a corresponding digital lineage graph, the source table is the table containing raw data, such as insurance_fact_user_base_information; the target table is the processed or transformed table, such as insurance_fact_user_base_information1. Edge relationships in a data lineage graph include field mappings and table dependencies. Field mapping maps fields in a table (such as user_id, job_type, and document_type) to corresponding fields in the target table; table dependencies refer to the target table's dependence on data from the source table.

[0067] After generating a data lineage graph based on the business data in the database, users can query based on conditions such as the name and type of the data object to quickly locate the data object and its upstream and downstream relationships.

[0068] Users can clearly view the relationships between data objects through a graphical interface. Support for topological queries and difference comparisons allows users to quickly locate data issues and analyze data flow paths. During data audits, the data lineage graph can be used to review the complete path of business data from source to end use. The data lineage graph records the processing process and user information of business data at each node. This provides strong support for data compliance checks and data security incident investigations, helping users quickly locate the root cause of problems and take appropriate measures.

[0069] When business data in the database changes, the database management system will re-collect the metadata information of the business data in the database and update the data lineage graph based on the newly collected metadata information. In the preferred implementation of this embodiment, the data lineage graph of different versions (time points) is retained, and the comparison of different versions of the data lineage graph is supported, so that changes in the data flow path can be intuitively displayed. Users can trace historical problems in the data processing process and analyze the causes and impacts of data changes, providing an important basis for data governance and system optimization.

[0070] S4. Encrypt the integrated business data and data lineage graph to complete the processing of the original business data.

[0071] In some examples of this embodiment, a symmetric encryption algorithm is used to encrypt static data stored in a storage medium such as a database or a file system. For example, the symmetric encryption algorithm AES-256 is used to process the integrated business data and data lineage graph. AES-256 is an advanced encryption standard that uses a 256-bit key length. AES-256 has high security and efficient encryption performance. It supports a 256-bit key length and can effectively resist brute force attacks to ensure the security of static data. AES-256 supports encryption of sensitive data fields (such as user personal information, financial data, etc.), and also supports encryption of the entire data file. Users can flexibly select the encryption scope according to actual needs to balance data security and system performance. In other examples of this embodiment, asymmetric encryption and hybrid encryption algorithms can also be used to process static integrated business data and data lineage graphs. Users can choose a suitable encryption method based on the actual situation of the business information, and this embodiment does not limit it.

[0072] Furthermore, TLS / SSL protocols can be used to achieve full-link encryption during data transmission; specifically, TLS 1.2 or higher can be used to ensure data security during transmission. TLS protects data from eavesdropping, tampering, or forgery during transmission by encrypting the communication channel. Full-link encryption covers all communication links from client to server, including data requests, responses, and forwarding at intermediate nodes. Whether on internal or external networks, data remains encrypted at all times, effectively preventing man-in-the-middle attacks and data leaks.

[0073] TLS / SSL full-link encryption and AES-256 static data encryption ensure data security during transmission and storage. The platform also provides comprehensive key management capabilities, supporting key generation, storage, distribution, updates, and rotation to ensure the security and reliability of the encryption process. This comprehensive data encryption and key management mechanism effectively prevents data leakage and tampering risks, protecting the company's core data assets.

[0074] When a user wants to view business data or a blood relationship diagram in a database, he or she may issue a data call instruction through an external terminal. The external terminal may be a computer, mobile phone, tablet computer or other electronic device used by the user, and this embodiment does not impose any restrictions.

[0075] In a preferred embodiment of the present application, a data governance method further includes:

[0076] S51. Monitoring a data call instruction, and when a data call instruction is detected, sending an access rights verification request to the terminal that issued the data call instruction;

[0077] S52, receiving access permission verification information sent by the terminal that issued the data call instruction in response to the access permission verification request;

[0078] S53. Evaluate the access permission verification information and obtain the evaluation result:

[0079] If the evaluation result is verification passed, a data reading interface is generated;

[0080] If the evaluation result is verification failure, the user will be denied access to the data content in the database.

[0081] There are two types of access control methods: role-based access control (RBAC) and attribute-based access control (ABAC).

[0082] The role-based access control model defines different roles, such as administrators, regular users, and auditors, based on the organizational structure and business needs. Each role has a predefined set of permissions that reflect the role's responsibilities and scope of operations within the system. Users are associated with corresponding roles, and the users inherit the permissions of the role. By assigning and adjusting roles, user permissions can be quickly managed and adjusted in batches, improving the efficiency and flexibility of permission management. Inheritance relationships and permission combinations between roles are supported. For example, a senior administrator role can inherit the permissions of a regular administrator role and add additional advanced permissions. This inheritance and combination mechanism makes permission management more flexible and adaptable to complex organizational structures and business scenarios.

[0083] The attribute-based access control model defines a rich set of user attributes, resource attributes, and environmental attributes. User attributes include the user's identity information, department, position, etc.; resource attributes include the sensitivity level of the data, data type, data ownership, etc.; environmental attributes include access time, access location, access device, etc. Flexible access control policies are formulated based on the relationship between attributes. Policies can be defined based on a single attribute or a combination of multiple attributes. For example, users can access low-sensitivity data during working hours, or users in a specific department can access the department's data. The ABAC model can achieve more refined permission control to meet complex business needs and security requirements. Policy Engine: The platform has a built-in powerful policy engine that is responsible for real-time evaluation and decision-making of access requests. Based on predefined policies and current attribute information, the policy engine quickly determines whether the access request is legitimate and makes corresponding access control decisions. The policy engine supports efficient policy matching and execution to ensure system performance and security.

[0084] In the preferred implementation of this embodiment, a hybrid RBAC and ABAC model is used to verify user identities. This hybrid model combines the efficiency of RBAC with the flexibility of ABAC, meeting the rights management needs of complex organizational structures and diverse business scenarios, and achieving more refined and flexible rights control. Users can flexibly select and combine RBAC and ABAC policies based on actual needs, improving the adaptability and scalability of rights management.

[0085] In this embodiment, the access permission verification information includes at least one of user role information and attribute information.

[0086] Methods for evaluating access verification information include:

[0087] Parse the access permission verification information. If the access permission verification information is user role information, select the role-based access control mode, evaluate the user's access permission based on the user role information, and obtain the evaluation result;

[0088] If the access permission verification information is attribute information, the attribute-based access control mode is selected to evaluate the user's access permission based on the attribute information to obtain an evaluation result.

[0089] In a preferred implementation of this embodiment, a data governance method further includes: monitoring business events related to user roles, and determining whether the user's access rights need to be adjusted based on the content of the business events.

[0090] Business events that can trigger adjustments to user access rights include user role changes, organization mergers or splits, and data classification changes. Specific scenarios may include the following:

[0091] (1) When a user's position or role changes, for example, an employee is promoted from an ordinary employee to a department manager, the system automatically detects this business event and adjusts the corresponding permissions of the employee;

[0092] (2) When the organizational structure changes, such as department mergers or splits, the user permissions involved need to be reallocated;

[0093] (3) When the sensitivity level or classification of data changes, the user permissions for accessing the data also need to be adjusted accordingly. For example, if a dataset is upgraded from a low sensitivity level to a high sensitivity level, the user permissions for accessing the dataset need to be re-evaluated.

[0094] In this embodiment, real-time business event information can be obtained by integrating with the company's existing human resources management system (HRMS), organizational structure management system, etc. Custom event triggers are also supported, allowing administrators to configure event trigger conditions based on specific business needs, ensuring that the system can fully cover various business change scenarios.

[0095] In another implementation of this embodiment, a data governance method further includes:

[0096] S6. Monitor business data in the database in real time, identify and repair abnormal data in the database.

[0097] Specifically, real-time monitoring of business data in the database, identification and repair of abnormal data in the database, including:

[0098] S61. Detect abnormal data points in the database using an Isolation Forest model to obtain a first abnormality detection result.

[0099] Isolation Forest, which stands for "isolation forest," is an unsupervised anomaly detection algorithm that randomly partitions the data space and constructs an isolation tree structure. The shorter the path length of a data point in the tree, the more likely it is an outlier. The Isolation Forest algorithm is suitable for high-dimensional data and can effectively identify outliers in the data.

[0100] S62. Capture abnormal patterns in the time series data through a long short-term memory network model to obtain a second abnormality detection result.

[0101] Long Short-Term Memory (LSTM) is a time-recurrent neural network that can identify abnormal patterns in data by learning the temporal dependencies of data. It is suitable for scenarios with large data fluctuations. In a preferred embodiment of this embodiment, the reference threshold for abnormal data judgment is adaptively adjusted based on the statistical characteristics of historical data (such as mean, variance, quantiles, etc.) and the changing trends of real-time data. The dynamic reference threshold can better adapt to the dynamic changes of business data and improve the accuracy and flexibility of anomaly detection results.

[0102] S63. Filter out abnormal data from the database according to the first abnormality detection result and the second abnormality detection result.

[0103] For the monitored business data, if at least one of the first anomaly detection result and the second anomaly detection result indicates that the business data is abnormal, the business data for which at least one result indicates that the business data is abnormal is marked as abnormal data.

[0104] S64: Analyze the abnormal data to determine the abnormal triggering condition of the abnormal data.

[0105] When anomalous data is detected, the system immediately pushes it to Kafka's "abnormal channel." As a high-performance message queuing system, Kafka can efficiently process large amounts of real-time data, ensuring the timely transmission and storage of abnormal data. "Kafka" refers to Apache Kafka, a distributed stream processing platform. It is primarily designed to process high-throughput real-time data streams and is used in scenarios such as message queuing, event sourcing, log aggregation, and streaming data processing. A dedicated "abnormal channel" is set up in Kafka to isolate and manage abnormal data. This channel supports high-concurrency access and can quickly receive and forward abnormal data, facilitating subsequent processing and analysis.

[0106] After receiving the reported abnormal data, the system uses the preset rule engine to further analyze and process the abnormal data. The rule engine can classify and mark the abnormal data according to the preset rule logic.

[0107] In the rule engine, detailed rule matching is performed on the abnormal data to determine the abnormal trigger conditions of the abnormal data.

[0108] Common abnormal data triggering conditions include but are not limited to:

[0109] (1) Field is empty: Check whether there are key fields in the data that are empty. This may be a problem during data collection or transmission.

[0110] (2) Value range out of limit: Check whether the data value exceeds the preset reasonable range, such as temperature data exceeds the normal working range.

[0111] (3) Data format error: Verify whether the data meets the preset format requirements, such as date format, numeric format, etc.

[0112] (4) Timestamp anomaly: Check whether the timestamp of the data is logical, such as timestamp jump or repetition.

[0113] S65: Determine a repair strategy according to the abnormal triggering condition of the abnormal data, and generate a repair instruction according to the repair strategy to control the database to respond to the repair instruction and repair the abnormal data.

[0114] Determine the repair strategy based on the abnormal trigger conditions of the abnormal data. Specifically:

[0115] For the case where the field is empty, data filling is used to repair the abnormal data. Specifically, filling is performed based on historical data or preset default values.

[0116] For abnormal data with value range exceeding the limit, data correction is used to repair the abnormal data and make corrections based on context logic, such as adjusting the out-of-range value to a reasonable range.

[0117] For serious abnormal data that cannot be repaired, the system will discard it and generate an alarm message to notify relevant personnel for further processing.

[0118] After completing the self-repair of abnormal data, the repaired data is verified to ensure its accuracy and integrity. If the verification fails, the repair process is retriggered until the data returns to normal.

[0119] In the preferred implementation of this embodiment, detailed logging is performed throughout the self-repair process. This logging includes information such as the time the abnormal data was detected, the repair strategy, and the repair results. This logging provides an important basis for subsequent analysis and optimization. The system provides feedback on the self-repair results to the user or related systems. For successfully repaired data, a sample of the repaired data is provided; for data that failed to be repaired, detailed warning information and recommended follow-up measures are provided.

[0120] Reference Figure 3 Based on the same inventive concept, an embodiment of the present invention provides a data governance system.

[0121] The data management system described in the present invention can be installed in an electronic device. According to the functions to be implemented, the data management system includes a data acquisition module, a data storage module, a data management module, a data security control module, and a data quality monitoring module, wherein:

[0122] The data acquisition module can obtain original business data;

[0123] The data storage module can store the original business data in a distributed manner to a specified location in the database, and convert the data format of the original business data into a unified structure model to obtain integrated business data;

[0124] The data management module can monitor the database management system's operations on the database. When it detects that the database management system is performing table structure changes on the database, it collects graph demand information of the business data in the database and constructs a data lineage graph of the business data based on the graph demand information. The data lineage graph consists of several nodes and several edges connecting the nodes. Each node represents a data object, and the edges represent the relationship between data objects.

[0125] The data security control module can encrypt the integrated business data and data lineage graph to complete the processing of the original business data;

[0126] The data quality monitoring module can monitor the business data in the database in real time, identify and repair abnormal data in the database.

[0127] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0128] The various variations and specific examples of the data governance method provided in the above embodiments are also applicable to the data governance system of this embodiment. Through the above detailed description of the data governance method, those skilled in the art can clearly understand the implementation method of the data governance system in this embodiment. For the sake of brevity, they will not be described in detail here.

[0129] This application also discloses an electronic device, such as Figure 4 FIG2 is a schematic diagram of the structure of an electronic device implementing a data governance method according to an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 communicatively coupled to the at least one processor, a communication bus 12, and a communication interface 13. The electronic device may also include a computer program stored in the memory 11 and executable on the processor 10, such as a data governance method program.

[0130] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the memory 11 (such as methods for executing data management, etc.), as well as calling data stored in the memory 11, to perform various functions of the electronic device and process data.

[0131] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various types of data installed in the electronic device, such as the code of a data management method program, but also to temporarily store data that has been output or is to be output.

[0132] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0133] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0134] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation of the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) to power each component. Preferably, the power supply can be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0135] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0136] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0137] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "one implementation," "a preferred implementation," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0138] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A data governance method, characterized in that: The method comprises: Obtain original business data; Distribute and store the original business data in a designated location in the database, and convert the data format of the original business data into a unified structure model to obtain integrated business data; Monitor the database management system's operations on the database. When the database management system is detected performing table structure changes on the database, the system collects graph demand information of the business data in the database and constructs a data lineage graph of the business data based on the graph demand information. The data lineage graph includes several nodes and several edges connecting the nodes. Each node represents a data object, and the edges represent the relationship between data objects. The integrated business data and data lineage graph are encrypted to complete the processing of the original business data.

2. The data governance method according to claim 1, wherein: The obtaining of original business data includes: Monitoring business data collection instructions; When a business data collection instruction is detected, responding to the business data collection instruction, determining the business scenario and data format of the data source corresponding to the original business data to be collected; According to the business scenario and data format of the data source, the data collection rules that match the data source are filtered out from the data collection rule library as the target data collection rules, and control instructions are generated based on the target data collection rules to control the collection plug-in to extract data from the data source according to the control instructions to obtain the original business data.

3. The data governance method according to claim 1, wherein: The graph requirement information includes the operation log events and data flow paths of the business data; the attributes corresponding to the nodes include at least one of the table name, field name, file path, data type, data source and data format.

4. The data governance method according to any one of claims 1 to 3, wherein: The method further includes: monitoring business data in the database in real time, and identifying and repairing abnormal data in the database.

5. The data governance method according to claim 4, wherein: The real-time monitoring of business data in the database and identification and repair of abnormal data in the database include: Detect abnormal data points in the database using the Isolation Forest model to obtain a first anomaly detection result; The long short-term memory network model is used to capture abnormal patterns in time series data and obtain the second anomaly detection result; filtering abnormal data from a database according to the first abnormality detection result and the second abnormality detection result; Analyze abnormal data to determine the abnormal trigger conditions of the abnormal data; A repair strategy is determined according to the abnormal triggering condition of the abnormal data, and a repair instruction is generated according to the repair strategy to control the database to respond to the repair instruction and repair the abnormal data.

6. The data governance method according to claim 1, 2, 3 or 5, wherein: The method further comprises: monitoring a data call instruction, and when the data call instruction is detected, sending an access rights verification request to the terminal that issued the data call instruction; receiving access permission verification information sent by the terminal that issued the data call instruction in response to the access permission verification request; Evaluate the access permission verification information to obtain an evaluation result: If the evaluation result is verification passed, a data reading interface is generated.

7. The data governance method according to claim 6, wherein: The access permission verification information includes at least one of user role information and attribute information. Methods for evaluating the access rights verification information include: Parse the access permission verification information. If the access permission verification information is user role information, evaluate the user's access permission based on the user role information to obtain an evaluation result. If the access permission verification information is attribute information, the user's access permission is evaluated based on the attribute information to obtain an evaluation result.

8. A data governance system for implementing the data governance method according to any one of claims 1 to 7, characterized in that: include: Data acquisition module, used to obtain original business data; The data storage module is used to store the original business data in a distributed manner in a designated location of the database, and convert the data format of the original business data into a unified structure model to obtain the integrated business data; The data management module is used to monitor the database management system's operations on the database. When it detects that the database management system is performing a table structure change operation on the database, it collects graph demand information of the business data in the database and constructs a data lineage graph of the business data based on the graph demand information. The data lineage graph includes several nodes and several edges used to connect the nodes. Each node represents a data object, and the edges represent the relationship between data objects. The data security control module is used to encrypt the integrated business data and data lineage graph to complete the processing of the original business data.

9. The data management system according to claim 8, wherein: The system further comprises: The data quality monitoring module is used to monitor the business data in the database in real time and identify and repair abnormal data in the database.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively coupled to the at least one processor (10); Wherein, the memory (11) stores a computer program that can be executed by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the data governance method as described in any one of claims 1 to 7.

Images