Data exploration method and system

Abstract

The invention provides a data probing method and system. The method comprises the steps of metadata basic information probing, metadata field security attribute probing and metadata blood relationship probing. Null value, NULL value and non-null-duty ratio exploration, data volume and distribution exploration, field attribute and distribution exploration and security exploration are carried out; data security analysis, data integrity and distribution analysis, and data relationship and distribution analysis are carried out. According to the invention, through intelligent automatic analysis and identification, null values, redundant information and performance bottlenecks in data are found in real time, and accurate support is provided for subsequent data governance; the probing task can be automatically completed only by inputting basic database connection information and related parameters, so that the efficiency is improved, and manual intervention is reduced; the universality is high, different database environments and data scales are adapted, and the probing process is efficient and accurate; the structure, the use condition and the potential risk of enterprise data assets can be deeply known, so that data management is accurate and efficient, and digital transformation and intelligent decision making are promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of data exploration, and in particular to a data exploration method and system. Background Art

[0002] With the increasing application of big data in various industries, the importance of data governance has become increasingly prominent. Especially with the gradual strengthening of data compliance and privacy protection requirements, the status of data governance in the organization's information security system has been further strengthened.

[0003] To promote digital transformation and accelerate the management and utilization of data assets, data governance must be taken as a core link. There are many methods of data governance, and technical means such as data quality management, data cleaning, and data integration can effectively improve the availability and reliability of data; however, in a complex big data environment, how to comprehensively investigate and deeply evaluate data assets to ensure that the structure, quality, and usage of data are accurately identified has become a problem that needs to be solved urgently.

[0004] Data exploration is a technology used to obtain, understand and evaluate database information and structure, including: exploration of database structure, including table design, relational model, data type, etc.; exploration of data content, including data distribution, outliers, missing values, etc.; exploration of performance optimization, including query optimization, index optimization, etc.; exploration of security, including access rights, data encryption, etc. Although existing data exploration technologies can help enterprises analyze and identify data assets in databases, they have the following obvious shortcomings:

[0005] 1. Most existing data exploration methods rely on manual or semi-automatic means to analyze databases, which is inefficient. Especially when faced with large and complex databases, it is impossible to quickly complete comprehensive data exploration.

[0006] 2. Traditional data exploration techniques can usually only identify basic data problems, such as null values, abnormal data, and duplicate records, but lack in-depth analysis of complex relationships and potential problems between data.

[0007] 3. The existing data exploration technology has weak intelligent analysis capabilities, is difficult to adapt to the exploration and analysis of large-scale data and distributed database environments, and cannot achieve comprehensive and real-time risk identification. Summary of the invention

[0008] In view of this, the purpose of the present invention is to propose a data exploration method and system, and to design an efficient and intelligent data exploration method, which can automatically identify various types of data assets in the database, quickly discover potential problems in the data, and deeply explore the relationships and dependencies between data; through intelligent analysis, the system can efficiently process large-scale data, adapt to distributed database environments, and identify potential performance bottlenecks and risks in real time; users only need to provide database connection information and related parameters, and the system can automatically complete the data exploration process, significantly improve work efficiency, reduce human intervention, and provide strong support for subsequent data governance.

[0009] The present invention provides a data exploration method, comprising the following steps:

[0010] S1. Metadata basic information exploration: input database connection information and related parameters, establish database connection and obtain metadata information, scan core tables in the database, obtain field details, count index and constraint information, and store metadata; Metadata field security attribute exploration: identify sensitive fields, check field security measures, and detect field permission allocation; Metadata lineage relationship exploration: analyze the primary and foreign key relationships between tables, explore and analyze field-level dependencies, and build a metadata lineage map;

[0011] S2. Exploration of empty values, NULL values ​​and non-empty ratios: empty value statistics, NULL value statistics, non-empty ratio calculation; data volume and distribution exploration: no data statistics, data volume statistics, empty value ratio analysis; field attribute and distribution exploration: primary key length statistics, duplicate data exploration, value distribution exploration, value range analysis, dimension distribution analysis, value description generation; security exploration: security scanning and security policy recommendation;

[0012] S3. Data security analysis: sensitive data identification, security measures recommendations; data integrity and distribution analysis: data null value distribution analysis, data range distribution analysis; data relationship and distribution analysis: data relationship analysis, data distribution analysis, data volume ratio analysis, no data TOP analysis.

[0013] Furthermore, the database connection information in the input database connection information and related parameters of step S1 is necessary parameters provided by the user for connecting to the target database, including the database IP address, port number, user name, password, and database type (such as MySQL, PostgreSQL, Oracle, etc.);

[0014] The relevant parameters are the exploration scope (such as a specific database name or table name) that needs to be selected by the user, so as to clarify the exploration target;

[0015] The method of establishing a database connection and obtaining metadata information in step S1 includes:

[0016] Connect to the target relational database according to the provided database connection information; obtain the database metadata information through the system table or database management interface; metadata information includes table structure, field definition, index, primary and foreign keys, etc.;

[0017] The method for scanning the core table in the database in step S1 includes:

[0018] Automatically identify user tables, system management tables, and space information tables, and match and filter target tables by table name patterns (such as "USER%", "SYS%", "SPATIAL%"); extract basic table information, such as table name, table type, storage engine, etc.

[0019] Furthermore, the method for obtaining detailed field information in step S1 includes:

[0020] For each table, scan the fields one by one and extract the field definition information, which includes the field name, data type (such as varchar, int, decimal, etc.), field length, whether null values ​​are allowed, default value, and whether it is a primary key or foreign key.

[0021] The method for statistical index and constraint information in step S1 includes:

[0022] Extract the primary key index, unique index, and common index information from the table definition, and count the coverage and number of each index; at the same time, check the constraint definition of the table, such as whether the foreign key constraint is associated with a valid primary key field. Preferably, analyze the index type and constraint information used in the table, and evaluate the index coverage and redundancy.

[0023] The method for storing metadata in step S1 includes:

[0024] The information of tables and fields is stored in the repository for managing metadata, and a metadata list is generated, including the basic information of the table, field definitions, indexes and constraints, etc., to support subsequent exploration and analysis.

[0025] Furthermore, the method for identifying sensitive fields in step S1 includes: filtering fields that may contain sensitive information by matching sensitive words (such as "ID number" and "bank card number") and field types (such as varchar, with a length greater than 20) by field names; preferably, identifying and marking fields that may contain sensitive data (such as ID number, mobile phone number, etc.) by field name and data type.

[0026] The method of checking the safety measures of the field in step S1 includes:

[0027] Check whether sensitive fields are encrypted and stored (for example, check whether they are in ciphertext format), whether they are desensitized (for example, partially displaying "****"), and whether access permissions are set. Specifically, analyze whether sensitive fields are encrypted, desensitized, or have access permissions restricted.

[0028] The method for detecting field permission allocation in step S1 includes: extracting access rights for sensitive fields from permission configuration, and checking whether there is unreasonable allocation (such as ordinary user groups having write permissions); specifically, evaluating whether the field access rights are too large or the allocation is unreasonable, especially for sensitive fields.

[0029] The method for analyzing the primary and foreign key relationships between tables in step S1 includes:

[0030] According to the definition of primary key and foreign key, the association relationship between tables is extracted to build table-level blood relationship; specifically, the primary and foreign key relationship between tables is analyzed to generate table-level blood relationship diagram;

[0031] The method for field-level dependency exploration and analysis in step S1 includes:

[0032] Analyze the derivation relationship and calculation logic between fields and mark the source of the fields. Specifically, analyze the dependency relationship between fields and generate a field-level lineage relationship diagram.

[0033] The method for constructing the metadata kinship map in step S1 includes:

[0034] Based on the dependency relationship between tables and fields, a visual lineage map is generated. Preferably, a complete database lineage map is generated to show the dependency relationship between tables and fields.

[0035] Furthermore, the method for counting null values ​​in step S2 includes:

[0036] Count the number of null values ​​(NULL values ​​and empty strings) in each field and calculate the ratio of null values ​​in the field to help identify which fields have a large number of null values ​​and mark the fields that may affect data integrity.

[0037] The method for counting NULL values ​​in step S2 includes:

[0038] Distinguish between explicit NULL values ​​and implicit null values ​​in a field (for example, the difference between an empty string and NULL), and calculate the total amount and proportion of explicit NULL values ​​and implicit null values ​​respectively to ensure the accuracy of NULL value statistics, especially the assessment of data quality.

[0039] The method for calculating the non-empty ratio in step S2 includes:

[0040] Calculate the non-null value ratio of each field, and evaluate the actual data utilization of the field by comparing the non-null value ratio; preferably, pay special attention to the non-null ratio of core business fields to ensure data availability.

[0041] The method without data statistics in step S2 includes:

[0042] Check each field to see if there is any data at all (that is, all values ​​are NULL or empty strings), and mark these abnormal fields. For fields with no data at all, generate a warning prompt and list the table name and field.

[0043] The method for counting the amount of data in step S2 includes:

[0044] Count the total number of records in each table, analyze the distribution of records in each field, help find tables or fields with too large or too small data volumes, and identify fields with abnormal data volumes; this step is important for detecting performance bottlenecks, storage capacity issues, and data missing.

[0045] The method for analyzing the null value ratio in step S2 includes:

[0046] According to the null value ratio of the field, record the fields with high null value ratio and the fields that are critical to the business logic. When the null value ratio of these fields exceeds the preset threshold, generate a warning message. Preferably, analyze the fields with high null value ratio and mark the abnormal fields that may affect the business.

[0047] Furthermore, the method for counting the primary key length in step S2 includes:

[0048] Check the length of all primary key fields and analyze whether the length of the primary key field affects the performance or storage efficiency of the database; specifically, count the length and rationality of the primary key field and check whether there are redundant or overly long primary key fields.

[0049] The method for repeated data exploration in step S2 includes:

[0050] Detect the number of duplicate records and duplicate values ​​of non-primary key fields in the table to ensure that there is no unnecessary redundant data in the table. By finding duplicate records, mark tables and fields containing a large amount of duplicate data (serious duplicate data).

[0051] The method for detecting the numerical distribution of step S2 includes:

[0052] Perform distribution analysis on all numeric fields (such as int and decimal), calculate the minimum, maximum, average, and standard deviation statistics of the fields, and mark extreme values ​​or outliers that do not meet expectations;

[0053] The method for range analysis in step S2 includes:

[0054] Analyze the value domain of enumeration types or limited value fields, and count the different values ​​of these fields. Through the value domain range, find abnormal values ​​or data outside the value domain in the field.

[0055] The method for dimensional distribution analysis in step S2 includes:

[0056] Analyze the value distribution of dimension fields (such as CITY, PRODUCT_CATEGORY, etc.) and mark fields with too concentrated or scattered value distribution (abnormal distribution).

[0057] The method for generating the value description of step S2 includes:

[0058] Generate a value description for each field based on the field's value range and frequency analysis; for enumeration fields, list the field's value range and main value proportions to help users understand the field content.

[0059] The method for security scanning and recommending security policies in step S2 includes:

[0060] Based on the data exploration results, security recommendation strategies are provided for sensitive fields, including whether encryption and desensitization are required, and whether access control needs to be increased; a security scan report is generated for each sensitive field, and security measures are recommended.

[0061] Furthermore, the method for identifying sensitive data in step S3 includes:

[0062] Combine business exploration results and sensitive field tags to analyze the distribution of sensitive data and assess the security of sensitive data (potential data security risks).

[0063] The recommended approach for the safety measures in step S3 includes:

[0064] Generate targeted security optimization suggestions based on the distribution of sensitive data and existing security measures; specifically, generate security optimization suggestions based on the results of the security scan report.

[0065] The method for analyzing the data null value distribution in step S3 includes:

[0066] Statistics on the null value distribution of all tables and fields are generated, a null value heat map is generated, and fields with too high null values ​​are marked; for business-critical fields, uneven distribution of null values ​​is marked as anomalies.

[0067] The method for analyzing the data range distribution in step S3 includes:

[0068] Analyze the concentration of the field value range, calculate the standard deviation, maximum value, minimum value, etc., identify extreme values ​​and outliers; preferably, mark outliers when found.

[0069] The method for data relationship analysis in step S3 includes:

[0070] Combined with the metadata lineage diagram, analyze the logical relationship between tables (analyze the dependencies and potential conflicts between data), and mark potential redundant fields, conflicting data, or unnecessary dependencies between tables by tracing the dependencies between fields.

[0071] The method for data distribution analysis in step S3 includes:

[0072] Count the data volume ratio of each field in the table, identify tables or fields with large data volume, and analyze possible performance bottlenecks. Performance bottlenecks can be found by analyzing the ratio of data tables or fields in the overall data volume.

[0073] The method for analyzing the data volume ratio in step S3 includes:

[0074] Generate a field data volume ratio chart to identify fields with severely unbalanced data volume and identify unbalanced data distribution.

[0075] The method for the no-data TOP analysis in step S3 includes: counting the number of fields without data, and generating fields without data for subsequent optimization.

[0076] The present invention also provides a data exploration system for executing the data exploration method as described above, comprising:

[0077] Metadata technical information exploration module: used for metadata basic information exploration: input database connection information and related parameters, establish database connection and obtain metadata information, scan core tables in the database, obtain field details, count index and constraint information, and store metadata; metadata field security attribute exploration: identify sensitive fields, check field security measures, and detect field permission allocation; metadata lineage exploration: analyze primary and foreign key relationships between tables, explore and analyze field-level dependencies, and build metadata lineage maps;

[0078] Business information exploration module: used for empty value, NULL value and non-empty ratio exploration: empty value statistics, NULL value statistics, non-empty ratio calculation; data volume and distribution exploration: no data statistics, data volume statistics, empty value ratio analysis; field attribute and distribution exploration: primary key length statistics, duplicate data exploration, value distribution exploration, value range analysis, dimension distribution analysis, value description generation; security exploration: security scanning and security policy recommendation;

[0079] Probe and analysis module: used for data security analysis: sensitive data identification, security measures recommendations; data integrity and distribution analysis: data null value distribution analysis, data range distribution analysis; data relationship and distribution analysis: data relationship analysis, data distribution analysis, data volume ratio analysis, no data TOP analysis.

[0080] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the data exploration method as described above are implemented.

[0081] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data exploration method as described above when executing the program.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] The data exploration method and system provided by the present invention can automatically identify various types of data assets in the database through intelligent automatic analysis and identification, quickly discover potential problems in the data, and deeply explore the relationships and dependencies between the data; through intelligent analysis, it can efficiently process large-scale data, adapt to distributed database environments, and can discover problems such as null values, redundant information and performance bottlenecks in the data in real time, providing accurate support for subsequent data governance; the user only needs to input basic database connection information and related parameters to automatically complete the exploration task, greatly improving work efficiency and reducing manual intervention; and the data exploration method has strong versatility, can adapt to different database environments and data scales, and ensures the efficiency and accuracy of the exploration process; through the data exploration method, it is possible to deeply understand the structure, usage and potential risks of enterprise data assets, improve the accuracy and efficiency of data governance, and promote digital transformation and intelligent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0085] In the attached picture:

[0086] Figure 1 is a basic flow chart of data exploration in an embodiment of the present invention;

[0087] Figure 2 A flow chart of a data exploration method of the present invention;

[0088] Figure 3The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0089] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and products consistent with some aspects of the present disclosure as detailed in the appended claims.

[0090] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0091] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0092] The embodiments of the present invention are described in further detail below.

[0093] The present invention provides a data exploration method. Figure 2 As shown, the following steps are included:

[0094] S1. Metadata basic information exploration: input database connection information and related parameters, establish database connection and obtain metadata information, scan core tables in the database, obtain field details, count index and constraint information, and store metadata; Metadata field security attribute exploration: identify sensitive fields, check field security measures, and detect field permission allocation; Metadata lineage relationship exploration: analyze the primary and foreign key relationships between tables, explore and analyze field-level dependencies, and build a metadata lineage map;

[0095] The database connection information in the input database connection information and related parameters is the necessary parameters provided by the user for connecting to the target database, including the database IP address, port number, user name, password and database type (MySQL);

[0096] The relevant parameters are the exploration scope (including the specific database name or table name) that the user needs to select in order to clarify the exploration target;

[0097] Methods for establishing a database connection and obtaining metadata information include:

[0098] Connect to the target relational database according to the provided database connection information; obtain the metadata information of the database through the system table or the database management interface; such as in this embodiment: Table 1: USER_INFO, field definition: Field 1: USER_ID, type definition: varchar(50); Field 2: CREATE_TIME, type definition: datetime; Field 3: STATUS, type definition: int;

[0099] Methods for scanning core tables in the database include:

[0100] Automatically identify user tables, system management tables, and space information tables, and match and filter target tables through table name patterns ("USER%", "SYS%", "SPATIAL%"); extract basic table information, such as in this embodiment: table name: ORDERS, table type: ordinary table, storage engine: InnoDB, number of rows: 5 million.

[0101] Methods for obtaining field details include:

[0102] For each table, scan the fields one by one and extract the definition information of the fields, which includes the field name, data type (varchar, int, decimal, etc.), field length, whether null values ​​are allowed, default value, and whether it is a primary key or a foreign key. For example, in this embodiment: table USER_INFO, field 1: USER_NAME, type definition: varchar(100), default value: NULL, whether null values ​​are allowed: yes, primary key flag: no;

[0103] Methods for collecting statistics on indexes and constraints include:

[0104] Extract the primary key index, unique index and common index information from the table definition, and count the coverage and quantity of each index; at the same time, check the constraint definition of the table, including whether the foreign key constraint is associated with a valid primary key field. The statistical information in this embodiment is as follows: table ORDERS, primary key index: ORDERS_PK, coverage field: ORDER_ID, common index: IDX_ORDER_DATE, coverage field: ORDER_DATE;

[0105] Methods for storing metadata include:

[0106] The information of the table and field is stored in a repository for managing metadata, and a metadata list is generated, including basic information of the table, field definitions, indexes and constraints, etc., to support subsequent exploration and analysis. For example, in this embodiment: a metadata table record is generated: table name: PRODUCTS, total number of fields: 10, primary key: PRODUCT_ID, foreign key: CATEGORY_ID.

[0107] The method for identifying sensitive fields includes: matching sensitive words ("ID number" and "bank card number") and field types (varchar, length greater than 20) by field names, and filtering fields that may contain sensitive information; for example, in this embodiment: table PAYMENTS, field: CARD_NUMBER, type: varchar(16), matching keyword: bank card number;

[0108] Methods for checking the security of a field include:

[0109] Check whether sensitive fields are encrypted and stored (check whether they are in ciphertext format), whether desensitization is used (partially displaying "****"), and whether access rights are set. For example, in this embodiment: table PAYMENTS, field: CARD_NUMBER, encryption status: yes, desensitization status: no, access rights: administrator group;

[0110] The method for detecting field permission allocation includes: extracting access rights of sensitive fields from permission configuration, and checking whether there is unreasonable allocation (such as ordinary user groups having write permissions); such as in this embodiment: table USER_INFO, field: PASSWORD, permission allocation: administrator (read and write), ordinary user (read), permission rationality: pass;

[0111] Methods for analyzing the primary and foreign key relationships between tables include:

[0112] According to the definition of primary key and foreign key, the association relationship between tables is extracted to build table-level blood relationship; for example, in this embodiment, table ORDERS (primary key: ORDER_ID) is associated with table ORDER_ITEMS (foreign key: ORDER_ID), relationship: one-to-many;

[0113] The methods for segment-level dependency exploration and analysis include:

[0114] Analyze the derived relationship and calculation logic between fields, and mark the source of the fields; for example, in this embodiment: the source field of field TOTAL_AMOUNT (table ORDERS) is QUANTITY (table ORDER_ITEMS) × UNIT_PRICE (table PRODUCTS);

[0115] Methods for constructing metadata lineage graphs include:

[0116] According to the dependency relationship between tables and fields, a visual bloodline map is generated. For example, in this embodiment, the table ORDERS is used as the root node, and the downstream tables include ORDER_ITEMS and PAYMENTS; the field-level relationship is displayed as the calculation of TOTAL_AMOUNT depending on multiple fields.

[0117] S2. Exploration of empty values, NULL values ​​and non-empty ratios: empty value statistics, NULL value statistics, non-empty ratio calculation; data volume and distribution exploration: no data statistics, data volume statistics, empty value ratio analysis; field attribute and distribution exploration: primary key length statistics, duplicate data exploration, value distribution exploration, value range analysis, dimension distribution analysis, value description generation; security exploration: security scanning and security policy recommendation;

[0118] The methods for null value statistics include:

[0119] Count the number of null values ​​(NULL values ​​and empty strings) in each field, and calculate the proportion of null values ​​in the field to help identify which fields have a large number of null values ​​and mark the fields that may affect data integrity; for example, in this embodiment: table USER_INFO, field EMAIL, number of null values: 5000, null value proportion: 30%.

[0120] The methods for counting NULL values ​​include:

[0121] Distinguish between explicit NULL values ​​and implicit null values ​​(such as the difference between an empty string and NULL) in a field, and calculate the total amount and proportion of explicit NULL values ​​and implicit null values ​​respectively, to ensure the accuracy of NULL value statistics, especially the assessment of data quality; for example, in this embodiment, the field ADDRESS has a NULL value proportion of 10% and an empty string proportion of 20%.

[0122] The methods for calculating the non-empty ratio include:

[0123] Calculate the non-null value ratio of each field, and evaluate the actual data utilization of the field by comparing the non-null value ratio; pay special attention to the non-null ratio of core business fields to ensure data availability; such as in this embodiment: field USER_ID, non-null ratio: 99%, field PHONE, non-null ratio: 80%.

[0124] Methods without data statistics include:

[0125] Check whether each field has no data at all (that is, all values ​​are NULL or empty strings), and mark these abnormal fields. For fields with no data at all, generate a warning prompt and list the table name and field; such as in this embodiment: field LAST_LOGIN_TIME, no data statistics: field values ​​are all empty.

[0126] The methods for data volume statistics include:

[0127] Count the total number of records in each table, analyze the distribution of records in each field, and help find tables or fields with too much or too little data; detect performance bottlenecks, storage capacity issues, and data missing. For example, in this embodiment: table ORDERS, total number of records: 5 million, field ORDER_DATE, number of records: 5 million.

[0128] The methods for analyzing the proportion of null values ​​include:

[0129] According to the null value ratio of the field, record the fields with high null value ratio and the fields that are critical to the business logic. When the null value ratio of these fields exceeds the preset threshold, generate a warning message. For example, in this embodiment, the null value ratio of the field ORDER_STATUS is 15%, which exceeds the preset threshold of 10% and is automatically marked as abnormal;

[0130] The methods for primary key length statistics include:

[0131] Check the length of all primary key fields and analyze whether the length of the primary key field affects the performance or storage efficiency of the database; for example, in this embodiment, the primary key length of the field USER_ID is varchar(50). Check whether the field length is suitable for the data scale and prompt for optimization.

[0132] Methods for iterative data exploration include:

[0133] Detect the number of duplicate records and duplicate values ​​of non-primary key fields in the table to ensure that there is no unnecessary redundant data in the table. By finding duplicate records, mark tables and fields containing a large amount of duplicate data; such as in this embodiment: table USER_INFO, field EMAIL, number of duplicate records: 2000, duplicate ratio: 10%.

[0134] Methods for numerical distribution exploration include:

[0135] Perform distribution analysis on all numeric fields (such as int and decimal), calculate the statistical information of the minimum, maximum, average, and standard deviation of the fields, and mark the extreme values ​​or outliers that do not meet expectations; for example, in this embodiment, the field SALES has a minimum value of -1000000 (outlier), a maximum value of 1000000, an average value of 500000, and a standard deviation of 200000, which is marked as an outlier.

[0136] Range analysis methods include:

[0137] Analyze the value range of enumeration types or limited value fields, and count the different values ​​of these fields. Through the value range, find abnormal values ​​or data outside the value range in the field; such as the field STATUS (value range: 1, 2, 3) in this embodiment, the data outside the value range: the field contains the value 4, which is an abnormal value.

[0138] Methods for dimensional distribution analysis include:

[0139] Analyze the value distribution of dimension fields (such as CITY, PRODUCT_CATEGORY, etc.) and mark fields with too concentrated or dispersed value distribution; for example, in this embodiment: the field CITY, the value range: 100 cities, the distribution is relatively uniform; the field PRODUCT_CATEGORY, the value range: 5 categories, the distribution is uneven, and some categories account for too high a proportion, and the system generates a warning.

[0140] The methods for generating value descriptions include:

[0141] According to the value range and frequency analysis of the field, a value description is generated for each field. For enumeration fields, the value range of the field and the main value proportions are listed to help users understand the field content. For example, in this embodiment: the field ORDER_STATUS, value description: [1: Pending, 2: Completed, 3: Canceled], value distribution: 40% Pending, 50% Completed, 10% Canceled.

[0142] Methods for security scanning and recommending security policies include:

[0143] According to the data exploration results, a security recommendation strategy is provided for sensitive fields, including whether encryption, desensitization, and access control are required; a security scan report is generated for each sensitive field, and security measures are recommended. For example, in this embodiment: field PASSWORD, encryption recommendation: yes, desensitization recommendation: no, access permission: only administrators can read and write.

[0144] S3. Data security analysis: sensitive data identification, security measures recommendations; data integrity and distribution analysis: data null value distribution analysis, data range distribution analysis; data relationship and distribution analysis: data relationship analysis, data distribution analysis, data volume ratio analysis, no data TOP analysis.

[0145] Methods for identifying sensitive data include:

[0146] Combine the business exploration results and sensitive field markings to analyze the distribution of sensitive data and evaluate the security of sensitive data; such as in this embodiment: field CREDIT_CARD_NUMBER, location: table PAYMENTS, data distribution: widely present in multiple tables, marked as high risk.

[0147] Recommended methods of safety measures include:

[0148] Generate targeted security optimization suggestions based on the distribution of sensitive data and existing security measures; for example, in this embodiment, it is recommended to add encryption processing to the field USER_PASSWORD and enable the desensitization strategy for the field CREDIT_CARD_NUMBER.

[0149] Methods for data null value distribution analysis include:

[0150] Statistics are collected on the distribution of null values ​​in all tables and fields, a null value heat map is generated, and fields with too high null values ​​are marked; for business-critical fields, the uneven distribution of null values ​​is marked as abnormal; for example, in this embodiment, the null value heat map of the field USER_EMAIL shows that null values ​​are concentrated in certain business areas.

[0151] Methods for data range distribution analysis include:

[0152] Analyze the concentration of the field value range, calculate the standard deviation, maximum value, minimum value, etc., and identify extreme values ​​and outliers. For example, in this embodiment, after analyzing the value distribution of the field SALES, it is found that it contains negative values ​​or abnormally large values, which are marked as data quality issues.

[0153] Methods for data relationship analysis include:

[0154] Combined with the metadata lineage diagram, analyze the logical relationship between tables, and mark potential redundant fields, conflicting data, or unnecessary dependencies between tables by tracing the dependencies between fields. For example, in this embodiment, the fields PRODUCT_ID (table ORDERS) and PRODUCT_ID (table PRODUCTS) have a redundant relationship.

[0155] Methods for data distribution analysis include:

[0156] Count the data volume ratio of each field in the table, identify tables or fields with large data volume, and analyze possible performance bottlenecks. For example, in this embodiment, the data volume of the PRODUCT_NAME field of the ORDERS table accounts for 80% of the total data volume, and it is necessary to evaluate whether it affects the query performance.

[0157] Methods for analyzing data volume ratio include:

[0158] Generate a field data volume ratio chart to identify fields with severely unbalanced data volume; for example, in the present embodiment, the data volume of the field ORDER_DATE in the table ORDERS accounts for 95% of the total data volume, while the field SHIPPING_DATE accounts for only 5%. The system will mark SHIPPING_DATE as a field with high potential for optimization.

[0159] The method of no-data TOP analysis includes: counting the number of fields without data and generating fields without data.

[0160] Figure 1 The basic flow of data exploration in this embodiment is shown.

[0161] The embodiment of the present invention further provides a data exploration system, which executes the data exploration method as described above, including:

[0162] Metadata technical information exploration module: used for metadata basic information exploration: input database connection information and related parameters, establish database connection and obtain metadata information, scan core tables in the database, obtain field details, count index and constraint information, and store metadata; metadata field security attribute exploration: identify sensitive fields, check field security measures, and detect field permission allocation; metadata lineage exploration: analyze primary and foreign key relationships between tables, explore and analyze field-level dependencies, and build metadata lineage maps;

[0163] Business information exploration module: used for empty value, NULL value and non-empty ratio exploration: empty value statistics, NULL value statistics, non-empty ratio calculation; data volume and distribution exploration: no data statistics, data volume statistics, empty value ratio analysis; field attribute and distribution exploration: primary key length statistics, duplicate data exploration, value distribution exploration, value range analysis, dimension distribution analysis, value description generation; security exploration: security scanning and security policy recommendation;

[0164] Probe and analysis module: used for data security analysis: sensitive data identification, security measures recommendations; data integrity and distribution analysis: data null value distribution analysis, data range distribution analysis; data relationship and distribution analysis: data relationship analysis, data distribution analysis, data volume ratio analysis, no data TOP analysis.

[0165] The data exploration method and system of this embodiment can detect problems such as null values, redundant information, and performance bottlenecks in the data in real time through intelligent automatic analysis and identification, and provide accurate support for subsequent data governance; the user only needs to input basic database connection information and related parameters to automatically complete the exploration task, which improves work efficiency and reduces manual intervention. The data exploration method has strong versatility and can adapt to different database environments and data scales to ensure the efficiency and accuracy of the exploration process; through this data exploration method, the structure, usage, and potential risks of enterprise data assets can be deeply understood, and the accuracy and efficiency of data governance can be improved.

[0166] An embodiment of the present invention further provides a computer device, Figure 3 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention; see the accompanying drawings Figure 3 As shown, the computer device includes: an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the data exploration method provided in the above embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 3 The example of connecting through bus is taken in the following.

[0167] The memory 22 is a readable and writable storage medium of a computing device, which can be used to store software programs, computer executable programs, such as program instructions corresponding to the data exploration method described in the embodiment of the present invention; the memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc.; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some instances, the memory 22 can further include a memory remotely arranged relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The input system 23 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the device; the output system 24 may include display devices such as display screens.

[0169] The processor 21 executes the software programs, instructions and modules stored in the memory 22 to execute various functional applications and data processing of the device, that is, to implement the above-mentioned data exploration method.

[0170] The computer device provided above can be used to execute the data exploration method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0171] The embodiment of the present invention also provides a storage medium containing computer executable instructions, which are used to execute the data exploration method provided in the above embodiment when executed by a computer processor. The storage medium is any of various types of memory devices or storage devices, and the storage medium includes: installation media, such as CD-ROM, floppy disk or tape system; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or a combination thereof; in addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet); the second computer system may provide program instructions to the first computer for execution. The storage medium includes two or more storage media that may reside in different locations (for example, in different computer systems connected via a network). The storage medium may store program instructions (for example, specifically implemented as a computer program) that can be executed by one or more processors.

[0172] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the data exploration method described in the above embodiment, and can also execute related operations in the data exploration method provided by any embodiment of the present invention.

[0173] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments, but it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data exploration method, characterized in that: The following steps are involved: S1. Metadata basic information exploration: input database connection information and related parameters, establish database connection and obtain metadata information, scan core tables in the database, obtain field details, count index and constraint information, and store metadata; Metadata field security attribute detection: identify sensitive fields, check field security measures, and detect field permission allocation; Metadata lineage relationship exploration: Analyze the primary and foreign key relationships between tables, explore and analyze field-level dependencies, and build a metadata lineage graph; S2. Exploration of empty values, NULL values ​​and non-empty ratios: empty value statistics, NULL value statistics, non-empty ratio calculation; data volume and distribution exploration: no data statistics, data volume statistics, empty value ratio analysis; field attribute and distribution exploration: primary key length statistics, duplicate data exploration, value distribution exploration, value range analysis, dimension distribution analysis, value description generation; security exploration: security scanning and security policy recommendation; S3. Data security analysis: sensitive data identification and security measures recommendations; Data integrity and distribution analysis: data null value distribution analysis, data value range distribution analysis; data relationship and distribution analysis: data relationship analysis, data distribution analysis, data volume proportion analysis, no data TOP analysis.

2. The data exploration method according to claim 1, characterized in that: The database connection information in the input database connection information and related parameters of step S1 is the necessary parameters provided by the user for connecting to the target database, including the database IP address, port number, user name, password and database type; The relevant parameters are the exploration ranges that need to be selected by the user in order to clarify the exploration target; The method of establishing a database connection and obtaining metadata information in step S1 includes: Connect to the target relational database according to the provided database connection information; obtain database metadata through system tables or database management interfaces; The method for scanning the core table in the database in step S1 includes: Automatically identify user tables, system management tables, and space information tables, filter target tables by table name pattern matching, and extract basic table information.

3. The data exploration method according to claim 1, characterized in that: The method for obtaining detailed field information in step S1 includes: For each table, scan the fields one by one and extract the field definition information, which includes the field name, data type, field length, whether null values ​​are allowed, default value, and whether it is a primary key or a foreign key; The method of statistical index and constraint information in step S1 includes: Extract primary key index, unique index and common index information from the table definition, count the coverage and quantity of each index; and check the constraint definition of the table at the same time; The method for storing metadata in step S1 includes: The information of tables and fields is stored in the repository for managing metadata, and a metadata list is generated, including the basic information of the table, field definitions, indexes, and constraints to support subsequent exploration and analysis.

4. The data exploration method according to claim 1, characterized in that: The method for identifying sensitive fields in step S1 includes: filtering fields that may contain sensitive information by matching sensitive words and field types with field names; The method of checking the safety measures of the field in step S1 includes: Check whether sensitive fields are stored in encrypted form, whether desensitization is used, and whether access permissions are set; The method for detecting field permission allocation in step S1 includes: extracting access permissions of sensitive fields from permission configuration and checking whether there is unreasonable allocation; The method for analyzing the primary and foreign key relationships between tables in step S1 includes: According to the definition of primary key and foreign key, extract the association between tables and build table-level blood relationship; The method for field-level dependency exploration and analysis in step S1 includes: Analyze the derivation relationship and calculation logic between fields, and mark the source of the fields; The method for constructing the metadata kinship map in step S1 includes: Generate a visual bloodline map based on the dependencies between tables and fields.

5. The data exploration method according to claim 1, characterized in that: The method for counting null values ​​in step S2 includes: Count the number of null values ​​in each field and calculate the percentage of null values ​​in the field to help identify which fields have a large number of null values ​​and mark the fields that may affect data integrity; The method for counting NULL values ​​in step S2 includes: Distinguish between explicit NULL values ​​and implicit NULL values ​​in the field, and calculate the total amount and proportion of explicit NULL values ​​and implicit NULL values ​​respectively; The method for calculating the non-empty ratio in step S2 includes: Calculate the proportion of non-null values ​​for each field and evaluate the actual data utilization of the field by comparing the proportion of non-null values; The method without data statistics in step S2 includes: Check each field to see if there is any data at all, and mark these abnormal fields. For fields with no data at all, generate a warning prompt and list the table name and field; The method for counting the amount of data in step S2 includes: Count the total number of records in each table, analyze the distribution of records in each field, and help find tables or fields with too large or too small data volumes; The method for analyzing the null value ratio in step S2 includes: Based on the null value ratio of the field, record the fields with high null value ratio and the fields that are critical to the business logic. When the null value ratio of these fields exceeds the preset threshold, a warning message is generated.

6. The data exploration method according to claim 1, characterized in that: The method for counting the primary key length in step S2 includes: Check the length of all primary key fields and analyze whether the length of the primary key field affects the performance or storage efficiency of the database; The method for repeated data exploration in step S2 includes: Detect the number of duplicate records and duplicate values ​​of non-primary key fields in the table to ensure that there is no unnecessary redundant data in the table. By finding duplicate records, mark the tables and fields containing a large amount of duplicate data; The method for detecting the numerical distribution of step S2 includes: Perform distribution analysis on all numeric fields, calculate the minimum, maximum, average, and standard deviation statistics of the fields, and mark extreme values ​​or outliers that do not meet expectations; The method for range analysis in step S2 includes: Analyze the value range of enumeration types or limited value fields, and count the different values ​​of these fields. Through the value range, find abnormal values ​​or data outside the value range in the field; The method for dimensional distribution analysis in step S2 includes: Analyze the value distribution of dimension fields and mark fields with values ​​that are too concentrated or too dispersed; The method for generating the value description of step S2 includes: Generate a value description for each field based on the value range and frequency analysis of the field; for enumeration fields, list the value range and main value proportion of the field to help users understand the field content; The method for security scanning and recommending security policies in step S2 includes: Based on the data exploration results, security recommendation strategies are provided for sensitive fields, including whether encryption and desensitization are required, and whether access control needs to be increased; a security scan report is generated for each sensitive field, and security measures are recommended.

7. The data exploration method according to claim 1, characterized in that: The method for identifying sensitive data in step S3 includes: Combine business exploration results and sensitive field tags to analyze the distribution of sensitive data and assess the security of sensitive data; The recommended approach for the safety measures in step S3 includes: Generate targeted security optimization suggestions based on the distribution of sensitive data and existing security measures; The method for analyzing the data null value distribution in step S3 includes: Count the null value distribution of all tables and fields, generate a null value heat map, and mark the fields with too high null values; for business-critical fields, mark the uneven distribution of null values ​​as abnormalities; The method for analyzing the data range distribution in step S3 includes: Analyze the concentration of field value ranges, calculate standard deviation, maximum value, minimum value, etc., and identify extreme values ​​and outliers; The method for data relationship analysis in step S3 includes: Combined with the metadata lineage diagram, analyze the logical relationship between tables, and mark potential redundant fields, conflicting data, or unnecessary dependencies between tables by tracing the dependencies between fields. The method for data distribution analysis in step S3 includes: Count the data volume ratio of each field in the table, identify tables or fields with large data volumes, and analyze possible performance bottlenecks; The method for analyzing the data volume ratio in step S3 includes: Generate a field data volume ratio chart to identify fields with severely unbalanced data volume; The method for the no-data TOP analysis in step S3 includes: counting the number of fields without data and generating fields without data.

8. A data exploration system, characterized in that: Executing the data exploration method according to any one of claims 1 to 7, comprising: Metadata technical information exploration module: used for metadata basic information exploration: input database connection information and related parameters, establish database connection and obtain metadata information, scan core tables in the database, obtain field details, count index and constraint information, and store metadata; metadata field security attribute exploration: identify sensitive fields, check field security measures, and detect field permission allocation; metadata lineage exploration: analyze primary and foreign key relationships between tables, explore and analyze field-level dependencies, and build metadata lineage maps; Business information exploration module: used for empty value, NULL value and non-empty ratio exploration: empty value statistics, NULL value statistics, non-empty ratio calculation; data volume and distribution exploration: no data statistics, data volume statistics, empty value ratio analysis; field attribute and distribution exploration: primary key length statistics, duplicate data exploration, value distribution exploration, value range analysis, dimension distribution analysis, value description generation; security exploration: security scanning and security policy recommendation; Exploration and analysis module: used for data security analysis: sensitive data identification, security measures recommendations; data integrity and distribution analysis: data null value distribution analysis, data range distribution analysis; data relationship and distribution analysis: data relationship analysis, data distribution analysis, data volume ratio analysis, no data TOP analysis.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the data exploration method according to any one of claims 1 to 7 are implemented.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data exploration method according to any one of claims 1 to 7 are implemented.

Images