Data asset management system and method

Abstract

The present application relates to the field of computer technology and discloses a data asset management system and method, which includes: a data asset import module that receives a data asset import request, obtains a data table or a batch of Excel files based on the request, and obtains data assets to be processed; a data asset quality monitoring module performs data verification on the data assets to be processed according to preset field quality monitoring rules and generates verification results; a data asset classification management module classifies the data assets to be processed that the verification results indicate have no problems from predefined business and technical dimensions based on preset user-defined classification names of different dimensions, and obtains multiple types of data assets; a data asset retrieval module receives a data retrieval request, performs an approximate search based on the data retrieval request, and obtains a target data asset; and queries the target data asset for detailed information and data records of the data asset requested by the data retrieval request and displays them. Therefore, the present application can improve data management efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data asset management system and method that can be used in a data security exchange and sharing system. Background Art

[0002] In the field of data asset management, enterprises face the challenge of managing massive and diverse data. To meet these challenges, enterprises generally need a system that can effectively manage and utilize data assets to support efficient data retrieval, analysis, and application, thereby helping enterprises extract valuable information from data and support decision-making.

[0003] In related technologies, existing data asset management systems provide basic data management functions, but often lack effective data quality monitoring mechanisms. For example, unreasonable data storage and archiving strategies make it difficult to track data owners. At the same time, there is a lack of effective data search and indexing mechanisms, making it time-consuming and labor-intensive to find specific data sets, thereby reducing data management efficiency. Summary of the Invention

[0004] The embodiments of this application provide a data asset management system. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key or important components, or delineate the scope of protection for these embodiments. Its sole purpose is to present some concepts in a simplified form, serving as a prelude to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a data asset management system, the system comprising:

[0006] Data asset import module, data asset quality monitoring module, data asset classification management module and data asset retrieval module; among them,

[0007] The data asset import module is used to receive data asset import requests for data table import or Excel batch import, obtain data tables or batch Excel files based on the data asset import request, and obtain the data assets to be processed;

[0008] The data asset quality monitoring module is used to perform data verification on the data assets to be processed according to the preset field quality monitoring rules and generate verification results. If there is a target field marked as triggering the preset field quality monitoring rules in the verification results, an early warning message for the target field is generated and sent to the client responsible for the target field.

[0009] The data asset classification management module is used to classify the data assets to be processed from predefined business and technical dimensions based on user-defined classification names in different dimensions when the target field that triggers the preset field quality monitoring rule does not exist in the verification results, thereby obtaining multiple types of data assets and storing them. Each type of data asset includes a classification tab and its data assets.

[0010] The data asset retrieval module is used to receive data retrieval requests, perform approximate searches on classification tabs and their data assets based on the data categories contained in the data retrieval requests, and obtain target data assets; from the target data assets, query the detailed information and data records of the data assets corresponding to the data fields contained in the data retrieval request, and display the detailed information and data records.

[0011] Optionally, the data assets to be processed include multiple fields and their field values; the preset field quality monitoring rules include a basic verification model and a deep verification model, and the basic verification model is used to verify the legitimacy of the field value of each field;

[0012] According to the preset field quality monitoring rules, data verification is performed on the data assets to be processed, and verification results are generated, including:

[0013] Input each field and its field value into the basic verification model to obtain the first check result of each field;

[0014] Through the deep verification model, normal distribution verification and association rule mining are performed on each field and its field value to obtain the second check result of each field;

[0015] The first check result and the second check result of each field are used as the verification result.

[0016] Optionally, each field and its value is input into the basic validation model to obtain the first check result for each field, including:

[0017] Check whether the field value of each field is empty through the basic validation model;

[0018] Check whether the field value of each field conforms to the preset format through the basic validation model;

[0019] Use the basic validation model to check whether the field value of each field is within the preset value range;

[0020] Check whether each field is unique through the basic validation model;

[0021] Mark the fields that are empty, do not conform to the preset format, are not within the preset numerical range, and are not unique, and obtain the target fields that trigger the preset field quality monitoring rules. Mark the fields that are not empty, conform to the preset format, are within the preset numerical range, and are unique, and obtain the target fields that do not trigger the preset field quality monitoring rules as the first inspection result of each field.

[0022] Optionally, normal distribution verification and association rule mining are performed on each field and its field value to obtain a second check result for each field, including:

[0023] Identify non-numeric type fields among the multiple fields, and convert the field values ​​of the existing non-numeric type fields into numeric types using natural language processing technology to obtain field values ​​of multiple second fields;

[0024] Calculate the mean and standard deviation of all fields based on the field value of each first field, where each first field is each field of the plurality of second fields and the numerical type fields present in the plurality of fields;

[0025] Based on the mean and standard deviation of all fields, the field value of each first field is converted to the Z score of the standard normal distribution to obtain the Z score of all fields;

[0026] Determine whether the Z scores of all fields are within the preset standard normal distribution range;

[0027] Marking the fields that are not within the preset standard normal distribution interval to obtain target fields that trigger the preset field quality monitoring rule and marking the fields that are within the preset standard normal distribution interval to obtain target fields that do not trigger the preset field quality monitoring rule as the normal distribution result of each field;

[0028] Perform association rule mining on each field and its field value to obtain the association relationship results of each field;

[0029] The normal distribution result and the association relationship result of each field are used as the second inspection result of each field.

[0030] Optionally, the average value of all fields is the ratio of the sum of the field values ​​of each first field to the number of first fields;

[0031] The standard deviation of all fields is calculated as:

[0032]

[0033] in, is the standard deviation of all fields, n is the number of the first field, For the The first field, is the average value of all fields;

[0034] The formula for calculating the Z score for all fields is:

[0035]

[0036] in, is the Z score of all fields, and X is the field value of each first field.

[0037] Optionally, perform association rule mining on each field and its field value to obtain the association relationship results of each field, including:

[0038] Simulate the system business processing transactions corresponding to each field and its field value to convert each field into a transaction set;

[0039] Calculate the proportion of the number of times each transaction appears in the transaction set;

[0040] Classify transactions whose proportion is greater than the preset proportion to obtain frequent item sets;

[0041] Apriori algorithm is used to analyze frequent item sets and obtain multiple association rules;

[0042] Calculate the confidence of each rule;

[0043] When the confidence of each rule is greater than a preset confidence threshold, it is determined that each field and its field value have an associated relationship; or when the confidence of each rule is less than or equal to the preset confidence threshold, it is determined that each field and its field value do not have an associated relationship;

[0044] Mark each field and its field value that have an association relationship, and obtain the target field that has not triggered the preset field quality monitoring rules and is marked, and mark each field and its field value that do not have an association relationship, and obtain the target field that has triggered the preset field quality monitoring rules and is marked, as the association relationship result of each field.

[0045] Optionally, predefined business dimensions include customer type and product category, and predefined technical dimensions include data format and data source;

[0046] Based on user-defined classification names in different preset dimensions, the data assets to be processed are classified from predefined business and technical dimensions to obtain multiple types of data assets, including:

[0047] Determine the business classification of each piece of data in the data assets to be processed based on customer type and product category;

[0048] Determine the technical classification of each piece of data in the data assets to be processed based on the data format and data source;

[0049] Create multiple category directories based on user-defined category names of different preset dimensions;

[0050] Identify the target category directory to which the business and technical categories of each piece of data belong from multiple category directories;

[0051] Each piece of data is divided into the target category directory to which the business classification and technical classification of each piece of data belong, and multiple types of data assets are obtained.

[0052] Optionally, determine the business classification of each piece of data in the data assets to be processed based on customer type and product category, including:

[0053] Extract features related to customer type and product category from each piece of data in the data assets to be processed, and obtain customer type features and product category features;

[0054] Use one-hot encoding or label encoding algorithms to convert customer type features and product category features into numerical feature vectors to obtain business dimension feature vectors.

[0055] Input the business dimension feature vector into the pre-trained business classification model and output the business classification of each data in the data asset to be processed;

[0056] Follow these steps to generate a pre-trained business classification model, including:

[0057] Use random forest algorithm to create business classification model;

[0058] Acquire historical data assets;

[0059] Extract features related to customer type and product category from each piece of historical data in the historical data assets, and obtain historical customer type features and historical product category features corresponding to each piece of historical data;

[0060] For each piece of historical data, the historical customer type characteristics and historical product category characteristics are marked with business classification data labels to obtain the first model training sample;

[0061] Input the first model training sample into the business classification model and output the first model loss value;

[0062] When the loss value of the first model reaches the minimum, a pre-trained business classification model is generated.

[0063] Optionally, determine the technical classification of each piece of data in the data asset to be processed based on the data format and source, including:

[0064] Extract features related to data format and data source from each piece of data in the data asset to be processed, and obtain data format features and data source features;

[0065] Use one-hot encoding or label encoding algorithms to convert data format features and data source features into numerical feature vectors to obtain technical dimension feature vectors;

[0066] Input the technical dimension feature vector into the pre-trained technical classification model and output the technical classification of each data in the data asset to be processed;

[0067] Follow these steps to generate a pre-trained technology classification model, including:

[0068] Use random forest algorithm to create business technology models;

[0069] Acquire historical data assets;

[0070] Extract features related to data format and data source from each piece of historical data in the historical data asset, and obtain the historical data format feature and historical data source feature corresponding to each piece of historical data;

[0071] Based on the historical data format characteristics and historical data source characteristics corresponding to each piece of historical data, a technical classification data label is annotated to obtain a second model training sample;

[0072] Input the second model training sample into the technical classification model and output the second model loss value;

[0073] When the loss value of the second model reaches the minimum, a pre-trained technical classification model is generated.

[0074] In a second aspect, a data asset management method is provided, the method comprising:

[0075] The data asset import module receives a data asset import request for data table import or Excel batch import, obtains the data table or Excel batch file based on the data asset import request, and obtains the data assets to be processed;

[0076] The data asset quality monitoring module verifies the data assets to be processed based on the preset field quality monitoring rules and generates verification results. If a target field is marked as triggering the preset field quality monitoring rules in the verification results, an early warning message for the target field is generated and sent to the client responsible for the target field.

[0077] When the target field that triggers the preset field quality monitoring rule is not found in the verification results, the data asset classification management module classifies the pending data assets from predefined business and technical dimensions based on the user-defined classification names of different dimensions, obtains multiple categories of data assets, and stores the multiple categories of data assets. Each category of data assets includes a classification tab and its data assets.

[0078] The data asset retrieval module receives a data retrieval request, performs an approximate search on the classification tab and its data assets based on the data category contained in the data retrieval request, and obtains the target data asset; from the target data asset, it queries the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and displays the detailed information and data records.

[0079] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the data quality monitoring capability by implementing preset field quality monitoring rules. It can not only promptly detect problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send early warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on customized classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0080] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0082] Figure 1 This is a schematic diagram of the system structure of a data asset management system provided by an embodiment of the present application;

[0083] Figure 2 This is a data representation of the first check result of each field provided by an embodiment of the present application;

[0084] Figure 3 This is a data representation of the second check result of each field provided by an embodiment of the present application;

[0085] Figure 4 This is a flow chart of a data asset management method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The following description and the drawings sufficiently illustrate specific embodiments of the application to enable those skilled in the art to practice them.

[0087] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0088] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0089] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0090] The present application provides a data asset management system and method to solve the problems existing in the above-mentioned related technical problems. In the embodiment of the present application, on the one hand, the data asset quality monitoring module can significantly improve the monitoring ability of data quality by implementing preset field quality monitoring rules. It can not only timely discover problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send early warning information to the relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on customized classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency. The following exemplary embodiments are used for detailed description.

[0091] See Figure 1 , Figure 1 1 is a schematic diagram of the system structure of a data asset management system provided by an embodiment of the present application, the system comprising: a data asset import module, a data asset quality monitoring module, a data asset classification management module, and a data asset retrieval module; wherein the data asset import module is used to receive a data asset import request for data table import or Excel batch import, obtain a data table or batch Excel file based on the data asset import request, and obtain the data asset to be processed; the data asset quality monitoring module is used to perform data verification on the data asset to be processed according to preset field quality monitoring rules and generate verification results; when there is a target field marked as triggering the preset field quality monitoring rule in the verification result, an early warning message of the target field is generated and sent to the client responsible for the target field; The data asset classification management module is used to classify the data assets to be processed from predefined business dimensions and technical dimensions based on the user-defined classification names of preset different dimensions when there is no target field marked by triggering the preset field quality monitoring rules in the verification results, so as to obtain multiple types of data assets and store multiple types of data assets, wherein each type of data asset includes a classification tab and its data assets; the data asset retrieval module is used to receive data retrieval requests, perform approximate retrieval of the classification tab and its data assets according to the data category contained in the data retrieval request, and obtain the target data asset; from the target data asset, query the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and display the detailed information and data records.

[0092] Data asset import requests refer to user-issued instructions requesting the system to import new data assets. These requests can be for single tables or batches of Excel files. Pending data assets refer to data that has been received by the system but has not yet been processed (e.g., validation or classification). Pre-defined field quality monitoring rules are a set of rules predefined by the system that are used to check the quality of fields in data assets, such as non-null values, formatting accuracy, and range validity. Verification results are the output of the data verification process, indicating whether the data complies with the quality monitoring rules and the specific fields that do not. Target fields refer to fields detected as violating quality monitoring rules during data verification. Warning messages are notification messages generated by the system when data verification discovers issues, alerting relevant stakeholders to take action. The responsible client is a user interface authorized to access and address data issues. It is a component of the data asset management system that allows responsible stakeholders to receive warning messages and take appropriate action. User-defined category names are category labels or names that users define in the system based on their business needs. Business dimensions typically involve business logic, such as customer type and product category. Technical dimensions involve technical attributes of data, such as data format and data source. Multi-category data assets refer to a collection of data assets of multiple categories that are obtained by classifying data assets from business and technical dimensions based on user-defined classification names. The classification tab is the part of the user interface used to distinguish different categories of data assets, similar to folders in a file directory. A data retrieval request is a query instruction issued by the user, requiring the system to find data assets of a specific category or field. Approximate retrieval allows users to use fuzzy query conditions to retrieve data, and even if the query conditions are not completely accurate, similar results can be found. Target data assets refer to data assets that meet the conditions and are retrieved by the system based on the user's data retrieval request. Detailed information and data records refer to the specific content of data assets, including the values ​​of data fields and other related metadata.

[0093] For example, imagine a large electronics retailer looking to establish a data management system to track and analyze sales of various products. The system needs to be able to process large volumes of sales transaction data, categorize the data by customer type and product category, and ensure data quality. The point-of-sale system generates a transaction record after each transaction and sends a data asset import request to the data management system.

[0094] The system receives data table import requests and retrieves real-time sales data tables or batch imports daily sales summary Excel files, creating pending data assets. The system verifies sales data based on preset field quality monitoring rules (such as transaction date not null, transaction amount greater than zero, and product number validity). After generating the verification results, the system checks whether any target fields violate the quality rules. If any issues are found in the verification results, the system automatically generates an alert and sends it to the person responsible for data quality via email or system notification. The person responsible can log in to the system, view detailed information about the problematic data, and take corrective measures. For data assets that pass verification, the system categorizes them according to preset dimensions (such as customer type: individual, enterprise; product category: computer, mobile phone, accessories).

[0095] The classified data assets are stored in corresponding classification directories, such as different tables in a database or different folders in a file system. Each type of data asset includes a classification tab (such as the "Personal Computer User" tab) and its data assets.

[0096] A user needs to analyze sales for a specific product category and submits a data retrieval request, specifying the product category as "mobile phones." Upon receiving the request, the data asset retrieval module performs an approximate search of the category tab and its data assets, finding all data assets related to "mobile phones." The system then queries the data fields included in the data retrieval request, such as sales volume and sales volume, and displays detailed information and data records for these data assets. The analysis results are then presented in the system as charts or reports for further analysis.

[0097] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the data quality monitoring capability by implementing preset field quality monitoring rules. It can not only promptly detect problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send early warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on customized classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0098] Among them, the data assets to be processed include multiple fields and their field values; the preset field quality monitoring rules include a basic verification model and a deep verification model, and the basic verification model is used to verify the legitimacy of the field value of each field.

[0099] In some embodiments of the present application, data verification is performed on the data assets to be processed according to preset field quality monitoring rules, and the specific process of generating verification results includes: inputting each field and its field value into the basic verification model to obtain the first inspection result of each field; through the deep verification model, normal distribution verification and association rule mining are performed on each field and its field value to obtain the second inspection result of each field; the first inspection result and the second inspection result of each field are used as the verification result.

[0100] A field refers to an attribute or data point in a data record, such as a customer name or transaction amount. The basic validation model is a pre-defined model used to perform basic data validation, such as checking basic data quality rules like data format, type, non-nullability, and range. A field value refers to the specific data in a field. For example, in the "Customer Age" field, the specific numeric value (such as 30) is the field value. The first check result is the result of the basic validation model's preliminary check of the field and its value. The deep validation model is used to perform more in-depth data quality checks, such as normal distribution verification and association rule mining. Normal distribution verification is a statistical method used to verify whether data conforms to a normal distribution and is commonly used to verify numerical data. Association rule mining analyzes the correlations between data items to uncover hidden patterns or rules in the data and is often used to discover interesting relationships between data. The second check result is the result of the deep validation model's in-depth check of the field and its value. The verification result is a combination of basic and deep validation.

[0101] In this embodiment of the present application, by inputting each field and its field value into the basic verification model, the system can quickly identify basic data issues, such as missing values ​​or format errors, and obtain a first verification result. Subsequently, a deeper verification model is used to conduct a deeper analysis of the data, including verifying whether the data follows the expected statistical distribution and exploring potential association rules between the data, to obtain a second verification result. Ultimately, combining the inspection results of these two levels can obtain the final verification result, ensuring the accuracy of subsequent data analysis and business decisions.

[0102] In some embodiments of the present application, each field and its field value are input into the basic verification model, and the specific process of obtaining the first inspection result of each field includes: checking whether the field value of each field is an empty value through the basic verification model; checking whether the field value of each field conforms to the preset format through the basic verification model; checking whether the field value of each field is within the preset numerical range through the basic verification model; checking whether each field is unique through the basic verification model; marking the fields as empty values, not conforming to the preset format, not within the preset numerical range, and not unique, and obtaining the target fields that trigger the preset field quality monitoring rules; marking the fields that are not empty values, conforming to the preset format, within the preset numerical range, and unique, and obtaining the target fields that do not trigger the preset field quality monitoring rules as the first inspection result of each field.

[0103] In some embodiments, the first check result of each field is, for example, Figure 2 As shown in the following figure, the result status after basic verification is displayed as "Passed" or "Failed". If the verification status is "Failed", the specific error message will be provided here.

[0104] In some embodiments of the present application, each field and its field value are subjected to normal distribution verification and association rule mining, and the specific process of obtaining the second inspection result of each field includes: identifying non-numeric type fields existing in multiple fields, and converting the field values ​​of the existing non-numeric type fields into numeric types using natural language processing technology to obtain field values ​​of multiple second fields; according to the field value of each first field, calculating the mean and standard deviation of all fields, each first field is multiple second fields and each field in the numeric type fields existing in the multiple fields; based on the mean and standard deviation of all fields, converting the field value of each first field into a standard numeric type; Normally distribute the Z score to obtain the Z score of all fields; determine whether the Z score of all fields is within the preset standard normal distribution interval; mark the fields that are not within the preset standard normal distribution interval to obtain the target fields that trigger the preset field quality monitoring rules and are marked, and mark the fields that are within the preset standard normal distribution interval to obtain the target fields that do not trigger the preset field quality monitoring rules and are marked, as the normal distribution results of each field; perform association rule mining on each field and its field value to obtain the association relationship results of each field; use the normal distribution results and association relationship results of each field as the second inspection results of each field.

[0105] Specifically, the average value of all fields is the ratio of the sum of the field values ​​of each first field to the number of first fields; wherein, the calculation formula for the standard deviation of all fields is:

[0106]

[0107] in, is the standard deviation of all fields, n is the number of the first field, For the The first field, is the average value of all fields; the calculation formula for the Z score of all fields is:

[0108]

[0109] in, is the Z score of all fields, and X is the field value of each first field.

[0110] A non-numeric field refers to a data field containing non-numeric data, such as text or dates. Natural language processing (NLP) is a technology used to process and analyze text data, including semantic parsing, sentiment analysis, and language detection. The second field value refers to the numeric field value converted using NLP. The first field refers to converted non-numeric fields and fields that were originally numeric.

[0111] In some embodiments, the second check result of each field is, for example, Figure 3 As shown, it includes different data fields, such as product ID, sales quantity, sales amount, customer evaluation, Z score, normal distribution verification result, association result and verification status.

[0112] In this embodiment of the present application, association rule mining is performed on each field to discover potential relationships between fields. Ultimately, the results of normal distribution analysis and association rule mining are combined to form a second check result for each field. This provides a more comprehensive perspective for data quality management, helping to identify and address anomalies and inconsistencies in the data.

[0113] In some embodiments of the present application, each field and its field value are subjected to association rule mining to obtain the specific process of the association relationship result of each field, including: simulating the system business processing transactions corresponding to each field and its field value to convert each field into a transaction set; calculating the proportion of the number of times each transaction occurs in the transaction set; classifying the transactions whose proportion is greater than the preset proportion to obtain a frequent item set; using the Apriori algorithm to analyze the frequent item set to obtain multiple association rules; calculating the confidence of each rule; when the confidence of each rule is greater than the preset confidence threshold, determining that each field and its field value have an association relationship; or when the confidence of each rule is less than or equal to the preset confidence threshold, determining that each field and its field value do not have an association relationship; marking each field and its field value with an association relationship to obtain the target field that has been marked for not triggering the preset field quality monitoring rule, and marking each field and its field value without an association relationship to obtain the target field that has been marked for triggering the preset field quality monitoring rule as the association relationship result of each field.

[0114] Among them, the predefined business dimensions include customer type and product category, and the predefined technical dimensions include data format and data source.

[0115] In some embodiments of the present application, based on preset user-defined classification names of different dimensions, the data assets to be processed are classified from predefined business dimensions and technical dimensions to obtain multiple categories of data assets. The specific process includes: determining the business classification of each data in the data assets to be processed according to the customer type and product category; determining the technical classification of each data in the data assets to be processed according to the data format and data source; creating multiple category directories based on preset user-defined classification names of different dimensions; identifying the target category directory to which the business classification and technical classification of each data belongs from the multiple category directories; and dividing each data into the target category directory to which the business classification and technical classification of each data belongs, to obtain multiple categories of data assets.

[0116] Among them, customer type refers to the classification of customers based on certain characteristics of the customers, such as purchasing habits, preferences, and values. Product category refers to the classification of products based on their characteristics, such as purpose, function, price range, etc. Category directory refers to the storage unit created for different categories or the directory structure in the database, which is used to organize and store classified data. Target category directory refers to the specific category directory to which each piece of data is assigned based on its business classification and technical classification attributes. Multi-category data assets refer to data collections organized according to different business classifications and technical classifications.

[0117] In the embodiments of the present application, by determining the business classification of each piece of data based on customer type and product category, and determining the technical classification of each piece of data based on data format and source, the data management system can organize and retrieve data more accurately. Creating multiple category directories and identifying the target category directory to which each piece of data belongs further improves the flexibility and efficiency of data management. Dividing data into corresponding business and technical classification directories not only makes data assets more organized, but also enables enterprises to access and analyze specific types of data more quickly, thereby accelerating decision-making and enhancing data-driven business insights.

[0118] In some embodiments of the present application, the specific process of determining the business classification of each piece of data in the data assets to be processed based on the customer type and product category includes: extracting features related to the customer type and product category from each piece of data in the data assets to be processed, respectively, to obtain customer type features and product category features; using a one-hot encoding or label encoding algorithm to convert the customer type features and product category features into a numerical feature vector to obtain a business dimension feature vector; inputting the business dimension feature vector into a pre-trained business classification model, and outputting the business classification of each piece of data in the data assets to be processed.

[0119] Specifically, a pre-trained business classification model is generated according to the following steps, including: using a random forest algorithm to create a business classification model; obtaining historical data assets; extracting features related to customer type and product category from each historical data in the historical data assets, and obtaining historical customer type features and historical product category features corresponding to each historical data; marking business classification data labels for the historical customer type features and historical product category features corresponding to each historical data, and obtaining a first model training sample; inputting the first model training sample into the business classification model, and outputting a first model loss value; when the first model loss value reaches the minimum, generating a pre-trained business classification model.

[0120] In some embodiments of the present application, the specific process of determining the technical classification of each piece of data in the data assets to be processed based on the data format and data source includes: extracting features related to the data format and data source from each piece of data in the data assets to be processed, respectively, to obtain data format features and data source features; using a one-hot encoding or label encoding algorithm to convert the data format features and data source features into a numerical feature vector to obtain a technical dimension feature vector; inputting the technical dimension feature vector into a pre-trained technical classification model, and outputting the technical classification of each piece of data in the data assets to be processed.

[0121] Specifically, a pre-trained technology classification model is generated according to the following steps, which specifically include: using a random forest algorithm to create a business technology model; obtaining historical data assets; extracting features related to the data format and data source from each historical data in the historical data assets, and obtaining the historical data format features and historical data source features corresponding to each historical data; marking the historical data format features and historical data source features corresponding to each historical data with technology classification data labels to obtain a second model training sample; inputting the second model training sample into the technology classification model, and outputting the second model loss value; when the second model loss value reaches the minimum, generating a pre-trained technology classification model.

[0122] In an embodiment of the present application, the business classification and technical classification of each piece of data can be predicted through a pre-trained business classification model and a pre-trained technical classification model, and the automated prediction method improves the classification efficiency.

[0123] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the data quality monitoring capability by implementing preset field quality monitoring rules. It can not only promptly detect problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send early warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on customized classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0124] See Figure 4 , provides a flow chart of a data asset management method according to an embodiment of the present application. Figure 4 As shown, the detection method of the embodiment of the present application may include the following steps:

[0125] S101: The data asset import module receives a data asset import request for data table import or Excel batch import, obtains the data table or Excel batch file based on the data asset import request, and obtains the data assets to be processed;

[0126] S102: The data asset quality monitoring module performs data verification on the data assets to be processed according to the preset field quality monitoring rules and generates verification results. If a target field is marked as triggering the preset field quality monitoring rules in the verification results, an early warning message for the target field is generated and sent to the client responsible for the target field.

[0127] S103: When the target field that triggers the preset field quality monitoring rule is not found in the verification result, the data asset classification management module classifies the pending data assets from predefined business and technical dimensions based on the preset user-defined classification names of different dimensions, obtains multiple categories of data assets, and stores the multiple categories of data assets, wherein each category of data assets includes a classification tab and its data assets;

[0128] S104, the data asset retrieval module receives the data retrieval request, performs an approximate search on the classification tab and its data assets based on the data category contained in the data retrieval request, and obtains the target data asset; from the target data asset, queries the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and displays the detailed information and data records.

[0129] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the data quality monitoring capability by implementing preset field quality monitoring rules. It can not only promptly detect problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send early warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on customized classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0130] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the data asset management methods provided by the above-mentioned various method embodiments.

[0131] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data asset management methods of the above-mentioned various method embodiments.

[0132] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The data asset management program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium for the data asset management program can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0133] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

[0134] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The data asset management program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The data asset management storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0135] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data asset management system, characterized in that: The system comprises: Data asset import module, data asset quality monitoring module, data asset classification management module and data asset retrieval module; among them, The data asset import module is used to receive a data asset import request for data table import or Excel batch import, and obtain the data table or batch Excel file based on the data asset import request to obtain the data assets to be processed; The data asset quality monitoring module is configured to perform data verification on the data assets to be processed according to preset field quality monitoring rules and generate verification results; when a target field is marked as triggering the preset field quality monitoring rules in the verification results, generate warning information for the target field and send it to the client responsible for the target field; The data assets to be processed include multiple fields and their field values; the preset field quality monitoring rules include a basic verification model and a deep verification model, and the basic verification model is used to verify the legitimacy of the field value of each field; The step of performing data verification on the data asset to be processed according to the preset field quality monitoring rules and generating a verification result includes: Input each field and its field value into the basic verification model to obtain a first check result for each field; By using the deep verification model, normal distribution verification and association rule mining are performed on each field and its field value to obtain a second inspection result for each field; wherein association rule mining is performed on each field to discover potential associations between fields; combining the results of normal distribution analysis and association rule mining to form the second inspection result for each field helps to identify and handle anomalies and inconsistencies in the data; Perform normal distribution verification and association rule mining on each field and its field value to obtain a second check result for each field, including: Identifying non-numeric fields among the multiple fields, and converting field values ​​of the non-numeric fields into numeric values ​​using natural language processing technology to obtain field values ​​of multiple second fields; Calculate the average value and standard deviation of all fields according to the field value of each first field, wherein each first field is each field of the plurality of second fields and the numerical type fields in the plurality of fields; Based on the mean and standard deviation of all the fields, converting the field value of each first field into a Z score of a standard normal distribution to obtain Z scores of all the fields; Determine whether the Z scores of all the fields are within a preset standard normal distribution interval; Marking fields that are not within a preset standard normal distribution interval to obtain target fields that trigger the preset field quality monitoring rule and marking fields that are within a preset standard normal distribution interval to obtain target fields that do not trigger the preset field quality monitoring rule as normal distribution results for each field; Performing association rule mining on each field and its field value to obtain an association relationship result of each field; using the normal distribution result of each field and the association relationship result as the second inspection result of each field; Using the first check result and the second check result of each field as the verification result; The data asset classification management module is configured to classify the data assets to be processed from predefined business and technical dimensions based on preset user-defined classification names of different dimensions when no target field that triggers the preset field quality monitoring rule is marked in the verification result, thereby obtaining multiple categories of data assets and storing the multiple categories of data assets, wherein each category of data assets includes a classification tab and its data assets; Among them, the predefined business dimensions include customer type and product category, and the predefined technical dimensions include data format and data source; Based on the user-defined classification names of different preset dimensions, the data assets to be processed are classified from predefined business dimensions and technical dimensions to obtain multiple types of data assets, including: Determine the business classification of each piece of data in the data assets to be processed based on the customer type and product category; Determine the technical classification of each piece of data in the data asset to be processed based on the data format and data source; Create multiple category directories based on user-defined category names of different preset dimensions; Identifying, from the plurality of category directories, a target category directory to which the business and technical categories of each piece of data belong; Classify each piece of data into a target category directory to which the business classification and technical classification of each piece of data belongs, to obtain multiple categories of data assets; The data asset retrieval module is used to receive a data retrieval request, perform an approximate search on the classification tab and its data assets based on the data category contained in the data retrieval request, and obtain the target data asset; from the target data asset, query the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and display the detailed information and data records.

2. The system according to claim 1, wherein: Each field and its field value are input into the basic verification model to obtain a first check result for each field, including: Check whether the field value of each field is null through the basic verification model; Check whether the field value of each field conforms to the preset format through the basic verification model; Checking whether the field value of each field is within a preset numerical range through the basic verification model; Check whether each field is unique through the basic validation model; Mark the fields that are empty, do not conform to the preset format, are not within the preset numerical range, and are not unique, and obtain the target fields that trigger the preset field quality monitoring rules. Mark the fields that are not empty, conform to the preset format, are within the preset numerical range, and are unique, and obtain the target fields that do not trigger the preset field quality monitoring rules as the first inspection result of each field.

3. The system according to claim 1, wherein: The average value of all fields is the ratio between the sum of the field values ​​of each first field and the number of first fields; wherein, The formula for calculating the standard deviation of all fields is: Where σ is the standard deviation of all fields, n is the number of first fields, x i is the first field of the i-th column, μ is the average value of all fields; The formula for calculating the Z score of all the fields is: Wherein, Z is the Z score of all the fields, and X is the field value of each first field.

4. The system according to claim 1, wherein: The association rule mining of each field and its field value to obtain the association relationship result of each field includes: Simulating system business processing transactions corresponding to each field and its field value to convert each field into a transaction set; Calculate the proportion of the number of occurrences of each transaction in the transaction set; Classify transactions whose proportion is greater than the preset proportion to obtain frequent item sets; Apriori algorithm is used to analyze the frequent itemsets to obtain multiple association rules; Calculate the confidence of each rule; When the confidence of each rule is greater than a preset confidence threshold, it is determined that each field and its field value have an associated relationship; or when the confidence of each rule is less than or equal to the preset confidence threshold, it is determined that each field and its field value do not have an associated relationship; Mark each field and its field value for which there is an association relationship, and obtain the target field that is marked without triggering the preset field quality monitoring rule, and mark each field and its field value for which there is no association relationship, and obtain the target field that is marked when triggering the preset field quality monitoring rule, as the association relationship result of each field.

5. The system according to claim 1, wherein: Determining the business classification of each piece of data in the data assets to be processed based on the customer type and product category includes: Extracting features related to the customer type and product category from each piece of data in the data asset to be processed, to obtain customer type features and product category features; Use a one-hot encoding or label encoding algorithm to convert the customer type feature and product category feature into a numerical feature vector to obtain a business dimension feature vector; Input the business dimension feature vector into a pre-trained business classification model, and output the business classification of each piece of data in the data asset to be processed; wherein, Follow these steps to generate a pre-trained business classification model, including: Use random forest algorithm to create business classification model; Acquire historical data assets; Extracting features related to the customer type and product category from each piece of historical data in the historical data asset to obtain historical customer type features and historical product category features corresponding to each piece of historical data; Marking business classification data labels for historical customer type features and historical product category features corresponding to each piece of historical data to obtain a first model training sample; Inputting the first model training sample into the business classification model, and outputting a first model loss value; When the loss value of the first model reaches the minimum, a pre-trained business classification model is generated.

6. The system according to claim 1, wherein: Determining the technical classification of each piece of data in the data asset to be processed based on the data format and data source includes: Extracting features related to the data format and data source from each piece of data in the data asset to be processed, to obtain data format features and data source features; Using a one-hot encoding or label encoding algorithm to convert the data format features and data source features into a numerical feature vector to obtain a technical dimension feature vector; Input the technical dimension feature vector into a pre-trained technical classification model, and output the technical classification of each piece of data in the data asset to be processed; wherein, Follow these steps to generate a pre-trained technology classification model, including: Use random forest algorithm to create business technology models; Acquire historical data assets; Extracting features related to the data format and data source from each piece of historical data in the historical data asset to obtain a historical data format feature and a historical data source feature corresponding to each piece of historical data; According to the historical data format characteristics and historical data source characteristics corresponding to each piece of historical data, a technical classification data label is marked to obtain a second model training sample; Inputting the second model training sample into the technical classification model and outputting a second model loss value; When the loss value of the second model reaches the minimum, a pre-trained technical classification model is generated.

7. A data asset management method implemented using the method according to any one of claims 1 to 6, characterized in that: The method comprises: The data asset import module receives a data asset import request for data table import or Excel batch import, obtains the data table or Excel batch file based on the data asset import request, and obtains the data assets to be processed; The data asset quality monitoring module performs data verification on the data assets to be processed according to the preset field quality monitoring rules and generates a verification result. If a target field is marked as triggering the preset field quality monitoring rule in the verification result, an early warning message for the target field is generated and sent to the client responsible for the target field. When the target field that triggers the preset field quality monitoring rule is not found in the verification result, the data asset classification management module classifies the data assets to be processed from predefined business dimensions and technical dimensions based on the user-defined classification names of different dimensions to obtain multiple categories of data assets, and stores the multiple categories of data assets, wherein each category of data assets includes a classification tab and its data assets; The data asset retrieval module receives a data retrieval request, performs an approximate search on the classification tab and its data assets based on the data category contained in the data retrieval request, and obtains the target data asset; from the target data asset, queries the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and displays the detailed information and data records.

Images