Data asset management system and method

By introducing data asset quality monitoring and classification management modules into the data asset management system, the problem of low data quality monitoring and retrieval efficiency in the existing system is solved, efficient data quality management and rapid positioning are achieved, and data management efficiency is improved. reliability.

CN118917565AActive Publication Date: 2024-11-08蒲惠智造科技股份有限公司

Patent Information

Application Number
CN202411419322.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-08
Estimated Expiration
2044-10-12

AI Technical Summary

Technical Problem

The existing data asset management system lacks an effective data quality monitoring mechanism and data search indexing mechanism, which leads to inefficient data management, making it difficult to track data leaders and find specific data sets time-consuming and labor-intensive.

Method used

Provide a data asset management system, including a data asset import module, a quality monitoring module and a classification management module, data verification is carried out through preset field quality monitoring rules, verification results are generated and warning information is sent, and data asset classification management and retrieval is combined with data asset classification management and retrieval. Module, classifies and approximates data based on custom dimensions.

Benefits of technology

It significantly improves data quality monitoring capabilities, improves data reliability and decision-making quality, enhances data retrievalability and access efficiency, reduces the time to find specific data sets, and improves overall data management efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917565A_ABST
    Figure CN118917565A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and discloses a data asset management system and method, and the system comprises the steps: a data asset import module receives a data asset import request, obtains a data table or batch excel files based on the request, and obtains a to-be-processed data asset; the data asset quality monitoring module performs data verification on the to-be-processed data assets according to a preset field quality monitoring rule to generate a verification result; the data asset classification management module classifies the to-be-processed data assets of which the verification result indicates that no problem exists from predefined business dimensions and technical dimensions based on classification names customized by different preset dimensions and a user, and multiple types of data assets are obtained; the data asset retrieval module receives the data retrieval request and performs approximate retrieval according to the data retrieval request to obtain target data assets; and querying detailed information and data records of the data assets requested by the data retrieval request from the target data assets, and displaying the detailed information and the data records. Therefore, the data management efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data asset management system and method, which can be used in a data security exchange and sharing system. Background Art

[0002] In the field of data asset management, enterprises are faced with massive and diverse data management challenges. In order to meet these challenges, enterprises usually need a system that can efficiently manage and utilize data assets to support efficient retrieval, analysis and application of data, thereby helping enterprises extract valuable information from data and provide support for decision-making.

[0003] In the related technology, the existing data asset management system provides basic data management functions, but often lacks an effective data quality monitoring mechanism. For example, unreasonable data storage and archiving strategies make it difficult to track data owners. At the same time, there is a lack of effective data search and indexing mechanisms, which makes it time-consuming and laborious to find specific data sets, thereby reducing data management efficiency. Summary of the invention

[0004] The embodiments of the present application provide a data asset management system. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a data asset management system, the system comprising: Data asset import module, data asset quality monitoring module, data asset classification management module and data asset retrieval module; among them, The data asset import module is used to receive a data asset import request for data table import or Excel batch import, obtain the data table or batch Excel file based on the data asset import request, and obtain the data assets to be processed; The data asset quality monitoring module is used to perform data verification on the data assets to be processed according to the preset field quality monitoring rules and generate verification results; when there is a target field marked by triggering the preset field quality monitoring rules in the verification results, the warning information of the target field is generated and sent to the client of the person responsible for the target field; The data asset classification management module is used to classify the data assets to be processed from the predefined business dimension and technical dimension based on the user-defined classification names of different dimensions when there is no target field marked by triggering the preset field quality monitoring rule in the verification result, obtain multiple types of data assets, and store multiple types of data assets, wherein each type of data asset includes a classification tab and its data assets; The data asset retrieval module is used to receive data retrieval requests, perform approximate retrieval of classification tabs and their data assets based on the data categories contained in the data retrieval requests, and obtain target data assets; from the target data assets, query the detailed information and data records of the data assets corresponding to the data fields contained in the data retrieval request, and display the detailed information and data records.

[0006] Optionally, the data asset to be processed includes multiple fields and their field values; the preset field quality monitoring rules include a basic verification model and a deep verification model, and the basic verification model is used to verify the legitimacy of the field value of each field; According to the preset field quality monitoring rules, data verification is performed on the data assets to be processed, and verification results are generated, including: Input each field and its field value into the basic verification model to obtain the first check result of each field; Through the deep verification model, normal distribution verification and association rule mining are performed on each field and its field value to obtain the second inspection result of each field; The first check result and the second check result of each field are used as the verification result.

[0007] Optionally, each field and its field value are input into the basic verification model to obtain the first check result of each field, including: Check whether the field value of each field is empty through the basic validation model; Check whether the field value of each field conforms to the preset format through the basic verification model; Check whether the field value of each field is within the preset value range through the basic verification model; Check whether each field is unique through the basic validation model; Mark the fields that are empty, do not conform to the preset format, are not within the preset numerical range, and are not unique, and obtain the target fields that trigger the preset field quality monitoring rules; mark the fields that are not empty, conform to the preset format, are within the preset numerical range, and are unique, and obtain the target fields that do not trigger the preset field quality monitoring rules as the first inspection result of each field.

[0008] Optionally, normal distribution check and association rule mining are performed on each field and its field value to obtain a second check result for each field, including: Identify non-numeric type fields existing in the multiple fields, and convert field values ​​of the existing non-numeric type fields into numeric types using natural language processing technology to obtain field values ​​of multiple second fields; Calculate the mean and standard deviation of all fields according to the field value of each first field, each first field being each field of the multiple second fields and the numeric type fields present in the multiple fields; Based on the mean and standard deviation of all fields, the field value of each first field is converted to the Z score of the standard normal distribution to obtain the Z score of all fields; Determine whether the Z scores of all fields are within the preset standard normal distribution range; Mark the fields that are not within the interval of the preset standard normal distribution to obtain the target fields that trigger the preset field quality monitoring rule and are marked, and mark the fields that are within the interval of the preset standard normal distribution to obtain the target fields that do not trigger the preset field quality monitoring rule and are marked, as the normal distribution result of each field; Perform association rule mining on each field and its field value to obtain the association relationship results of each field; The normal distribution result and the association relationship result of each field are used as the second inspection result of each field.

[0009] Optionally, the average value of all fields is the ratio between the sum of the field values ​​of each first field and the number of first fields; wherein, The standard deviation of all fields is calculated as:

[0010] in, is the standard deviation of all fields, n is the number of the first field, For the The first field, is the average value of all fields; The formula for calculating the Z-score for all fields is:

[0011] in, is the Z score of all fields, X is the field value of each first field.

[0012] Optionally, perform association rule mining on each field and its field value to obtain the association relationship results of each field, including: Simulate the system business processing transactions corresponding to each field and its field value to convert each field into a transaction set; Calculate the proportion of the number of occurrences of each transaction in the transaction set; The transactions whose proportion is greater than the preset proportion are classified to obtain frequent item sets; The Apriori algorithm is used to analyze frequent item sets and obtain multiple association rules; Calculate the confidence of each rule; When the confidence of each rule is greater than a preset confidence threshold, it is determined that each field and its field value have an associated relationship; or when the confidence of each rule is less than or equal to the preset confidence threshold, it is determined that each field and its field value do not have an associated relationship; Mark each field and its field value for which there is an association relationship, and obtain the target fields that are marked without triggering the preset field quality monitoring rules, and mark each field and its field value for which there is no association relationship, and obtain the target fields that are marked for triggering the preset field quality monitoring rules, as the association relationship result for each field.

[0013] Optionally, predefined business dimensions include customer type and product category, and predefined technical dimensions include data format and data source; Based on the user-defined classification names of different preset dimensions, the data assets to be processed are classified from the predefined business dimensions and technical dimensions to obtain multiple types of data assets, including: Determine the business classification of each piece of data in the data assets to be processed based on customer type and product category; Determine the technical classification of each piece of data in the data assets to be processed based on the data format and data source; Create multiple category directories based on user-defined category names of different preset dimensions; From multiple category directories, identify the target category directory to which the business classification and technical classification of each piece of data belongs; Each piece of data is divided into the target category directory to which the business classification and technical classification of each piece of data belong, and multiple types of data assets are obtained.

[0014] Optionally, determine the business classification of each piece of data in the data assets to be processed based on the customer type and product category, including: Extract features related to customer type and product category from each piece of data in the data assets to be processed, and obtain customer type features and product category features; Use one-hot encoding or label encoding algorithm to convert customer type features and product category features into numerical feature vectors to obtain business dimension feature vectors; Input the business dimension feature vector into the pre-trained business classification model, and output the business classification of each data in the data asset to be processed; Follow these steps to generate a pre-trained business classification model, including: Use random forest algorithm to create business classification model; Acquire historical data assets; Extract features related to customer type and product category from each piece of historical data in the historical data assets, and obtain historical customer type features and historical product category features corresponding to each piece of historical data; For each piece of historical data, the historical customer type characteristics and historical product category characteristics are marked with business classification data labels to obtain the first model training sample; Input the first model training sample into the business classification model, and output the first model loss value; When the loss value of the first model reaches the minimum, a pre-trained business classification model is generated.

[0015] Optionally, determine the technical classification of each piece of data in the data asset to be processed based on the data format and data source, including: Extract features related to data format and data source from each piece of data in the data asset to be processed, and obtain data format features and data source features; Use one-hot encoding or label encoding algorithm to convert data format features and data source features into numerical feature vectors to obtain technical dimension feature vectors; Input the technical dimension feature vector into the pre-trained technical classification model, and output the technical classification of each data in the data asset to be processed; Follow these steps to generate a pre-trained technology classification model, including: Use random forest algorithm to create business technology models; Acquire historical data assets; Extract features related to data format and data source from each piece of historical data in the historical data assets, and obtain historical data format features and historical data source features corresponding to each piece of historical data; According to the historical data format characteristics and historical data source characteristics corresponding to each piece of historical data, a technical classification data label is marked to obtain a second model training sample; Input the second model training sample into the technical classification model and output the second model loss value; When the second model loss value reaches the minimum, a pre-trained technical classification model is generated.

[0016] In a second aspect, a data asset management method is provided, the method comprising: The data asset import module receives a data asset import request for data table import or Excel batch import, obtains the data table or batch Excel file based on the data asset import request, and obtains the data asset to be processed; The data asset quality monitoring module performs data verification on the data assets to be processed according to the preset field quality monitoring rules and generates verification results. When there is a target field marked by triggering the preset field quality monitoring rules in the verification results, the warning information of the target field is generated and sent to the client responsible for the target field. When there is no target field marked by triggering the preset field quality monitoring rule in the verification result, the data asset classification management module classifies the data assets to be processed from the predefined business dimension and technical dimension based on the preset user-defined classification names of different dimensions, obtains multiple types of data assets, and stores multiple types of data assets, where each type of data asset includes a classification tab and its data assets; The data asset retrieval module receives a data retrieval request, performs an approximate search on the classification tab and its data assets based on the data category contained in the data retrieval request, and obtains the target data asset; from the target data asset, queries the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and displays the detailed information and data records.

[0017] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the monitoring capability of data quality by implementing preset field quality monitoring rules. It can not only timely discover problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on custom classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 It is a schematic diagram of the system structure of a data asset management system provided by an embodiment of the present application; Figure 2 This is a data representation of the first check result of each field provided by an embodiment of the present application; Figure 3 This is a data representation of the second check result of each field provided by an embodiment of the present application; Figure 4 It is a flow chart of a data asset management method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0022] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0023] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of systems and methods consistent with some aspects of the present application as detailed in the attached claims.

[0024] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0025] The present application provides a data asset management system and method to solve the problems existing in the above-mentioned related technical problems. In the embodiment of the present application, on the one hand, the data asset quality monitoring module can significantly improve the monitoring ability of data quality by implementing preset field quality monitoring rules. It can not only timely discover problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send early warning information to the relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on customized classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency. The following uses an exemplary embodiment for detailed description.

[0026] See also Figure 1 , Figure 1 1 is a system structure diagram of a data asset management system provided by an embodiment of the present application, the system comprising: a data asset import module, a data asset quality monitoring module, a data asset classification management module and a data asset retrieval module; wherein the data asset import module is used to receive a data asset import request for data table import or Excel batch import, obtain a data table or a batch of Excel files based on the data asset import request, and obtain a data asset to be processed; the data asset quality monitoring module is used to perform data verification on the data asset to be processed according to a preset field quality monitoring rule and generate a verification result; when there is a target field marked by triggering the preset field quality monitoring rule in the verification result, an early warning information of the target field is generated and sent to the client of the person responsible for the target field; The data asset classification management module is used to classify the data assets to be processed from predefined business dimensions and technical dimensions based on the user-defined classification names of preset different dimensions when there is no target field marked by triggering the preset field quality monitoring rules in the verification results, so as to obtain multiple types of data assets and store multiple types of data assets, wherein each type of data asset includes a classification tab and its data assets; the data asset retrieval module is used to receive a data retrieval request, perform an approximate search on the classification tab and its data assets according to the data category contained in the data retrieval request, and obtain the target data asset; from the target data asset, query the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and display the detailed information and data records.

[0027] Among them, data asset import request refers to the instruction issued by the user, requiring the system to import new data assets. These requests can be to import a single data table or a batch of Excel files. Pending data assets refer to data that has been received by the system but has not yet been processed (such as verification, classification, etc.). Preset field quality monitoring rules are a set of rules pre-defined by the system to detect the quality of fields in data assets, such as non-null, format correctness, and range rationality. The verification result is the output of the data verification process, indicating whether the data complies with the quality monitoring rules and the specific fields that do not comply with the rules. The target field refers to the field that is detected to violate the quality monitoring rules during the data verification process. Warning information is a notification message generated by the system when a problem is found in the data verification, which is used to remind the relevant person in charge to pay attention and take action. The responsible person client refers to the user interface that has the right to access and handle data problems. It is a part of the data asset management system that allows the responsible person to receive warning information and take corresponding measures. User-defined category names refer to category labels or names that users customize in the system according to their own business needs. Business dimensions usually involve business logic, such as customer type and product category. Technical dimensions involve technical attributes of data, such as data format and data source. Multi-category data assets refer to a collection of data assets of multiple categories that are obtained by classifying data assets from business and technical dimensions according to user-defined classification names. The classification tab is the part of the user interface used to distinguish data assets of different categories, similar to folders in a file directory. A data retrieval request is a query instruction issued by a user, requiring the system to find data assets of a specific category or field. Approximate retrieval allows users to use fuzzy query conditions to retrieve data, and similar results can be found even if the query conditions are not completely accurate. Target data assets refer to eligible data assets retrieved by the system based on the user's data retrieval request. Detailed information and data records refer to the specific content of data assets, including the values ​​of data fields and other related metadata.

[0028] For example, suppose a large electronics retailer wants to build a data management system to track and analyze sales of various products. The system needs to be able to handle a large amount of sales transaction data and categorize the data by customer type and product category while ensuring data quality. The point-of-sale system generates a transaction record after each transaction and sends a data asset import request to the data management system.

[0029] The system receives a data table import request, obtains a real-time sales data table or batch imports a daily sales summary Excel file to form a data asset to be processed. The system verifies the sales data according to the preset field quality monitoring rules (such as transaction date is not empty, transaction amount is greater than zero, product number validity, etc.). After the verification result is generated, the system checks whether there are target fields that violate the quality rules. If there is a problem in the verification result, the system automatically generates an early warning message and sends it to the person responsible for data quality via email or system notification. The person in charge can log in to the system, view the detailed information of the problem data, and take corrective measures. For data assets that have passed the verification, the system classifies them according to different preset dimensions (such as customer type: individual, enterprise; product category: computer, mobile phone, accessories).

[0030] The classified data assets are stored in corresponding classification directories, such as different tables in a database or different folders in a file system. Each type of data asset includes a classification tab (such as a "personal computer user" tab) and its data assets.

[0031] The user needs to analyze the sales of a specific product category and submits a data retrieval request, specifying the product category as "mobile phone". After receiving the request, the data asset retrieval module performs an approximate search on the category tab and its data assets to find all data assets related to "mobile phone". The system queries the data fields included in the data retrieval request, such as sales volume, sales volume, etc., and displays the detailed information and data records of these data assets. The analysis results are presented in the system in the form of charts or reports for further study by users.

[0032] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the monitoring capability of data quality by implementing preset field quality monitoring rules. It can not only timely discover problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on custom classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0033] The data assets to be processed include multiple fields and their field values; the preset field quality monitoring rules include a basic verification model and a deep verification model, and the basic verification model is used to verify the legitimacy of the field value of each field.

[0034] In some embodiments of the present application, data verification is performed on the data assets to be processed according to preset field quality monitoring rules, and the specific process of generating verification results includes: inputting each field and its field value into the basic verification model to obtain the first inspection result of each field; through the deep verification model, each field and its field value are subjected to normal distribution verification and association rule mining to obtain the second inspection result of each field; the first inspection result and the second inspection result of each field are used as the verification result.

[0035] Among them, field refers to an attribute or data point in the data record, such as customer name, transaction amount, etc. The basic validation model is a preset model used to perform basic validation of data, such as checking basic data quality rules such as data format, type, non-null, range, etc. The field value refers to the specific data in the field. For example, in the "customer age" field, the specific value (such as 30) is the field value. The first check result refers to the result of the basic validation model's preliminary check of the field and its value. The deep validation model is used to perform more in-depth data quality checks, such as normal distribution verification and association rule mining. Normal distribution verification is a statistical method used to check whether data conforms to the normal distribution, and is often used for the verification of numerical data. Association rule mining is to mine hidden patterns or rules in the data by analyzing the correlation between data items, and is often used to discover interesting relationships between data. The second check result refers to the result of the deep validation model's in-depth check of the field and its value. The verification result is the result of combining basic verification and deep verification.

[0036] In the embodiment of the present application, by inputting each field and its field value into the basic verification model, the system can quickly identify basic problems with the data, such as missing values ​​or format errors, and obtain the first inspection result. Subsequently, the deep verification model is used to perform a deeper analysis of the data, including checking whether the data follows the expected statistical distribution and mining the potential association rules between the data, to obtain the second inspection result. Finally, combining the inspection results of these two levels, the final verification result can be obtained, which ensures the accuracy of subsequent data analysis and business decisions.

[0037] In some embodiments of the present application, each field and its field value are input into the basic verification model, and the specific process of obtaining the first inspection result of each field includes: checking whether the field value of each field is a null value through the basic verification model; checking whether the field value of each field conforms to the preset format through the basic verification model; checking whether the field value of each field is within the preset numerical range through the basic verification model; checking whether each field is unique through the basic verification model; marking fields that are null values, do not conform to the preset format, are not within the preset numerical range, and are not unique, and obtaining target fields that trigger the preset field quality monitoring rules; marking fields that are not null values, conform to the preset format, are within the preset numerical range, and are unique, and obtaining target fields that do not trigger the preset field quality monitoring rules as the first inspection result of each field.

[0038] Among them, in some embodiments, the first check result of each field is, for example, Figure 2 As shown in the figure, the result status after basic verification is displayed as "passed" or "failed". If the verification status is "failed", the specific error information will be provided here.

[0039] In some embodiments of the present application, each field and its field value are subjected to normal distribution verification and association rule mining, and the specific process of obtaining the second inspection result of each field includes: identifying non-numeric type fields existing in multiple fields, and converting the field values ​​of the existing non-numeric type fields into numeric types using natural language processing technology to obtain field values ​​of multiple second fields; according to the field value of each first field, calculating the mean and standard deviation of all fields, each first field is each field in the multiple second fields and the numeric type fields existing in the multiple fields; based on the mean and standard deviation of all fields, converting the field value of each first field into a standard Normally distribute the Z score to obtain the Z score of all fields; determine whether the Z score of all fields is within the preset standard normal distribution interval; mark the fields that are not within the preset standard normal distribution interval to obtain the target fields that trigger the preset field quality monitoring rules, and mark the fields that are within the preset standard normal distribution interval to obtain the target fields that do not trigger the preset field quality monitoring rules as the normal distribution result of each field; perform association rule mining on each field and its field value to obtain the association relationship result of each field; use the normal distribution result and the association relationship result of each field as the second inspection result of each field.

[0040] Specifically, the average value of all fields is the ratio of the sum of the field values ​​of each first field to the number of first fields; wherein the calculation formula for the standard deviation of all fields is:

[0041] in, is the standard deviation of all fields, n is the number of the first field, For the The first field, is the average of all fields; the calculation formula for the Z score of all fields is:

[0042] in, is the Z score of all fields, X is the field value of each first field.

[0043] Among them, non-numeric type fields refer to data that is not of numeric type contained in the data field, such as text or date type. Natural language processing technology is a technology used to process and analyze text data, including word meaning analysis, sentiment analysis, language detection, etc. The field value of the second field refers to the numeric type field value converted by natural language processing technology. The first field refers to the converted non-numeric type field and the field that is originally of numeric type.

[0044] Among them, in some embodiments, the second check result of each field is, for example, Figure 3 As shown, it includes different data fields, such as product ID, sales quantity, sales amount, customer evaluation, Z score, normal distribution verification result, association result and verification status.

[0045] In the embodiment of the present application, the present application will perform association rule mining on each field to discover the potential association relationship between the fields. Finally, the results of normal distribution analysis and association rule mining are combined to form the second inspection result of each field. This provides a more comprehensive perspective for data quality management and helps to identify and handle anomalies and inconsistencies in the data.

[0046] In some embodiments of the present application, each field and its field value are subjected to association rule mining, and the specific process of obtaining the association relationship result of each field includes: simulating the system business processing transactions corresponding to each field and its field value to convert each field into a transaction set; calculating the proportion of the number of occurrences of each transaction in the transaction set; classifying the transactions whose proportion is greater than the preset proportion to obtain a frequent item set; using the Apriori algorithm to analyze the frequent item set to obtain multiple association rules; calculating the confidence of each rule; when the confidence of each rule is greater than the preset confidence threshold, determining that each field and its field value have an association relationship; or when the confidence of each rule is less than or equal to the preset confidence threshold, determining that each field and its field value do not have an association relationship; marking each field and its field value that have an association relationship to obtain the target field that has not triggered the preset field quality monitoring rule and is marked, and marking each field and its field value that do not have an association relationship to obtain the target field that has triggered the preset field quality monitoring rule and is marked as the association relationship result of each field.

[0047] Among them, the predefined business dimensions include customer type and product category, and the predefined technical dimensions include data format and data source.

[0048] In some embodiments of the present application, based on preset user-defined classification names of different dimensions, the data assets to be processed are classified from predefined business dimensions and technical dimensions, and the specific process of obtaining multiple categories of data assets includes: determining the business classification of each data in the data assets to be processed according to the customer type and product category; determining the technical classification of each data in the data assets to be processed according to the data format and data source; creating multiple category directories based on preset user-defined classification names of different dimensions; identifying the target category directory to which the business classification and technical classification of each data belongs from the multiple category directories; dividing each data into the target category directory to which the business classification and technical classification of each data belongs, and obtaining multiple categories of data assets.

[0049] Among them, customer type refers to the classification of customers based on certain characteristics of the customers, such as purchasing habits, preferences, values, etc. Product category refers to the classification of products based on their characteristics, such as purpose, function, price range, etc. Category directory refers to the storage unit created for different categories or the directory structure in the database, which is used to organize and store classified data. Target category directory refers to the specific category directory to which each piece of data is assigned according to its business classification and technical classification attributes. Multi-category data assets refer to data sets organized according to different business classifications and technical classifications.

[0050] In the embodiment of the present application, by determining the business classification of each piece of data according to the customer type and product category, and determining the technical classification of each piece of data according to the data format and source, the data management system can organize and retrieve data more accurately. Creating multiple category directories and identifying the target category directory to which each piece of data belongs further improves the flexibility and efficiency of data management. Dividing data into corresponding business classification and technical classification directories not only makes data assets more orderly, but also enables enterprises to access and analyze specific types of data more quickly in this way, thereby accelerating the decision-making process and enhancing data-driven business insight capabilities.

[0051] In some embodiments of the present application, the specific process of determining the business classification of each data in the data assets to be processed based on the customer type and product category includes: extracting features related to the customer type and product category from each data in the data assets to be processed, respectively, to obtain customer type features and product category features; using a one-hot encoding or label encoding algorithm to convert the customer type features and product category features into numerical feature vectors to obtain a business dimension feature vector; inputting the business dimension feature vector into a pre-trained business classification model, and outputting the business classification of each data in the data assets to be processed.

[0052] Specifically, a pre-trained business classification model is generated according to the following steps, including: using a random forest algorithm to create a business classification model; obtaining historical data assets; extracting features related to customer type and product category from each historical data in the historical data assets, respectively, to obtain historical customer type features and historical product category features corresponding to each historical data; marking business classification data labels for historical customer type features and historical product category features corresponding to each historical data, to obtain a first model training sample; inputting the first model training sample into the business classification model, and outputting a first model loss value; when the first model loss value reaches a minimum, generating a pre-trained business classification model.

[0053] In some embodiments of the present application, the specific process of determining the technical classification of each data in the data assets to be processed based on the data format and the data source includes: extracting features related to the data format and the data source from each data in the data assets to be processed, respectively, to obtain data format features and data source features; using a one-hot encoding or label encoding algorithm to convert the data format features and the data source features into a numerical feature vector to obtain a technical dimension feature vector; inputting the technical dimension feature vector into a pre-trained technical classification model, and outputting the technical classification of each data in the data assets to be processed.

[0054] Specifically, a pre-trained technology classification model is generated according to the following steps, including: using a random forest algorithm to create a business technology model; obtaining historical data assets; extracting features related to the data format and the data source from each historical data in the historical data assets, and obtaining the historical data format features and historical data source features corresponding to each historical data; annotating the technology classification data labels with respect to the historical data format features and historical data source features corresponding to each historical data, and obtaining a second model training sample; inputting the second model training sample into the technology classification model, and outputting the second model loss value; when the second model loss value reaches the minimum, generating a pre-trained technology classification model.

[0055] In an embodiment of the present application, the business classification and technical classification of each piece of data can be predicted through a pre-trained business classification model and a pre-trained technical classification model, and the automated prediction method improves the classification efficiency.

[0056] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the monitoring capability of data quality by implementing preset field quality monitoring rules. It can not only timely discover problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on custom classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0057] See also Figure 4 , which is a flow chart of a data asset management method provided in an embodiment of the present application. Figure 4 As shown, the detection method of the embodiment of the present application may include the following steps: S101, a data asset import module receives a data asset import request for data table import or Excel batch import, obtains a data table or a batch of Excel files based on the data asset import request, and obtains data assets to be processed; S102, the data asset quality monitoring module performs data verification on the data assets to be processed according to the preset field quality monitoring rules and generates verification results; when there is a target field marked as triggering the preset field quality monitoring rules in the verification results, an early warning message of the target field is generated and sent to the client of the person in charge of the target field; S103, when there is no target field marked as triggering the preset field quality monitoring rule in the verification result, the data asset classification management module classifies the data assets to be processed from the predefined business dimension and technical dimension based on the preset user-defined classification names of different dimensions, obtains multiple types of data assets, and stores the multiple types of data assets, wherein each type of data asset includes a classification tab and its data assets; S104, the data asset retrieval module receives the data retrieval request, performs an approximate search on the classification tab and its data assets according to the data category contained in the data retrieval request, and obtains the target data asset; from the target data asset, queries the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and displays the detailed information and data records.

[0058] In the embodiments of the present application, on the one hand, the data asset quality monitoring module can significantly improve the monitoring capability of data quality by implementing preset field quality monitoring rules. It can not only timely discover problems in the data, such as inaccurate or outdated information, and generate verification results, but also automatically send warning information to relevant responsible persons when problems are found, thereby speeding up the response and resolution of problems. This proactive data quality management method improves the reliability of data and the quality of decision-making. On the other hand, the combined use of the data asset classification management module and the data asset retrieval module greatly enhances the retrievability and access efficiency of data. By allowing users to manage data based on custom classification dimensions and supporting multi-dimensional approximate retrieval, users can quickly locate the required data set. This efficient data retrieval mechanism not only reduces the time to find a specific data set, but also improves the overall data management efficiency.

[0059] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the data asset management methods provided by the above-mentioned various method embodiments.

[0060] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data asset management methods of the above-mentioned various method embodiments.

[0061] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the data asset management program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium of the data asset management program can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0062] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

[0063] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the data asset management program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the embodiments of the above-mentioned methods. The storage medium for data asset management can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0064] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data asset management system, characterized in that: The system comprises: Data asset import module, data asset quality monitoring module, data asset classification management module and data asset retrieval module; among them, The data asset import module is used to receive a data asset import request for data table import or Excel batch import, and obtain a data table or batch Excel file based on the data asset import request to obtain the data asset to be processed; The data asset quality monitoring module is used to perform data verification on the data assets to be processed according to the preset field quality monitoring rules and generate verification results; when there is a target field marked by triggering the preset field quality monitoring rules in the verification results, generate warning information of the target field and send it to the client of the person responsible for the target field; The data asset classification management module is used to classify the data assets to be processed from predefined business dimensions and technical dimensions based on preset user-defined classification names of different dimensions when there is no target field that triggers the preset field quality monitoring rule to be marked in the verification result, to obtain multiple types of data assets, and store the multiple types of data assets, wherein each type of data asset includes a classification tab and its data assets; The data asset retrieval module is used to receive a data retrieval request, and perform an approximate search on the classification tab and its data assets based on the data category contained in the data retrieval request to obtain the target data asset; from the target data asset, query the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and display the detailed information and data records.

2. The system according to claim 1, characterized in that The data assets to be processed include multiple fields and their field values; the preset field quality monitoring rules include a basic verification model and a deep verification model, and the basic verification model is used to verify the legitimacy of the field value of each field; The step of performing data verification on the data asset to be processed according to the preset field quality monitoring rules and generating a verification result includes: Input each field and its field value into the basic verification model to obtain a first inspection result of each field; By using the deep verification model, normal distribution verification and association rule mining are performed on each field and its field value to obtain a second inspection result for each field; The first check result and the second check result of each field are used as the verification result.

3. The system according to claim 2, characterized in that Each field and its field value are input into the basic verification model to obtain a first check result of each field, including: Check whether the field value of each field is a null value through the basic verification model; Check whether the field value of each field conforms to the preset format through the basic verification model; Checking whether the field value of each field is within a preset value range through the basic verification model; Check whether each field is unique through the basic verification model; Mark the fields that are empty, do not conform to the preset format, are not within the preset numerical range, and are not unique, and obtain the target fields that trigger the preset field quality monitoring rules; mark the fields that are not empty, conform to the preset format, are within the preset numerical range, and are unique, and obtain the target fields that do not trigger the preset field quality monitoring rules as the first inspection result of each field.

4. The system according to claim 2, characterized in that Perform normal distribution check and association rule mining on each field and its field value to obtain a second check result for each field, including: Identify non-numeric type fields existing in the multiple fields, and convert field values ​​of the existing non-numeric type fields into numeric types using natural language processing technology to obtain field values ​​of multiple second fields; Calculate the average value and standard deviation of all fields according to the field value of each first field, wherein each first field is each field of the plurality of second fields and the numerical type fields in the plurality of fields; Based on the mean value and standard deviation of all the fields, convert the field value of each first field into a Z score of a standard normal distribution to obtain the Z scores of all the fields; Determine whether the Z scores of all the fields are within a preset standard normal distribution interval; Marking the fields that are not within the interval of a preset standard normal distribution to obtain the target fields that trigger the preset field quality monitoring rule and marking the fields that are within the interval of a preset standard normal distribution to obtain the target fields that do not trigger the preset field quality monitoring rule as the normal distribution result of each field; Performing association rule mining on each of the fields and their field values ​​to obtain association relationship results for each of the fields; The normal distribution result of each field and the association relationship result are used as the second inspection result of each field.

5. The system according to claim 4, characterized in that The average value of all fields is the ratio between the sum of the field values ​​of each first field and the number of first fields; wherein, The calculation formula for the standard deviation of all fields is: in, is the standard deviation of all fields, n is the number of the first field, For the The first field, is the average value of all fields; The calculation formula for the Z score of all fields is: in, is the Z score of all the fields, and X is the field value of each first field.

6. The system according to claim 4, characterized in that The performing of association rule mining on each field and its field value to obtain the association relationship result of each field includes: Simulating the system business processing transactions corresponding to each field and its field value to convert each field into a transaction set; Calculate the proportion of the number of occurrences of each transaction in the transaction set; Classify the transactions whose proportion is greater than the preset proportion to obtain frequent item sets; The Apriori algorithm is used to analyze the frequent item sets to obtain multiple association rules; Calculate the confidence of each rule; When the confidence of each rule is greater than a preset confidence threshold, it is determined that each field and its field value have an associated relationship; or when the confidence of each rule is less than or equal to the preset confidence threshold, it is determined that each field and its field value do not have an associated relationship; Mark each field and its field value for which there is an association relationship, and obtain the target fields that are marked without triggering the preset field quality monitoring rules, and mark each field and its field value for which there is no association relationship, and obtain the target fields that are marked for triggering the preset field quality monitoring rules, as the association relationship results of each field.

7. The system according to claim 1, characterized in that Predefined business dimensions include customer type and product category, and predefined technical dimensions include data format and data source; Based on the preset user-defined classification names of different dimensions, the data assets to be processed are classified from the predefined business dimension and technical dimension to obtain multiple types of data assets, including: Determine the business classification of each piece of data in the data assets to be processed according to the customer type and product category; Determine the technical classification of each piece of data in the data asset to be processed according to the data format and data source; Create multiple category directories based on user-defined category names of different preset dimensions; From the plurality of category directories, identifying a target category directory to which the business category and technical category of each piece of data belongs; Each piece of data is classified into a target category directory to which the business classification and technical classification of each piece of data belong, so as to obtain multiple types of data assets.

8. The system according to claim 7, characterized in that Determining the business classification of each piece of data in the data assets to be processed according to the customer type and product category includes: Extracting features related to the customer type and product category from each piece of data in the data asset to be processed, respectively, to obtain customer type features and product category features; Use a one-hot encoding or label encoding algorithm to convert the customer type feature and the product category feature into a numerical feature vector to obtain a business dimension feature vector; The business dimension feature vector is input into a pre-trained business classification model, and the business classification of each piece of data in the data asset to be processed is output; wherein, Follow these steps to generate a pre-trained business classification model, including: Use random forest algorithm to create business classification model; Acquire historical data assets; Extracting features related to the customer type and product category from each piece of historical data in the historical data assets, respectively, to obtain historical customer type features and historical product category features corresponding to each piece of historical data; For each piece of historical data corresponding to the historical customer type feature and the historical product category feature, a business classification data label is marked to obtain a first model training sample; Inputting the first model training sample into the business classification model, and outputting a first model loss value; When the loss value of the first model reaches the minimum, a pre-trained business classification model is generated.

9. The system according to claim 7, characterized in that Determining the technical classification of each piece of data in the data asset to be processed according to the data format and the data source includes: Extracting features related to the data format and the data source from each piece of data in the data asset to be processed, respectively, to obtain data format features and data source features; Use a one-hot encoding or label encoding algorithm to convert the data format features and data source features into a numerical feature vector to obtain a technical dimension feature vector; The technical dimension feature vector is input into a pre-trained technical classification model, and the technical classification of each piece of data in the data asset to be processed is output; wherein, Follow these steps to generate a pre-trained technology classification model, including: Use random forest algorithm to create business technology models; Acquire historical data assets; Extracting features related to the data format and the data source from each piece of historical data in the historical data asset, respectively, to obtain historical data format features and historical data source features corresponding to each piece of historical data; According to the historical data format characteristics and historical data source characteristics corresponding to each piece of historical data, a technical classification data label is marked to obtain a second model training sample; Inputting the second model training sample into the technical classification model, and outputting a second model loss value; When the loss value of the second model reaches the minimum, a pre-trained technical classification model is generated.

10. A data asset management method, characterized in that: The method comprises: The data asset import module receives a data asset import request for data table import or Excel batch import, obtains a data table or a batch of Excel files based on the data asset import request, and obtains the data assets to be processed; The data asset quality monitoring module performs data verification on the data assets to be processed according to the preset field quality monitoring rules and generates verification results; when there is a target field marked by triggering the preset field quality monitoring rules in the verification results, the warning information of the target field is generated and sent to the client of the person in charge of the target field; When there is no target field marked as triggering the preset field quality monitoring rule in the verification result, the data asset classification management module classifies the data assets to be processed from predefined business dimensions and technical dimensions based on the preset user-defined classification names of different dimensions to obtain multiple categories of data assets, and stores the multiple categories of data assets, wherein each category of data assets includes a classification tab and its data assets; The data asset retrieval module receives a data retrieval request, performs an approximate search on the classification tab and its data assets based on the data category contained in the data retrieval request, and obtains the target data asset; from the target data asset, queries the detailed information and data records of the data asset corresponding to the data field contained in the data retrieval request, and displays the detailed information and data records.

Citation Information

Patent Citations

  • Cloud automatic early warning operation and maintenance monitoring system based on Hadoop

    CN108512691A

  • Excel data integration analysis system and method

    CN112214554A

  • Data center station system

    CN112396404A

  • Data verification method and device, computer equipment and storage medium

    CN114386088A

  • Data verification method, device, equipment, system, storage medium and product

    CN115640283A

Cited By

  • Intelligent traffic asset data management method and system

    CN119622556A

  • Report data asset identification method, system, equipment and medium

    CN120371926A