A data asset quality detection method and detection device

Abstract

The application provides a data asset quality detection method and device, the method comprising: obtaining a to-be-detected data asset list; extracting keywords from the to-be-detected data asset list to construct a keyword corpus; labeling the keywords in the keyword corpus based on a preset self-defined word library to construct a detection feature keyword library; mapping the detection feature keyword library and the to-be-detected data asset list according to the labeled feature types to generate a feature list of each data asset; and respectively detecting the quality of each data asset feature list based on a preset detection rule list corresponding to each feature type to generate a quality detection result. According to the labeled features as the index of the selected detection rules, when the data quality is detected, the appropriate detection rules can be quickly matched, thereby reducing a large amount of repeated work and improving the work efficiency of data quality detection.

Description

Technical Field

[0001] This invention relates to the field of data quality inspection technology, and specifically to a data asset quality inspection method and inspection device. Background Technology

[0002] With the rapid development of information systems, the importance of data is becoming increasingly prominent. In the era of big data, data quality has become a key focus in data interaction and application, and the meaning of data quality is also subtly changing. Electricity data mainly originates from various stages of electricity production and use, directly reflecting the power operation status and enterprise operations. Data has become a fundamental strategic resource for company development, and data quality, to some extent, restricts the company's digital transformation. However, a large amount of data collected actually suffers from various quality problems, making accurate data quality detection a crucial foundation for improving data quality levels. Data quality detection methods suffer from problems such as high manual input, long processing times, high business dependence, and slow results, which have always been challenges in data governance. Many data quality teams treat data governance as a consulting and implementation project. With the development of artificial intelligence technology, intelligent and automated data governance processes have become possible. Among these, rule-based verification methods are widely used due to their relatively high accuracy. However, because they require extensive business knowledge and rule traversal of all content, their detection efficiency is low, time-consuming, and the use of detection data is delayed, reducing production efficiency. Therefore, improving data detection efficiency has become an urgent problem to be solved. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data asset quality detection method and detection device to overcome the problem of low efficiency in data asset quality detection in the prior art.

[0004] According to a first aspect, embodiments of the present invention provide a data asset quality detection method, the method comprising:

[0005] Obtain the list of data assets to be tested;

[0006] Keyword extraction is performed on the list of data assets to be detected to construct a keyword corpus;

[0007] Based on a preset custom thesaurus, the keywords in the keyword corpus are labeled with feature types to construct a detection feature keyword library;

[0008] According to the labeled feature types, the detection feature keyword library is mapped to the list of data assets to be detected to generate a feature list of each data asset.

[0009] Based on the list of preset detection rules corresponding to each feature type, data asset quality detection is performed on each data asset feature list to generate quality detection results.

[0010] Optionally, the step of labeling the keywords in the keyword corpus with feature types based on a preset custom thesaurus to construct a detection feature keyword library includes:

[0011] Select keywords from the keyword corpus at a first preset ratio as training samples;

[0012] The training samples are labeled using custom words from the preset custom lexicon as feature types;

[0013] The labeled training samples are input into the preset feature type labeling model for training, and the trained preset feature type labeling model is obtained.

[0014] The pre-trained preset feature type annotation model is used to annotate the feature types of all keywords in the keyword corpus to obtain the detection feature keyword library.

[0015] Optionally, before constructing a detection feature keyword library by annotating the keywords in the keyword corpus with feature types based on a preset custom thesaurus, the method further includes:

[0016] The number of identical words in the keyword corpus is counted.

[0017] Sort the statistical results from smallest to largest;

[0018] Based on the sorting results, remove a second preset proportion of keywords from the keyword corpus.

[0019] Optionally, the step of mapping the detection feature keyword library to the list of data assets to be detected according to the labeled feature types, and generating feature lists for each data asset, includes:

[0020] Obtain the keywords corresponding to the current feature type from the detection feature keyword library;

[0021] The keywords corresponding to the current feature type are mapped to the list of data assets to be detected;

[0022] Generate a list of current data asset features corresponding to the current feature type based on the mapping results.

[0023] Optionally, the step of performing data asset quality detection on each data asset feature list based on a preset detection rule list corresponding to each feature type, and generating quality detection results, includes:

[0024] Obtain the list of currently preset detection rules corresponding to the current feature type;

[0025] The current data asset feature list is subjected to data asset quality detection using the detection rules contained in the current preset detection rule list, and the current quality detection data corresponding to the current feature type is obtained.

[0026] The quality inspection results are generated based on the current quality inspection data corresponding to different feature types.

[0027] Optionally, the step of extracting keywords from the list of data assets to be detected and constructing a keyword corpus includes:

[0028] Based on the content attributes of the data asset list to be detected, corpus is extracted from the data asset list to be detected to obtain corpus corresponding to different content attributes;

[0029] The keyword corpus is constructed based on the corpora corresponding to the different content attributes.

[0030] Optionally, constructing the keyword corpus based on the corpora corresponding to the different content attributes includes:

[0031] Segment all corpora;

[0032] The results of word segmentation are used as keywords to construct the keyword corpus.

[0033] According to a second aspect, embodiments of the present invention provide a data asset quality detection device, applied to electronic devices, the device comprising:

[0034] The information acquisition module obtains a list of data assets to be tested;

[0035] The corpus extraction module extracts corpus data from the list of data assets to be detected and constructs a keyword corpus.

[0036] The feature annotation module annotates the keywords in the keyword corpus with feature types based on a preset custom lexicon, and constructs a detection feature keyword library;

[0037] The data asset feature list management module maps the detection feature keyword library to the data asset list to be detected according to the labeled feature types, and generates each data asset feature list.

[0038] The detection module, based on a pre-defined list of detection rules corresponding to each feature type, performs data asset quality detection on each data asset feature list and generates quality detection results.

[0039] According to a third aspect, embodiments of the present invention provide an electronic device, comprising:

[0040] A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect, or any alternative embodiment of the first aspect.

[0041] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing the computer to perform the method described in the first aspect, or any alternative embodiment of the first aspect.

[0042] The technical solution of this invention has the following advantages:

[0043] This invention provides a data asset quality detection method and apparatus, comprising: acquiring a list of data assets to be detected; extracting keywords from the list of data assets to be detected to construct a keyword corpus; labeling the keywords in the keyword corpus with feature types based on a preset custom thesaurus to construct a detection feature keyword library; mapping the detection feature keyword library to the list of data assets to be detected according to the labeled feature types to generate a feature list of each data asset; and performing data asset quality detection on each feature list based on a preset detection rule list corresponding to each feature type to generate quality detection results. By performing keyword extraction and labeling preprocessing on the original data asset list, and using the labeled features as indexes for selecting detection rules, the corresponding detection rules can be quickly matched and detected during data quality inspection, thus avoiding the need to traverse all rules for all content, avoiding a large number of invalid detection processes, and improving the efficiency of data asset quality detection. Attached Figure Description

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a data asset quality detection method according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating the process of constructing a detection feature keyword library for a data asset quality detection method according to an embodiment of the present invention;

[0047] Figure 3This is a schematic diagram illustrating the process of generating a data asset feature list according to a data asset quality detection method of this invention.

[0048] Figure 4 This is a schematic diagram of the detection process of a data asset quality detection method according to an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating the working process of a data asset quality detection method according to an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of a data asset quality detection device according to an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] The technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0054] Data assets refer to data resources, recorded in physical or electronic form, owned or controlled by individuals or businesses, that can bring future economic benefits to the business. Specifically, data assets refer to data in file form, such as photos, documents, drawings, videos, and digital copyrights, existing in data form in contrast to physical assets. Data assets are considered one of the most important asset forms in the digital age.

[0055] Electricity data primarily originates from various stages of electricity production and consumption, directly reflecting the operational status of the power industry and the company's overall operations. Data has become a fundamental strategic resource for the company's development, and data quality, to some extent, constrains the company's digital transformation. Accurate detection of data quality issues is a crucial foundation for improving data quality. Data quality detection methods suffer from high manual input, long processing times, high business dependence, and slow results, which have long been challenges in data governance. Many data quality teams treat data governance as a consulting and implementation project. With the development of artificial intelligence technology, intelligent and automated data governance processes have become possible. Among these methods, rule-based verification is widely used due to its relatively high accuracy. However, it requires extensive business knowledge and rule traversal of all content, resulting in low detection efficiency, long processing times, and a lag in the use of detection data, thus reducing production efficiency. Therefore, improving data detection efficiency is an urgent problem to be solved.

[0056] To address the aforementioned problems, embodiments of the present invention provide a data asset quality detection method, such as... Figure 1 As shown, the data asset quality inspection method mainly includes the following steps:

[0057] Step S101: Obtain the list of data assets to be tested.

[0058] Specifically, the list of data assets to be detected obtained in this embodiment of the invention is metadata information of production data, equipment data, etc. collected during the power industry production process. The metadata information is summarized in the form of an electronic spreadsheet. In other industries, the data can also be financial data, cost data, game data, etc. This invention is not limited to these.

[0059] Step S102: Extract keywords from the list of data assets to be tested and construct a keyword corpus.

[0060] Specifically, based on the content attributes of the data list to be tested, short texts are extracted from it, and these extracted short texts are used as keywords to form a keyword corpus.

[0061] Step S103: Based on a preset custom thesaurus, the keywords in the keyword corpus are labeled with feature types to construct a detection feature keyword library.

[0062] Specifically, in the power industry, much data consists of industry-specific terms. Therefore, common categories do not adequately correspond to the content attributes of these terms. Thus, a custom thesaurus is pre-defined based on industry characteristics to better classify and label industry keywords. Based on the different content attributes of the keywords in the keyword corpus, appropriate types of words in the custom thesaurus are labeled with features. For example, the keywords "type," "category," and "unit" are all labeled as "enumeration" feature types. Labeling methods can include manual labeling, clustering labeling, and supervised machine learning labeling; this invention is not limited to these. This embodiment of the invention employs a supervised learning method for model-based automatic labeling, significantly improving the efficiency and accuracy of feature labeling.

[0063] Step S104: Map the detection feature keyword library to the list of data assets to be detected according to the labeled feature types to generate feature lists for each data asset.

[0064] Specifically, methods such as fuzzy search, similarity calculation, and multi-result fusion can be used to map the detection feature keyword library to the data asset list. Each table and each field can be mapped to features in the detection feature keyword library, and the mapping results generate table-level and field-level data asset feature lists. The mapping method is not unique and belongs to existing technology, so it will not be elaborated here.

[0065] Step S105: Based on the preset detection rule list corresponding to each feature type, perform data asset quality detection on each data asset feature list and generate quality detection results.

[0066] Specifically, corresponding detection rules are set for keywords in each feature category. The pre-set list of detection rules includes feature type detection rules, corpus integrity detection rules, business rules from the table, and rules parsed from the descriptive corpus. Feature-based detection rules are used to directly detect data assets corresponding to different features. For example, if a field in an asset feature list table is labeled as an "enumeration" feature, then for this field in the table, the detection is performed directly using the detection rules corresponding to the "enumeration" type, the corpus integrity detection rules, and the field's business rules, without using detection rules corresponding to other types. This avoids the problem of testing an entire data asset list through all detection rules in one round, improving detection efficiency. Finally, the detection results of each category are merged into a quality detection result.

[0067] Specifically, in one embodiment, step S102 above includes the following steps:

[0068] Step S211: Based on the content attributes of the data asset list to be tested, extract corpus from the data asset list to obtain corpus corresponding to different content attributes. Specifically, extract short texts based on the content attributes of the data in the tables of the data asset list to be tested, such as: level, major / department, system, Chinese table name, English table name, table description, English field name, Chinese field name, field description, field type, whether it is a primary key, whether it is sensitive data, and extract key information such as Chinese table name, English table name, Chinese field name, English field name, and field description.

[0069] Step S212: Construct a keyword corpus based on the corpora corresponding to different content attributes as keywords.

[0070] Specifically, in one embodiment, step S212 above further includes the following step:

[0071] Step S301: Perform word segmentation on all corpora. Specifically, existing word segmentation model techniques can be used for word segmentation.

[0072] Step S302: Use the segmented results as keywords to construct a keyword corpus.

[0073] Specifically, in one embodiment, before step S103 above, the method further includes the following steps:

[0074] Step S105: Count the number of identical words in the keyword corpus. Specifically, count the number of times the same keyword appears in the keyword corpus.

[0075] Step S106: Sort the statistical results from smallest to largest. Specifically, based on the statistical results in step S105, sort the frequency of occurrence of the same keyword in ascending order.

[0076] Step S107: Remove keywords from the keyword corpus that represent a second preset proportion based on the ranking results. Specifically, based on the ranking results of step S106, keywords with high frequency are considered valid samples, while keywords with low frequency are considered invalid samples. Samples with a frequency higher than the second preset proportion are selected as valid samples, and samples with a frequency lower than the second preset proportion are considered invalid samples, and these invalid samples are removed. For example, in this embodiment of the invention, the preset proportion is 80%, but any preset proportion above 50% is acceptable; this invention is not limited to this. Removing samples with excessively low frequency can reduce interference factors in the annotation and training process, making the trained model more accurate.

[0077] Specifically, in one embodiment, such as Figure 2 As shown, step S103 above specifically includes the following steps:

[0078] Step S201: Select keywords from the keyword corpus at a first preset ratio as training samples.

[0079] Specifically, training samples can be selected from the full keyword corpus, with the quantity chosen according to a first preset ratio. For example, if the first preset ratio is 10%, then 10% of the keywords in the full keyword corpus are selected as training samples; this invention is not limited to this. After selecting the keywords, the selected keywords are vectorized to form a word matrix for the next step.

[0080] Step S202: Use custom words from the preset custom thesaurus as feature types to label the training samples.

[0081] Specifically, based on the different content attributes of the corpus in the keyword corpus, features are labeled for words of appropriate types in the custom thesaurus. For example, the keywords "type," "category," and "unit" are all labeled as "enumeration" feature type. Labeling methods can include manual labeling, clustering labeling, and supervised machine learning labeling; this invention is not limited to these. This embodiment of the invention uses a supervised learning method to achieve automatic feature labeling. It requires determining the training samples for the training model. Based on the training samples obtained in step S201, each training sample is labeled as the model's expected output, constructing a supervised learning model, and proceeding to step S203. Using a supervised learning method for word feature labeling reduces manual workload, improves efficiency, and accelerates the detection process for a large number of samples.

[0082] Step S203: Input the labeled training samples into the preset feature type labeling model for training to obtain the trained preset feature type labeling model.

[0083] Specifically, supervised training is performed on the preset feature type labeled model based on the training samples obtained in step S202. The model parameters can be adjusted to the optimal result using methods such as gradient descent or minimum distance. This invention is not limited to these methods, and the training of the model is finally completed.

[0084] Step S204: Use the trained preset feature type annotation model to annotate the feature types of all keywords in the keyword corpus to obtain the detection feature keyword library.

[0085] Specifically, all keywords in the keyword corpus are used as input, and the corresponding feature types of the keywords are obtained through the pre-trained feature type labeling model obtained in step S203. The processed keywords and feature type words are then merged to form the detection feature keyword library.

[0086] Specifically, in one embodiment, such as Figure 3 As shown, step S104 above specifically includes the following steps:

[0087] Step S205: Obtain the keywords corresponding to the current feature type in the detection feature keyword library. Specifically, for example: for one feature type, obtain all keywords corresponding to that feature type in the detection feature keyword library.

[0088] Step S206: Map the keywords corresponding to the current feature type to the list of data assets to be detected. Specifically, fuzzy search, similarity calculation, and multi-result fusion can be used to map the keywords corresponding to the current feature to the list of data assets. Based on these keywords, each corresponding table and each field can be mapped to the current feature.

[0089] Step S207: Generate a current data asset feature list corresponding to the current feature type based on the mapping result. Specifically, based on the mapping result of the current feature, generate a table-level and field-level data asset feature list for the current feature. For example, the data asset feature list contains form information such as Chinese table names, English table names, Chinese field names, and English field names. The features corresponding to the current list are: such as main transformer, zbyq, equipment addition method, sbzjfs, and enumeration. The obtained data asset feature list can be directly used for detection in the subsequent detection stage using the rules corresponding to the features, greatly improving detection efficiency.

[0090] Specifically, in one embodiment, such as Figure 4 As shown, step S105 above specifically includes the following steps:

[0091] Step S208: Obtain the list of currently preset detection rules corresponding to the current feature type. Specifically, different feature types have different type detection rules. In addition, there are corpus integrity detection rules, business rules in the table, and rules parsed from the descriptive corpus. For example, the table itself contains "voltage > 50KV", which is a business rule. The above rules are combined to form the list of currently preset detection rules corresponding to the current feature type.

[0092] Step S209: Perform data asset quality detection on the current data asset feature list using the detection rules contained in the current preset detection rule list to obtain the current quality detection data corresponding to the current feature type.

[0093] Specifically, data asset quality detection is performed on the current data asset feature list based on the current preset detection rule list of the current features obtained in step S208. For example, based on the rule detection list of the "enumeration" feature, quality detection is performed on the data asset feature list corresponding to the "enumeration" feature. This avoids unnecessary rule detection steps and improves detection efficiency.

[0094] Step S210: Generate quality inspection results based on the current quality inspection data corresponding to different feature types. Specifically, the feature lists of data assets of different feature types will yield different inspection data after inspection, and the obtained inspection data will be organized into quality inspection results.

[0095] In summary, as described above, Figure 5 As shown, by extracting and filtering keywords from the list of data assets to be inspected, and then labeling features based on a custom thesaurus within the industry, a data asset list with feature tags is obtained. In the quality inspection stage, inspection can be directly performed using the corresponding feature type detection rules, reducing numerous rule-based inspection steps, shortening inspection time, improving inspection efficiency, and lowering costs for subsequent production stages.

[0096] This invention also provides a data asset quality detection device, such as... Figure 6 As shown, the data asset quality inspection device includes:

[0097] The information acquisition module 101 obtains the list of data assets to be detected. For details, please refer to the relevant description of step S101 in the above method embodiment, which will not be repeated here.

[0098] The corpus extraction module 102 extracts corpus data from the list of data assets to be detected and constructs a keyword corpus. For details, please refer to the relevant description of step S102 in the above method embodiment, which will not be repeated here.

[0099] The feature annotation module 103 annotates the keywords in the keyword corpus with feature types based on a preset custom lexicon, constructing a detection feature keyword library. For details, please refer to the relevant description of step S103 in the above method embodiment, which will not be repeated here.

[0100] The data asset feature list management module 104 maps the detection feature keyword library to the list of data assets to be detected according to the labeled feature types, generating feature lists for each data asset. For details, please refer to the relevant description of step S104 in the above method embodiment, which will not be repeated here.

[0101] The detection module 105 performs data asset quality detection on each data asset feature list based on a preset list of detection rules corresponding to each feature type, and generates quality detection results. For details, please refer to the relevant description of step S105 in the above method embodiment, which will not be repeated here.

[0102] The data asset quality detection device provided in this embodiment of the invention is used to execute the data asset quality detection method provided in the above embodiment. Its implementation method and principle are the same. For details, please refer to the relevant description of the above method embodiment, which will not be repeated here.

[0103] Through the collaborative operation of the aforementioned components, the data quality inspection device provided in this embodiment of the invention extracts and filters keywords from the list of data assets to be inspected in advance, and then performs feature annotation based on a custom thesaurus within the industry to obtain a data asset list with feature tags. In the quality inspection stage, inspection can be performed directly using detection rules corresponding to the feature types, reducing numerous rule-based inspection steps, shortening inspection time, improving inspection efficiency, and lowering costs for subsequent production stages.

[0104] An electronic device provided in this embodiment of the invention, such as... Figure 7 As shown, the electronic device includes a processor 901 and a memory 902, wherein the processor 901 and the memory 902 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0105] Processor 901 can be a Central Processing Unit (CPU). Processor 901 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0106] The memory 902, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above method embodiments. The processor 901 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory 902, thereby implementing the methods in the above method embodiments.

[0107] The memory 902 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor 901, etc. Furthermore, the memory 902 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 902 may optionally include memory remotely located relative to the processor 901, and these remote memories may be connected to the processor 901 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] One or more modules are stored in memory 902, and when executed by processor 901, they perform the methods described in the above method embodiments.

[0109] The specific details of the aforementioned electronic device can be understood by referring to the relevant descriptions and effects in the above method embodiments, and will not be repeated here.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The implemented program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0111] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for detecting the quality of data assets, characterized in that, include: Obtain the list of data assets to be tested; Keyword extraction is performed on the list of data assets to be detected to construct a keyword corpus; The step of extracting keywords from the list of data assets to be detected and constructing a keyword corpus includes: extracting corpus from the list of data assets to be detected based on the content attributes of the list of data assets to be detected, obtaining corpus corresponding to different content attributes; segmenting all corpus into words; and using the segmented results as keywords to construct the keyword corpus. The process of labeling keywords in the keyword corpus with feature types based on a preset custom lexicon and constructing a detection feature keyword library includes: selecting keywords of a first preset proportion from the keyword corpus as training samples; labeling the training samples with custom words from the preset custom lexicon as feature types; inputting the labeled training samples into a preset feature type labeling model for training to obtain a trained preset feature type labeling model; and labeling all keywords in the keyword corpus with feature types using the trained preset feature type labeling model to obtain the detection feature keyword library. According to the labeled feature types, the detection feature keyword library is mapped to the data asset list to be detected to generate a feature list of each data asset. This mapping includes: using fuzzy query, similarity calculation, and / or multi-result fusion to map the detection feature keyword library to the data asset list, so that each table and each field of the data asset list can be mapped to a feature in the detection feature keyword library, generating table-level and field-level data asset feature lists. The mapping also includes: obtaining the keywords corresponding to the current feature type in the detection feature keyword library; mapping the keywords corresponding to the current feature type to the data asset list to be detected; and generating a current data asset feature list corresponding to the current feature type based on the mapping result. Based on the list of preset detection rules corresponding to each feature type, data asset quality detection is performed on each data asset feature list to generate quality detection results.

2. The method according to claim 1, characterized in that, Before constructing a detection feature keyword library by annotating the keywords in the keyword corpus with feature types based on a preset custom thesaurus, the method further includes: The number of identical words in the keyword corpus is counted. Sort the statistical results from smallest to largest; Based on the sorting results, remove a second preset proportion of keywords from the keyword corpus.

3. The method according to claim 1, characterized in that, The preset detection rule list corresponding to each feature type is used to perform data asset quality detection on each data asset feature list, generating quality detection results, including: Get the list of currently preset detection rules corresponding to the current feature type; The current data asset feature list is subjected to data asset quality detection using the detection rules contained in the current preset detection rule list, and the current quality detection data corresponding to the current feature type is obtained. The quality inspection results are generated based on the current quality inspection data corresponding to different feature types.

4. A data asset quality inspection device, applied to electronic equipment, characterized in that, The device includes: The information acquisition module obtains a list of data assets to be tested; The corpus extraction module extracts corpus data from the list of data assets to be detected and constructs a keyword corpus. This extraction includes: extracting corpus data from the list of data assets to be detected based on its content attributes, obtaining corpus data corresponding to different content attributes; segmenting all corpus data into words; and using the segmented results as keywords to construct the keyword corpus. The feature annotation module annotates the keywords in the keyword corpus with feature types based on a preset custom lexicon, constructing a detection feature keyword library. The step of annotating the keywords in the keyword corpus with feature types based on the preset custom lexicon to construct the detection feature keyword library includes: selecting a first preset proportion of keywords from the keyword corpus as training samples; annotating the training samples with custom words from the preset custom lexicon as feature types; inputting the annotated training samples into a preset feature type annotation model for training, obtaining a trained preset feature type annotation model; and using the trained preset feature type annotation model to annotate the feature types of all keywords in the keyword corpus, obtaining the detection feature keyword library. The data asset feature list management module maps the detection feature keyword library to the data asset list to be detected according to the labeled feature types, generating various data asset feature lists. This mapping, based on the labeled feature types, includes: using fuzzy query, similarity calculation, and / or multi-result fusion to map the detection feature keyword library to the data asset list, ensuring that each table and field in the data asset list can be mapped to a feature in the detection feature keyword library, generating table-level and field-level data asset feature lists. The mapping also includes: obtaining keywords corresponding to the current feature type in the detection feature keyword library; mapping the keywords corresponding to the current feature type to the data asset list to be detected; and generating the current data asset feature list corresponding to the current feature type based on the mapping result. The detection module performs data asset quality detection on each data asset feature list based on a list of preset detection rules corresponding to each feature type, and generates quality detection results.

5. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in any one of claims 1-3.

Images