Method and system for rapid quality check and dynamic supplement of data asset pre-evaluation

Abstract

The application relates to the technical field of digital asset management, and discloses a data asset pre-evaluation quality rapid checking and dynamic supplement method and system, which comprises the following steps: obtaining a data asset registration package, analyzing and generating sample data objects and metadata objects; performing full-quality dimension checking to generate an initial quality profile; performing screening based on a minimum reliability threshold to generate a missing dimension list; judging whether to output a pre-evaluation report; when it is judged that the pre-evaluation report is not output, according to the missing dimension list, a supplement request template is matched from a dimension-supplement strategy mapping table to generate a supplement instruction package, which is sent to a data provider; receiving returned supplement data messages, performing compliance checking, updating the supplement data messages that pass the compliance checking through information fusion to generate an evaluation data set; and repeatedly performing the full-quality dimension checking, screening and judgment until the pre-evaluation report is output. The application improves the comprehensiveness, objectivity and reliability of the pre-evaluation result.

Description

Technical Field

[0001] This invention relates to the field of digital asset management technology, and more specifically, to a method and system for rapid quality verification and dynamic supplementation of data asset pre-assessment. Background Technology

[0002] Because data providers typically offer only a small sample of data from the complete dataset during the registration phase, third-party evaluation systems can only make quality judgments based on limited, localized data. For example, the absence of missing values ​​in the sample data does not accurately reflect the missing status of the corresponding fields in the complete dataset; the absence of duplicate primary keys in the sample data does not accurately reflect the uniqueness of the complete dataset. For large-scale data assets, the greater the difference between the number of sample data and the number of data in the complete dataset, the more difficult it is for quality indicators obtained from the samples to directly reflect the actual quality status of the complete dataset. When existing technologies find that the sample evaluation results are insufficient, they usually expand the sample size or acquire more data for re-evaluation. However, they lack a processing mechanism to determine the specific deficient quality dimensions based on the current evaluation results and obtain corresponding supplementary information for those dimensions. This means that the quality evaluation results obtained under limited sample conditions may still have significant uncertainty, while also increasing the amount of data interaction between the data provider and the evaluation system. Therefore, existing data asset pre-evaluation technologies, when only limited sample data is available, cannot dynamically obtain matching supplementary information for the quality dimensions with insufficient evaluation information based on the reliability of the evaluation results formed during the quality evaluation process. Consequently, it is difficult to obtain reliable data asset quality pre-evaluation results under limited data interaction conditions.

[0003] Therefore, it is necessary to design a method and system for rapid quality verification and dynamic supplementation of data asset pre-assessment in order to solve the problems existing in the current technology. Summary of the Invention

[0004] In view of this, the present invention proposes a method and system for rapid quality verification and dynamic supplementation of data asset pre-assessment, aiming to solve the problem that existing data asset pre-assessment technologies, when only limited sample data is available, cannot dynamically obtain matching supplementary information for quality dimensions with insufficient assessment information based on the reliability of the assessment results formed during the quality assessment process, thus making it difficult to obtain reliable data asset quality pre-assessment results under limited data interaction conditions.

[0005] In one aspect, this invention proposes a method for rapid quality verification and dynamic supplementation of data asset pre-assessment, comprising: Obtain the data asset registration package, parse and generate sample data objects and metadata objects; Perform full-dimensional quality verification on the sample data object and metadata object to generate an initial quality profile; The initial quality profile is screened based on the minimum reliability threshold, and a list of missing dimensions is generated based on the screening results. Determine whether to output a pre-assessment report based on the missing dimension list; When it is determined that no pre-evaluation report should be output, a supplementary request template is matched from the dimension-complementary strategy mapping table according to the missing dimension list, and deduplication and merging are performed to generate a supplementary instruction package; and the supplementary instruction package is sent to the data provider. Receive supplementary data messages returned by the data provider, perform compliance verification on the supplementary data messages, and merge and update the supplementary data messages that have passed the compliance verification with the sample data object and metadata object to generate an evaluation dataset; Based on the evaluation dataset, the full quality dimension verification, screening, and judgment are repeatedly performed until the pre-evaluation report is output.

[0006] Furthermore, the parsing and generation of sample data objects and metadata objects includes: The data asset registration package includes a sample data subset and a metadata description file; the sample data subset is transformed into a two-dimensional table to generate a sample data object, and the metadata description file is used to extract declaration items to generate a metadata object.

[0007] Furthermore, when performing full-quality dimension verification on the sample data object and metadata object, it includes: The sample data object is traversed, and the frequency of null values ​​or meaningless placeholders is counted to determine the missing rate of each field. According to the format constraints of the metadata object, the compliance ratio of the sample field values ​​is matched to obtain the standardization score. According to the primary key field combination of the metadata object, a hash comparison is performed between rows of the sample data object to identify duplicate key values ​​and output the duplication rate. Based on the timestamp of the sample data object and the update cycle and current time of the metadata object, the data freshness score is determined.

[0008] Furthermore, when performing full-quality dimension verification on the sample data object and metadata object, the following steps are also included: While performing the full quality dimension verification, the reliability level of each quality dimension is evaluated.

[0009] Furthermore, when screening the initial quality profile based on a minimum reliability threshold and generating a list of missing dimensions based on the screening results, the following are included: The reliability level of each quality dimension is compared with the minimum reliability threshold. The quality dimensions whose reliability level is less than the minimum reliability threshold are identified as missing quality dimensions. The missing cause code is determined according to the evaluation index of the missing quality dimension. Based on the missing quality dimension and the missing cause code, the missing dimension list is generated.

[0010] Furthermore, when determining whether to output a pre-assessment report based on the missing dimension list, the process includes: Determine whether the missing dimension list contains a key quality dimension; if the missing dimension list does not contain the key quality dimension, determine whether to output a pre-assessment report; otherwise, determine whether to output the pre-assessment report.

[0011] Furthermore, based on the missing dimension list, a supplementation request template is matched from the dimension-complementation strategy mapping table, and deduplication and merging are performed to generate a supplementation instruction package, including: Based on the missing quality dimension and missing cause encoding, the complement strategy and complement request template are matched in the dimension-complement strategy mapping table; duplicate request items are identified and merged in the matched complement request templates to generate a complement instruction package; the complement instruction package includes the target quality dimension, the complement data type, and the complement data requirements.

[0012] Furthermore, when generating the evaluation dataset, the following are included: The data types, data fields, statistical items, and data formats in the supplementary data message are matched and verified to determine whether the supplementary data message matches the supplementary data requirements in the supplementary instruction packet. When a match is determined, information fusion is performed based on the supplementary data message. The full statistical information is associated with the corresponding quality dimension, and the hierarchical statistical information is fused with the sample data object to generate the evaluation dataset.

[0013] Furthermore, until the output of the aforementioned pre-assessment report, it includes: Based on the evaluation dataset, a full quality dimension verification and screening is performed again to generate an updated list of missing dimensions; when the updated list of missing dimensions does not contain the key quality dimension, a pre-evaluation report is output. Alternatively, when the number of times the supplementary instruction package is generated reaches the maximum number of times, dynamic supplementation stops, and the pre-evaluation report is generated based on the current quality profile.

[0014] Compared to existing technologies, the advantages of this invention are as follows: It incorporates automatic identification of missing dimensions and a dynamic replenishment loop of minimum necessary information into the pre-assessment process. This allows the assessment to proactively transcend the boundaries of initial information, legally and lightweightly obtaining key statistical features or representative samples that effectively verify the overall quality of the data from the data provider through precise questioning. In the final pre-assessment report, each quality dimension's score is accompanied by highly reliable confidence support, eliminating extreme misjudgments caused by coincidence or deliberate selection of a single sample. This improves the comprehensiveness, objectivity, and credibility of the pre-assessment results with almost no increase in the burden on the data provider or the risk of privacy breaches.

[0015] On the other hand, this application also provides a rapid quality verification and dynamic supplementation system for data asset pre-assessment, used to apply the above-mentioned rapid quality verification and dynamic supplementation method for data asset pre-assessment, including: The registration information receiving module is configured to acquire the data asset registration package and parse it to generate sample data objects and metadata objects; The initial quality verification module is configured to perform full-dimensional quality verification on the sample data object and metadata object, and generate an initial quality profile. The missing dimension identification module is configured to screen the initial quality profile based on a minimum reliability threshold and generate a list of missing dimensions based on the screening results. The iterative evaluation control module is configured to determine whether to output a pre-evaluation report based on the list of missing dimensions. The supplementary strategy and request generation module is configured to, when it is determined that no pre-evaluation report should be output, match a supplementary request template from the dimension-complementary strategy mapping table according to the missing dimension list, perform deduplication and merging, generate a supplementary instruction package, and send the supplementary instruction package to the data provider; The supplementary data receiving and fusion module is configured to receive supplementary data messages returned by the data provider, perform compliance verification on the supplementary data messages, and perform information fusion and update the supplementary data messages that have passed the compliance verification with the sample data object and metadata object to generate an evaluation dataset. The iterative evaluation control module is also configured to repeatedly perform full-quality dimension verification, screening, and judgment based on the evaluation dataset until the pre-evaluation report is output.

[0016] It is understandable that the aforementioned methods and systems for rapid quality verification and dynamic supplementation of data asset pre-assessment have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the rapid quality verification and dynamic supplementation method for data asset pre-assessment provided in this embodiment of the invention; Figure 2 A functional block diagram of the data asset pre-assessment quality rapid verification and dynamic supplementation system provided in this embodiment of the invention. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] In some embodiments of this application, see Figure 1 As shown, a method for rapid quality verification and dynamic supplementation of data asset pre-assessment is proposed, including: S100: Obtain the data asset registration package, parse and generate sample data objects and metadata objects; S200: Perform full-quality dimension verification on sample data objects and metadata objects, and generate an initial quality profile; S300: Screen the initial quality profile based on the minimum reliability threshold and generate a list of missing dimensions based on the screening results; S400: Determine whether to output a pre-assessment report based on the list of missing dimensions; S500: When it is determined that no pre-assessment report should be output, the supplementary request template is matched from the dimension-complementary strategy mapping table according to the missing dimension list, and deduplication and merging are performed to generate a supplementary instruction package; and the supplementary instruction package is sent to the data provider. S600: Receives supplementary data messages returned by the data provider, performs compliance verification on the supplementary data messages, and merges and updates the supplementary data messages that have passed the compliance verification with the sample data objects and metadata objects to generate an evaluation dataset; S700: Repeatedly perform full-quality dimension verification, screening and judgment based on the evaluation dataset until a pre-evaluation report is output.

[0020] This embodiment provides a method for rapid quality verification and dynamic supplementation of data asset pre-assessment, applied to a third-party assessment system. The third-party assessment system can be deployed in government open data platforms, data exchanges, enterprise data trading platforms, data asset registration platforms, or other platforms used for quality assessment before the registration, opening, circulation, or trading of data assets. The third-party assessment system is used to complete the initial quality verification of data assets using a small amount of sample data and metadata description information provided by the data provider, even when the data provider does not directly submit a complete dataset. Based on the reliability of each quality dimension during the initial quality verification process, it identifies information gaps in the current assessment. When further information is needed, it generates targeted supplementation instructions based on the specific missing dimension and its cause, obtains supplementary information corresponding to the missing dimension from the data provider, and integrates the supplementary information with existing assessment information. Based on this, a new quality verification and reliability assessment are performed, thus forming a closed-loop assessment process of "initial verification—reliability screening—gap identification—targeted supplementation—information fusion—re-verification."

[0021] In this embodiment, when a data provider needs to submit data assets to a data trading platform, data open platform, or data asset registration platform for pre-evaluation, it first triggers the data asset registration process and submits the data asset registration package to the third-party evaluation system.

[0022] A data asset registration package includes at least a sample data subset and a metadata description file. The sample data subset is a small number of data records extracted from the complete dataset held by the data provider, not the complete dataset itself. The complete dataset may include all business data actually held by the data provider, such as supply chain transaction data, customer transaction data, equipment operation data, production and operation data, or other data to be registered as data assets. To reduce the amount of data transferred during the data asset registration phase and to avoid directly exposing the complete dataset to third-party evaluation systems, the data provider extracts only a portion of records from the complete dataset to form the sample data subset, which serves as the data basis for the initial quality assessment.

[0023] A subset of sample data can be extracted from the complete dataset according to preset sampling rules. For example, a number of records can be selected using random sampling, or a certain number of records can be obtained according to sampling rules preset by the data provider. In some implementations, the subset of sample data may include dozens, hundreds, or other orders of magnitude of data records, the specific number of which can be determined based on the scale of the data assets, the sensitivity of the data, and the platform's preset registration requirements. This invention does not limit the specific number of records in the subset of sample data; its core is that the subset of sample data is a relatively small local data set relative to the complete dataset.

[0024] Metadata description files are used to describe the structural and data claim information of the data assets to be evaluated. Metadata description files can be in structured, semi-structured, or other file formats that can be parsed by a third-party evaluation system. They may include field names, field types, field meanings, field format constraints, claimed total number of data rows, data update cycle, primary key field definitions, and other declarations related to quality verification.

[0025] Upon receiving the data asset registration package, an integrity check is first performed to determine if a sample data subset and metadata description file exist within the package. The format, structure, and content of these files are then preliminarily analyzed. For data asset registration packages that can be parsed successfully, the sample data subset is further read, and the data records are converted into a two-dimensional table-like collection of records according to field relationships, thus forming the sample data object.

[0026] In one specific implementation, the sample data object may include multiple data rows and multiple data fields, where each row corresponds to one sample data record and each column corresponds to one data field. A field identifier can be established for each field, and basic information such as the field name, field position, and field value type can be recorded to facilitate subsequent traversal and statistics of the sample data according to field dimensions.

[0027] Simultaneously, the metadata description file is used to extract declaration items, including the field list, data types, field meanings, format constraints, declared total number of data rows, update cycle, and primary key field definitions, forming a metadata object. This metadata object provides the data structure and quality judgment basis for subsequent quality verification.

[0028] For example, when a field is declared as a date field in the metadata description file, the metadata object can store the data type and date format constraints corresponding to that field; when two fields are declared to form a primary key in the metadata description file, the metadata object can store the combination relationship between the two fields; when the metadata description file declares that the complete dataset includes nearly one million records and is updated daily, the metadata object can record the declaration of the total number of rows and the declaration of the update cycle, respectively.

[0029] After completing the above analysis, multiple quality dimensions are verified on the data asset to be evaluated based on the sample data object and metadata object. In this embodiment, the quality dimensions may include at least completeness, standardization, uniqueness, and timeliness. These quality dimensions evaluate the data asset from aspects such as whether data is missing, whether the data format meets constraints, whether key fields are duplicated, and whether the data remains in the expected updated state.

[0030] In this embodiment, multiple quality verification rules are set up that can be executed independently, and each quality verification rule corresponds to a quality dimension. These multiple quality verification rules can be executed in a preset order or in parallel, thereby improving the efficiency of the initial quality verification.

[0031] For the integrity quality dimension, iterate through each field in the sample data object and count the occurrences of null values, missing values, or meaningless placeholders for each field. Meaningless placeholders can include preset characters used to represent unknown, unfilled, or invalid information, such as specific strings, special symbols, or platform-defined placeholder identifiers.

[0032] Based on the correlation between the number of missing records and the number of records in the sample data obtained from statistics, the missing rate of each field is determined, and the corresponding completeness indicators are recorded according to the importance of the field. For key business fields, their completeness indicators can be recorded separately to facilitate subsequent determination of whether the field belongs to a key quality dimension.

[0033] For example, if a sample data object includes 200 records, and 10 records show a missing value for a core business field, then the missing value of that field can be converted into a corresponding missing rate metric. If no missing values ​​are found for a core field in any of the 200 sample records, then the corresponding sample missing rate can be considered to be of low missing value. However, this result only represents the observation result in the current 200 sample records and cannot directly conclude that the field is not missing in the complete dataset. Therefore, this embodiment not only records the missing rate but also further assesses the reliability of that missing rate with the current sample size.

[0034] For the normative quality dimension, the format constraints of the corresponding fields in the metadata object are read, and the field values ​​in the sample data object are matched item by item against the format constraints. Format constraints may include date format, encoding format, numeric format, character length, data type, or other format rules declared by the data provider in the metadata.

[0035] In one specific implementation, for fields declared as date type, it can be determined whether the corresponding field value can be parsed according to the date format declared in the metadata; for fields declared as having a specific encoding format, matching can be performed according to the corresponding format rules; for fields declared as having a fixed length, it can be determined whether the length of the actual field value meets the length requirement. The compliance ratio of sample field values ​​is determined based on the number of data records that pass the matching and the number of data records participating in the verification, and a standardization score is obtained accordingly.

[0036] For the uniqueness quality dimension, the primary key field combination in the metadata object is read first. The primary key field combination can include one field or multiple fields. When the primary key consists of multiple fields, these fields are combined in a preset order to form a single primary key record.

[0037] To improve the efficiency of inter-row comparison, a corresponding hash identifier can be generated for the primary key field combination of each sample data record, and hash comparisons can be performed between different data rows of the sample data object. When different data rows correspond to the same primary key hash identifier, the data row is identified as having duplicate key values, and the duplication rate in the sample data is determined based on the duplication situation.

[0038] It should be noted that uniqueness verification in sample data primarily reflects the presence of duplicate primary keys in the current sample set. When the sample data size is small, even if no duplicate primary keys are found in the sample, it cannot be directly inferred that no duplicate primary keys exist in the complete dataset. Therefore, the reliability level of the uniqueness indicator also needs to be determined in conjunction with the current sample size and available auxiliary information.

[0039] For the timeliness quality dimension, time-related fields or time fields specified in the metadata are extracted from the sample data objects, and the timestamp information in the sample records is determined. The latest timestamp in the sample records can be determined, and combined with the data update cycle declared in the metadata object and the current time, the time interval between the current data and its expected update time can be determined, thereby determining the data freshness score.

[0040] For example, if the metadata states that a data asset is updated on a daily basis, but the latest record in the sample data is older than the expected update cycle, the corresponding data freshness score can be lowered; if the interval between the latest timestamp in the sample and the current time matches the declared update cycle, a higher data freshness score can be obtained.

[0041] The aforementioned integrity, standardization, uniqueness, and timeliness checks can generate corresponding quality indicators, which can then be combined to form the current data asset quality evaluation results.

[0042] Unlike the traditional approach of generating quality scores directly based solely on sample metrics, this embodiment evaluates the reliability of each quality metric under the current information conditions while performing verification of each quality dimension.

[0043] Because data providers only offer a small subset of sample data extracted from the complete dataset during the registration phase, quality indicators obtained directly from the sample data are evaluation results based on localized data. Especially when the complete dataset is large, even if a quality problem is observed to occur infrequently or not at all in the sample, it cannot be directly assumed that the actual quality situation in the complete dataset is completely consistent with the sample observations.

[0044] Therefore, while calculating the scores for each quality indicator, a corresponding reliability evaluation result is also generated for each quality indicator. The reliability evaluation result can be represented by discrete levels such as "high," "medium," and "low," or it can use other level formats preset by the platform.

[0045] In one specific implementation, the statistical reliability of the current quality indicator is evaluated based on information such as the number of sample records, the number of observed quality events, the type of quality indicator, and a preset confidence level. For example, for a completeness indicator, the possible range of variation of the missing record rate in the complete dataset can be determined based on the number of missing records observed in the current sample and the sample size; when the sample size is small and the corresponding possible range of variation is large, the completeness indicator is marked as low reliability.

[0046] In another specific implementation, confidence interval estimation can be used to evaluate the reliability of quality indicators. Taking the field missing rate as an example, when the sample size is tens or hundreds of records and the observed missing rate is zero, the possible range of variation of the missing rate in the complete dataset can still be determined based on statistical methods such as binomial distribution or Wilson intervals, and the corresponding evaluation range can be determined according to a preset confidence level. When the evaluation range is wide, it indicates that the current sample data is insufficient to form a stable judgment on the missing status of the complete dataset, and the reliability of the indicator can be marked as "low".

[0047] For normative indicators, the reliability of the indicators can be determined based on the number of samples, the number of format violation records, and the corresponding statistical scope. For uniqueness indicators, the reliability of the indicators can be determined based on the number of samples, the number of duplicate key values, and the structure of the primary key field. For timeliness indicators, the reliability of the indicators can be determined by combining the relationship between the sample time coverage, the update time declaration, and the current time.

[0048] It should be noted that the reliability level in this embodiment does not simply represent the score of a certain quality dimension, but rather indicates whether the current quality indicator has sufficient information to support it. For example, if no missing values ​​are found in 200 sample data points for a certain field, its completeness score may be high. However, since the sample size is relatively small compared to a complete dataset of millions, the reliability of this completeness score may still be "low". Therefore, this embodiment distinguishes between "quality level" and "reliability of quality assessment".

[0049] After completing the above processing, the quality indicators, quality scores, and reliability levels corresponding to each quality dimension are combined to form an initial quality profile. The initial quality profile may include information such as quality dimension identifiers, quality indicators, quality scores, reliability levels, and the evaluation criteria corresponding to the reliability levels.

[0050] Obtain the initial quality profile and compare the reliability level corresponding to each quality dimension with the preset minimum reliability threshold.

[0051] The minimum reliability threshold is used to determine whether the current quality indicators have sufficient information to support them. The minimum reliability threshold can be preset by the data trading platform, data evaluation agency, or data asset registration platform. For example, when using a three-level reliability representation of "high," "medium," and "low," "medium" can be set as the minimum reliability threshold. When the reliability level of a certain quality dimension is "low," it indicates that the current sample data is insufficient to form a reliable evaluation result that meets the platform's requirements; therefore, this quality dimension is identified as a missing quality dimension.

[0052] Further determine the missing cause code corresponding to the missing quality dimension. In this embodiment, a missing cause rule set is pre-established, which includes at least the following missing cause rules: insufficient sample size, insufficient number of quality event observations, incomplete primary key field definition, insufficient field distribution information, and insufficient time coverage. For each quality dimension below the minimum reliability threshold, the corresponding evaluation metric, sample size, quality event statistics, metadata completeness, and time coverage are matched with the missing cause rule set. When the triggering condition of the corresponding missing cause rule is met, the code corresponding to that missing cause rule is determined as the missing cause code for the quality dimension.

[0053] In one specific implementation, the missing reason code can be used to distinguish different sources of information gaps. For example, when the integrity index of a certain field cannot obtain a sufficiently reliable evaluation result due to an insufficient number of sample records, it can correspond to the missing reason code "number of sample records is lower than the minimum threshold for reliable evaluation"; when uniqueness verification requires a complete primary key definition but the metadata does not provide a complete set of primary keys, it can correspond to the missing reason code "missing a complete set of primary keys"; when consistency verification requires cross-table reference information but the corresponding information does not exist in the current registration package, it can correspond to the missing reason code "missing cross-table reference information".

[0054] Therefore, the missing dimension list not only records "which metric is unreliable", but also "why it is unreliable". In one implementation, the missing dimension list may include fields such as quality dimension identifier, specific quality metric, reliability level, minimum reliability threshold, missing reason code, and whether the quality dimension belongs to a critical quality dimension.

[0055] After obtaining the list of missing dimensions, the platform further determines whether any critical quality dimensions exist within this list. Critical quality dimensions can be pre-determined by the platform based on the business purpose of the data assets. For example, the completeness of core business fields and the uniqueness of primary keys can be set as critical quality dimensions; for some data assets, the timeliness of data updates can also be set as a critical quality dimension according to business requirements. The specific configuration method for critical quality dimensions can be adjusted according to different data asset types.

[0056] When the missing dimension list does not contain any key quality dimensions, it means that although there may be some auxiliary indicators with insufficient reliability in the current assessment results, these indicators are not enough to affect the current data asset pre-assessment conclusion, so the output pre-assessment report can be directly controlled.

[0057] If a key quality dimension is present in the missing dimension list, the information used for the current quality assessment is considered insufficient, and the final pre-assessment result cannot be directly output. In this case, a dynamic supplementation process is initiated.

[0058] The dimension-complement strategy mapping table is used to establish the correspondence between missing quality dimensions, missing cause codes, complement strategies, and complement request templates. The dimension-complement strategy mapping table can be pre-established during the deployment phase or maintained by the assessment agency based on its experience in assessing actual data assets. The mapping table includes at least the quality dimension identifier, missing cause code, complement strategy, and complement request template.

[0059] For example, when the missing quality dimension is completeness, and the missing reason is encoded as "insufficient sample size leading to insufficient reliability of completeness assessment," the dimension-complement strategy mapping table can map it to a "column-level overview statistics request." A column-level overview statistics request does not require the data provider to re-upload a large amount of original data; instead, it requires the data provider to return statistical information such as the total number of records in the complete dataset and the number of non-empty records for the target field.

[0060] In one implementation, the column-level overview statistics request can also require the data provider to return distribution information related to the target field, such as interval statistics divided according to the field value length. Since the statistics are summary information, the data provider can complete the statistics using the aggregation query capabilities of its local database without having to transmit the corresponding original records to a third-party evaluation system.

[0061] When the reason for the missing data is "missing field distribution information, resulting in the inability to verify validity or accuracy," the dimension-complement strategy mapping table can map it to a "stratified summary request." A stratified summary request is used to request the data provider to perform stratified statistics on the complete dataset according to the category, value range, or other preset stratification conditions of the target field, and return the count, mean, typical value, or other descriptive statistical information corresponding to each stratum.

[0062] When the reason for the missing information is "missing complete primary key information", it can be mapped to a primary key supplementation request to request the data provider to supplement the primary key field combination information; when the reason for the missing information is "missing cross-table reference information", it can be mapped to a reference relationship supplementation request to obtain the relationship information between the target field and other data tables.

[0063] Read the quality dimensions and their missing reason codes one by one from the missing dimension list, and look up the corresponding complement strategy and complement request template in the dimension-complement strategy mapping table.

[0064] Since different missing dimensions may correspond to the same or partially overlapping request items, duplicate request item identification and request item merging are further performed on the matched supplementary request templates. For example, when both the integrity dimension and the normalization dimension require the total number of records in a certain field, the same statistical items in the two requests can be merged into one request item, without having to repeatedly request the same data from the data provider.

[0065] After deduplication and merging, a supplementary instruction package is formed. The supplementary instruction package may include information such as target quality dimension, target quality index, missing reason code, supplementary data type, supplementary data requirements, statistical range, statistical fields, hierarchical conditions, and request identifier.

[0066] The supplementary instruction packet is then sent to the data provider via a pre-established secure interface between the third-party evaluation system and the data provider. This secure interface can employ a platform-preset data exchange interface or other secure data communication methods, enabling the supplementary request to be transmitted in a structured message format.

[0067] After receiving the supplementary instruction packet, the data provider generates the corresponding supplementary data message in the local data environment based on the target quality dimension and supplementary data requirements contained therein.

[0068] In this embodiment, the data provider does not need to send the complete dataset directly to the third-party evaluation system. Instead, it performs corresponding statistical operations based on the supplementary instruction package. For example, when the supplementary instruction package requests the return of the total number of rows and the number of non-empty records in the complete dataset for a certain core field, the data provider can obtain the total number of rows and the number of non-empty records in its local database through aggregation statistical operations.

[0069] When the supplementary instruction package requests the return of distribution statistics divided according to the length of field values, the data provider can perform group statistics according to the preset length interval, obtain the number of records corresponding to each length interval, and form hierarchical statistical information from the statistical results of each interval.

[0070] When the supplementary instruction package requests the return of statistics stratified by category or value range, the data provider can perform grouped statistics in the local database according to the corresponding category or value range and obtain the number of records, mean or other descriptive statistics for each stratum.

[0071] When the supplementary instruction package requests information on primary key duplication, the data provider can perform grouping statistics on the primary key field within the complete dataset to determine the number of duplicate primary keys, the number of duplicate records, or the distribution of duplication patterns, and return the corresponding statistical results.

[0072] In this way, the third-party evaluation system obtains supplementary information for specific quality dimensions, rather than the complete dataset itself. Therefore, while ensuring that the information required for quality assessment is supplemented, the amount of data output from the data provider can be reduced.

[0073] After completing the statistics, the data provider encapsulates the statistical results into a supplementary data message according to the data format specified in the supplementary instruction package, and returns it to the third-party evaluation system through a secure interface.

[0074] Upon receiving the supplementary data message from the data provider, a compliance check is first performed on the supplementary data message. The compliance check determines whether the data returned by the data provider meets the supplementary data requirements in the supplementary instruction packet. The compliance check can include at least data type validation, data field validation, statistical item validation, and data format validation.

[0075] Among them, data type validation is used to determine whether the returned data belongs to the statistical data, hierarchical data or other specified data types required by the supplementary instruction package; data field validation is used to determine whether the returned result contains the target field specified by the supplementary request; statistical item validation is used to determine whether the returned result contains the total number of rows, the number of non-empty records, the number of duplicate records or other statistical items specified in the request; and data format validation is used to determine whether the returned message conforms to the preset structure and format requirements.

[0076] For example, when a supplementary instruction packet requests the data provider to return the total number of rows and the number of non-empty records for a certain core field, it checks whether the supplementary data message contains both the total number of rows and the number of non-empty records for the corresponding field, and determines whether the data type and format of the corresponding statistical item meet the requirements.

[0077] When the supplementary instruction packet requests the return of hierarchical statistics, it can further check whether there are preset hierarchical identifiers and corresponding statistical values ​​for each hierarchical in the supplementary data message, and determine whether the number of hierarchical and statistical structure returned matches the request requirements.

[0078] When a supplementary data message fails the compliance check, the message can be marked as invalid supplementary data, and the specific reason for non-compliance can be returned so that a supplementary request can be regenerated or exception handling can be performed later.

[0079] Once the supplementary data message passes compliance verification, it enters the information fusion processing stage. Different information fusion methods are executed depending on the specific type of the supplementary data message. For the full statistical information returned by the data provider, since the full statistical information directly performs statistics on the complete dataset, it can be associated with the corresponding quality dimensions, and the original quality indicators can be updated using the full statistical information.

[0080] For example, regarding the integrity quality dimension, if the data provider returns the total number of records and the number of non-empty records for a core business field in the complete dataset, the actual number of missing records in the complete dataset can be determined based on the total number of records and the number of non-empty records, and this result can be correlated with the integrity quality dimension. In this case, the integrity index originally inferred from a small number of samples can be updated by the corresponding full statistical information, thereby reducing the uncertainty caused by insufficient sample size.

[0081] For primary key duplication statistics returned by the data provider, they can be associated with the uniqueness quality dimension, and the original uniqueness index can be updated according to the number of duplicate records or the distribution of duplication patterns.

[0082] For the stratified statistical information returned by the data provider, the stratified statistical information can be integrated with the original sample data objects. Specifically, firstly, the supplementary data is divided into multiple stratified data sets according to the stratification identifier. Then, the stratified data sets are combined with the original sample data objects according to field identifiers, stratification identifiers, or other association relationships to form an evaluation data set containing original sample information and supplementary stratification information.

[0083] In this embodiment, the fused data is not required to be equivalent to the complete dataset held by the data provider, but rather an evaluation dataset used for quality assessment. The evaluation dataset may simultaneously include the original sample data, full statistical information, hierarchical statistical information, and supplementary evaluation information corresponding to the quality dimensions.

[0084] After information fusion is completed, the fusion results are repackaged into an evaluation dataset. Upon receiving the evaluation dataset, the quality evaluation process is retried.

[0085] Unlike the first round of quality verification, the evaluation dataset used in the second round of quality verification already contains dynamically supplemented information. Therefore, it is possible to re-perform verification of quality dimensions such as completeness, standardization, uniqueness, and timeliness based on more sufficient data.

[0086] For quality dimensions for which full statistical information is available, the corresponding quality indicators can be updated directly using the full statistical information, instead of relying solely on estimations based on the original sample data. For example, once the total number of records and the number of non-empty records in the complete dataset are obtained for the integrity quality dimension, the actual missing data for the corresponding fields in the complete dataset can be determined based on this statistical information.

[0087] For quality dimensions supplemented by hierarchical statistical information, the corresponding quality indicators can be redefined using the fused evaluation dataset, and the reliability level of the indicator can be reassessed based on the newly added hierarchical information.

[0088] While recalculating the quality metrics, the reliability level of each quality dimension is reassessed, and an updated quality profile is generated. Furthermore, a list of missing dimensions is regenerated based on the new quality profile.

[0089] For example, in the first round of verification, the integrity reliability level of the core business field is "low". After obtaining the total number of rows and the number of non-empty records in the complete dataset, the second round of verification can directly obtain the actual missing status of the field in the complete dataset, thereby improving the reliability level of the integrity indicator to "high".

[0090] Regarding the uniqueness quality dimension, if the first round of verification is based solely on 200 sample data points for primary key duplication detection, the reliability may be "low". After obtaining stratified statistical information in the first round, the second round of verification can use the newly added stratified information to reassess the primary key duplication situation, thereby improving the reliability of the uniqueness indicator from "low" to "medium" or other higher levels.

[0091] After completing the second round of evaluation, it is determined again whether key quality dimensions exist in the missing dimension list. If the updated missing dimension list does not contain any key quality dimensions, it indicates that the key information required for the current evaluation has met the preset reliability requirements. The dynamic supplementation process is then terminated, and a pre-evaluation report is generated based on the current quality profile.

[0092] If the updated list of missing dimensions still contains key quality dimensions, then the number of times the current supplementary instruction package has been generated is obtained and compared with the preset maximum number of generation times.

[0093] In one implementation, the maximum number of generation attempts can be set to three. The purpose of setting a maximum number of generation attempts is to prevent the supplementation process from continuing due to a lack of sufficient information for a particular quality dimension, thereby limiting the number of interactions between the third-party evaluation system and the data provider.

[0094] If the current number of supplementary instruction packets generated has not yet reached the maximum number of generation times, the dimension-complementary strategy mapping table is queried again based on the latest missing quality dimension and missing reason encoding, and the next round of supplementary requests is generated.

[0095] Since the previous round of supplementation has changed the evaluation dataset and quality profile, the next round of supplementation requests can differ from the previous round. For example, the first round might only require the total number of records and the number of non-empty records for a core field; after the first round of supplementation, if the integrity of that field has reached high reliability, but the primary key uniqueness is still at medium reliability, then the second round could request the exact number of duplicate records and the distribution of duplicate patterns in the complete dataset for the primary key field, instead of repeatedly requesting the integrity statistics already obtained.

[0096] In this way, each round of supplement requests uses the latest list of missing dimensions as input, so that the supplementary content changes dynamically as the quality assessment results change.

[0097] When the maximum number of supplementary instruction packages can be generated, even if some key quality dimensions still exist in the missing dimension list, the generation of supplementary instruction packages will stop, and a pre-assessment report will be generated based on the currently obtained quality assessment information. In this case, the pre-assessment report can record the quality dimensions that have not yet met the preset reliability requirements and their current reliability level, so that subsequent data asset auditors can understand the information support of the current pre-assessment results.

[0098] Once the conditions for outputting the pre-assessment report are met, the report generation phase begins. This triggers the pre-assessment report generation process. The quality profile generated from the final round of quality verification is retrieved, and the final assessment results are organized according to the quality dimensions.

[0099] The pre-assessment report can include the final quality score, quality indicators, reliability level, and related assessment information corresponding to quality dimensions such as completeness, standardization, uniqueness, and timeliness. Additionally, the pre-assessment report can record the number of supplementary interactions that occurred during this quality pre-assessment process, the target quality dimensions corresponding to each round of supplementary requests, the data types of supplementary requests, and the supplementary results.

[0100] For example, for the integrity quality dimension, the report can record the final integrity assessment results of the core fields and the corresponding reliability level; for the uniqueness quality dimension, it can record the primary key duplication and its reliability level; for the standardization quality dimension, it can record the field format compliance; and for the timeliness quality dimension, it can record the data freshness score and the corresponding reliability level.

[0101] When a quality dimension is updated using full statistical information, the report can further record that the quality dimension used full statistical information as the evaluation basis; when a quality dimension is supplemented using stratified statistical information, the corresponding supplementary round and stratified information type can be recorded.

[0102] Therefore, the final pre-assessment report not only reflects the quality evaluation results of data assets, but also the reliability of the evaluation results and the formation process, providing a basis for subsequent data asset registration, listing review, pricing or transaction decisions.

[0103] In one specific embodiment, the following example illustrates a manufacturing company registering a supply chain transaction dataset with a data exchange.

[0104] A manufacturing company possesses a supply chain transaction dataset containing nearly one million records. This dataset describes supply chain transaction information such as suppliers, purchase orders, purchase quantities, and transaction times. The manufacturing company plans to submit the supply chain transaction dataset as a data asset to a data exchange for registration and listing. The data exchange is equipped with the third-party evaluation system described in this embodiment.

[0105] During the data asset registration phase, manufacturing enterprises do not directly upload nearly one million complete data entries to the third-party evaluation system. Instead, in accordance with the registration requirements stipulated by the data exchange, they extract 200 records from the complete supply chain transaction dataset to form a sample data subset and generate a structured metadata description file.

[0106] The metadata description file records information such as supplier ID, purchase order ID, purchase quantity, and transaction time. It also declares that the total number of records in the dataset is approximately 1 million, the data is updated daily, and that the supplier ID and purchase order ID together constitute the primary key field combination.

[0107] The manufacturing company encapsulates the aforementioned sample data subset and metadata description file into a data asset registration package and sends it to a third-party evaluation system through the registration interface.

[0108] Upon receiving the registration package, the sample data subset and metadata description file in the registration package are parsed, and the 200 sample records are converted into sample data objects in the form of a two-dimensional table. At the same time, declaration items such as field list, field type, field meaning, total number of rows claimed, update cycle, and primary key field combination are extracted from the metadata description file to form a metadata object.

[0109] After obtaining the sample data object and metadata object, perform the first round of full-dimensional quality verification on the supply chain transaction data.

[0110] For completeness, the 200 sample records are iterated through, and null values ​​and meaningless placeholders in each field are counted. For example, the supplier number field has valid values ​​in all 200 sample records, thus a high sample completeness score is obtained.

[0111] For standardization, the format of fields such as supplier number, purchase order number, and transaction time is matched according to the field formats declared in the metadata description file. For example, the transaction time field is checked to see if it conforms to the date and time format specified in the metadata, and a standardization score is determined based on the matching results.

[0112] For uniqueness, the primary key field combination consisting of supplier ID and purchase order ID is read, and a corresponding primary key hash identifier is generated for 200 sample records. When the same primary key hash identifier exists between different data rows, it is identified as a duplicate key value, and a uniqueness index is formed based on the duplication.

[0113] Regarding timeliness, the latest timestamp of the transaction time field in the sample data is extracted and combined with the daily update cycle declared in the metadata and the current time to form a data freshness score.

[0114] Simultaneously, a reliability assessment was conducted on the aforementioned quality indicators. Since the complete dataset contains nearly one million records, while the initial sample only included 200 records, although the supplier ID field had no missing values ​​in the 200 samples, the current sample size was still relatively small compared to the complete dataset. Based on the pre-defined statistical reliability evaluation mechanism, the reliability level of this completeness indicator was determined to be "low".

[0115] Regarding primary key uniqueness, with only 200 sample records, even if no duplicate primary keys are found in the sample, it cannot fully demonstrate that there are no duplicate primary keys in nearly one million complete records. Therefore, the reliability level of this uniqueness indicator may also be determined as "low". Assuming that the timeliness indicator can form a more sufficient judgment based on the latest timestamp and daily update cycle, its reliability level can be determined as "medium". Based on this, an initial quality profile is formed and screened.

[0116] The reliability level of each quality dimension is compared with the minimum reliability threshold. Assuming that "medium" is preset as the minimum reliability threshold, the integrity and uniqueness quality dimensions are identified as missing quality dimensions because their reliability level is "low".

[0117] Furthermore, the reasons for the missing data are analyzed. The missing data for the completeness indicator can be identified as "insufficient number of sample records leading to insufficient reliability of the completeness assessment"; the missing data for the uniqueness indicator can be identified as "the current sample size is insufficient to support the uniqueness judgment of the complete dataset", and corresponding missing data reason codes are formed.

[0118] Since completeness and uniqueness are key quality dimensions of this supply chain data asset, the current information is deemed insufficient, and no pre-assessment report is output. Instead, a dynamic supplementation process is initiated.

[0119] After obtaining the list of missing dimensions, the query dimension-complement strategy mapping table is coded according to the completeness of the dimensions and their reasons for being missing. The complement strategy for missing completeness is determined to be a column-level overview statistics request, and a corresponding complement request is generated, requiring the manufacturing company to return the total number of records and the number of non-empty records in the supplier ID field of the complete dataset.

[0120] To address the issue of missing uniqueness, a corresponding supplementation strategy can be determined based on the current cause of the missing information. This can involve generating a primary key hierarchical summary request or a primary key duplication statistics request to obtain a description of the duplication status of the primary key field in the complete dataset.

[0121] After deduplication and merging of the above supplementary requests, a first round of supplementary instruction packets is generated and sent to the manufacturing enterprise through a secure interface.

[0122] After receiving the supplementary instruction package, the manufacturing company does not need to send nearly a million complete supply chain transaction data records to a third-party evaluation system. Instead, it executes the corresponding aggregation query in its local database. For integrity requests, the manufacturing company counts the total number of records and the number of non-empty records corresponding to the supplier number field. For uniqueness requests, the manufacturing company performs grouping statistics based on the primary key combination of supplier number and purchase order number, and obtains the corresponding duplicate information.

[0123] The manufacturing company encapsulates the above statistical results into a supplementary data message according to the structure specified in the supplementary instruction package, and returns it to the third-party evaluation system through a secure interface.

[0124] Upon receiving the supplementary data message, a compliance check is first performed. This involves checking if the target field corresponding to the request exists in the supplementary data message, and determining if statistical items such as the total number of rows, the number of non-null records, and primary key statistics are complete. It also verifies whether the data type and format of the statistical items conform to the requirements of the supplementary instruction package.

[0125] Once the supplementary data message passes compliance verification, information fusion begins. For the total number of rows and non-empty records in the supplier ID field, the full statistical information is directly associated with the integrity quality dimension, and this statistical information is used to update the integrity index originally formed based on 200 sample data.

[0126] For primary key statistics, the statistical results are correlated with the uniqueness quality dimension to supplement the primary key duplication information in the original sample data.

[0127] After fusion, an evaluation dataset is formed, and a second round of full-quality dimension verification is performed based on the new evaluation dataset. Since the integrity dimension has obtained the total number of rows and non-empty records of the complete dataset, this dimension no longer relies solely on 200 sample records for judgment, and its reliability level can be improved from "low" to "high".

[0128] For the uniqueness dimension, since the primary key statistics of the complete dataset have been obtained, its reliability level can be upgraded from "low" to "medium" or other levels that meet the preset conditions.

[0129] The missing dimension list is regenerated based on the new quality profile. Assuming that the completeness dimension is no longer a missing dimension, while the uniqueness dimension is still at "medium" reliability, it can be further determined whether the uniqueness has met the reliability standard required by the current key quality dimension.

[0130] If the uniqueness determination still indicates insufficient information, it is determined that the current number of times the supplementary instruction packet has been generated has not reached the maximum number of generation times. For example, if only one supplementation is performed at present, while a maximum of three supplementation operations are allowed, the second round of supplementation process is initiated.

[0131] The second round of supplementation no longer requests completeness statistics repeatedly. Instead, based on the latest list of missing dimensions, it generates more refined supplementation requests only for the uniqueness quality dimension. For example, it could request a manufacturing company to return the exact number of primary key duplicate records in the complete dataset, as well as information on the distribution of duplication patterns.

[0132] Manufacturing enterprises perform corresponding statistical operations in their local databases according to the second round of supplementary instructions and send the statistical results back to the third-party evaluation system. After completing the compliance verification of the second round of supplementary data, the third-party evaluation system integrates the statistical results into the evaluation dataset and re-executes full-quality dimension verification and reliability assessment.

[0133] After the second round of information supplementation, the reliability level of the uniqueness quality dimension is further improved. If the missing dimension list no longer contains the key quality dimensions, further supplementation will stop, and a pre-assessment report will be generated.

[0134] If critical quality dimensions still exist after the second round, but the maximum number of supplements has not yet been reached, a third round of supplement instructions can be generated. The third round of supplement instructions is only generated for quality dimensions that have not yet met reliability requirements, avoiding duplicate requests for quality information that has already met reliability requirements.

[0135] Once the third round of verification is completed, if all key quality dimensions meet the preset minimum reliability requirements, the dynamic supplementation process will be terminated, and a final pre-assessment report will be generated based on the quality profile formed by the last quality verification.

[0136] The final pre-assessment report can record the final assessment scores and reliability levels corresponding to quality dimensions such as completeness, standardization, uniqueness, and timeliness. It also records the actual supplementary rounds that occurred during the assessment process, as well as the supplementary data types acquired in each round. For example, the report can record that the first round acquired full statistical information of core business fields and primary key hierarchical statistical information, while the second round acquired statistical information on duplicate primary key records.

[0137] If, after reaching the maximum number of supplementary instruction packages generated, there are still key quality dimensions that do not meet reliability requirements, then further supplementation will be stopped, and a pre-assessment report will be output based on the currently formed quality profile. At the same time, the report will mark the reliability deficiencies of the corresponding quality dimensions so that subsequent auditors can understand the applicability of the assessment results.

[0138] Based on another preferred embodiment described above, see [link to preferred embodiment]. Figure 2 As shown, this embodiment provides a rapid quality verification and dynamic supplementation system for data asset pre-assessment, used to apply the above-mentioned rapid quality verification and dynamic supplementation method for data asset pre-assessment, including: The registration information receiving module is configured to acquire the data asset registration package and parse it to generate sample data objects and metadata objects; The initial quality verification module is configured to perform full-dimensional quality verification on sample data objects and metadata objects, and generate an initial quality profile. The missing dimension identification module is configured to screen the initial quality profile based on the lowest reliability threshold and generate a list of missing dimensions based on the screening results. The iterative evaluation control module is configured to determine whether to output a pre-evaluation report based on the list of missing dimensions. The supplementary strategy and request generation module is configured to, when it is determined that no pre-evaluation report should be output, match the supplementary request template from the dimension-complementary strategy mapping table according to the missing dimension list, perform deduplication and merging, generate a supplementary instruction package, and send the supplementary instruction package to the data provider; The supplementary data receiving and fusion module is configured to receive supplementary data messages returned by the data provider, perform compliance verification on the supplementary data messages, and merge and update the supplementary data messages that pass the compliance verification with the sample data objects and metadata objects to generate an evaluation dataset. The iterative evaluation control module is also configured to repeatedly perform full-quality dimension verification, screening, and judgment based on the evaluation dataset until a pre-evaluation report is output.

[0139] In summary, the pre-assessment process incorporates automatic identification of missing dimensions and a dynamic supplementation loop for minimum necessary information. This allows the assessment to proactively transcend the boundaries of initial information, legally and lightweightly extracting key statistical features or representative samples from data providers through precise questioning. The resulting pre-assessment report includes highly reliable confidence support for each quality dimension's score, eliminating extreme misjudgments caused by coincidence or deliberate selection of a single sample. This enhances the comprehensiveness, objectivity, and credibility of the pre-assessment results with minimal increase in the burden on data providers and privacy risks.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for rapid quality verification and dynamic supplementation of data asset pre-assessment, characterized in that, include: Obtain the data asset registration package, parse and generate sample data objects and metadata objects; Perform full-dimensional quality verification on the sample data object and metadata object to generate an initial quality profile; The initial quality profile is screened based on the minimum reliability threshold, and a list of missing dimensions is generated based on the screening results. Determine whether to output a pre-assessment report based on the missing dimension list; When it is determined that no pre-evaluation report should be output, a supplementary request template is matched from the dimension-complementary strategy mapping table according to the missing dimension list, and deduplication and merging are performed to generate a supplementary instruction package; and the supplementary instruction package is sent to the data provider. Receive supplementary data messages returned by the data provider, perform compliance verification on the supplementary data messages, and merge and update the supplementary data messages that have passed the compliance verification with the sample data object and metadata object to generate an evaluation dataset; Based on the evaluation dataset, the full quality dimension verification, screening and judgment are repeatedly performed until the pre-evaluation report is output; Until the output of the aforementioned pre-assessment report, including: Based on the evaluation dataset, a full quality dimension verification and screening is performed again to generate an updated list of missing dimensions; when the updated list of missing dimensions does not contain any key quality dimensions, a pre-evaluation report is output. Alternatively, when the number of times the supplementary instruction package is generated reaches the maximum number of times, dynamic supplementation stops, and the pre-evaluation report is generated based on the current quality profile.

2. The method for rapid quality verification and dynamic supplementation of data asset pre-assessment according to claim 1, characterized in that, When parsing and generating sample data objects and metadata objects, the following are included: The data asset registration package includes a sample data subset and a metadata description file; the sample data subset is transformed into a two-dimensional table to generate a sample data object, and the metadata description file is used to extract declaration items to generate a metadata object.

3. The method for rapid quality verification and dynamic supplementation of data asset pre-assessment according to claim 2, characterized in that, When performing full-dimensional quality verification on the sample data object and metadata object, the following are included: The sample data object is traversed, and the frequency of null values ​​or meaningless placeholders is counted to determine the missing rate of each field. According to the format constraints of the metadata object, the compliance ratio of the sample field values ​​is matched to obtain the standardization score. According to the primary key field combination of the metadata object, a hash comparison is performed between rows of the sample data object to identify duplicate key values ​​and output the duplication rate. Based on the timestamp of the sample data object and the update cycle and current time of the metadata object, the data freshness score is determined.

4. The method for rapid quality verification and dynamic supplementation of data asset pre-assessment according to claim 3, characterized in that, When performing full-dimensional quality verification on the sample data object and metadata object, the following is also included: While performing the full quality dimension verification, the reliability level of each quality dimension is evaluated.

5. The method for rapid quality verification and dynamic supplementation of data asset pre-assessment according to claim 4, characterized in that, When screening the initial quality profile based on a minimum reliability threshold and generating a list of missing dimensions based on the screening results, the following are included: The reliability level of each quality dimension is compared with the minimum reliability threshold. The quality dimensions whose reliability level is less than the minimum reliability threshold are identified as missing quality dimensions. The missing cause code is determined according to the evaluation index of the missing quality dimension. Based on the missing quality dimension and the missing cause code, the missing dimension list is generated.

6. The method for rapid quality verification and dynamic supplementation of data asset pre-assessment according to claim 5, characterized in that, When determining whether to output a pre-assessment report based on the missing dimension list, the following are included: Determine whether the missing dimension list contains a key quality dimension; if the missing dimension list does not contain the key quality dimension, determine whether to output a pre-assessment report; otherwise, determine whether to output the pre-assessment report.

7. The method for rapid quality verification and dynamic supplementation of data asset pre-assessment according to claim 6, characterized in that, Based on the missing dimension list, a supplementation request template is matched from the dimension-complementation strategy mapping table, and deduplication and merging are performed to generate a supplementation instruction package, including: Based on the missing quality dimension and missing cause encoding, the complement strategy and complement request template are matched in the dimension-complement strategy mapping table; duplicate request items are identified and merged in the matched complement request templates to generate a complement instruction package; the complement instruction package includes the target quality dimension, the complement data type, and the complement data requirements.

8. The method for rapid quality verification and dynamic supplementation of data asset pre-assessment according to claim 7, characterized in that, When generating the evaluation dataset, the following are included: The data types, data fields, statistical items, and data formats in the supplementary data message are matched and verified to determine whether the supplementary data message matches the supplementary data requirements in the supplementary instruction packet. When a match is determined, information fusion is performed based on the supplementary data message. The full statistical information is associated with the corresponding quality dimension, and the hierarchical statistical information is fused with the sample data object to generate the evaluation dataset.

9. A rapid quality verification and dynamic supplementation system for data asset pre-assessment, used to apply the rapid quality verification and dynamic supplementation method for data asset pre-assessment as described in any one of claims 1-8, characterized in that, include: The registration information receiving module is configured to acquire the data asset registration package and parse it to generate sample data objects and metadata objects; The initial quality verification module is configured to perform full-dimensional quality verification on the sample data object and metadata object, and generate an initial quality profile. The missing dimension identification module is configured to screen the initial quality profile based on a minimum reliability threshold and generate a list of missing dimensions based on the screening results. The iterative evaluation control module is configured to determine whether to output a pre-evaluation report based on the list of missing dimensions. The supplementary strategy and request generation module is configured to, when it is determined that no pre-evaluation report should be output, match a supplementary request template from the dimension-complementary strategy mapping table according to the missing dimension list, perform deduplication and merging, generate a supplementary instruction package, and send the supplementary instruction package to the data provider; The supplementary data receiving and fusion module is configured to receive supplementary data messages returned by the data provider, perform compliance verification on the supplementary data messages, and perform information fusion and update the supplementary data messages that have passed the compliance verification with the sample data object and metadata object to generate an evaluation dataset. The iterative evaluation control module is also configured to repeatedly perform full-quality dimension verification, screening, and judgment based on the evaluation dataset until the pre-evaluation report is output.

Images