Intelligent evaluation system and method for data quality based on large model

Abstract

The application provides a large model-based data quality intelligent evaluation system and method, relates to the technical field of data processing, and the method comprises the following steps: dividing data fields of basic data into several levels, constructing a hierarchical description vector for each data field, counting the common occurrence times of value sequences of each data field in the same level, updating the semantics of each data field through multi-layer back propagation of a large model, adjusting the field embedding weight, calculating the coordination degree of each data field in the same level, obtaining a hierarchical harmony factor, calculating the stability degree of the semantic relationship of each data field in the layer under the disturbance condition, obtaining a semantic drift factor, constructing a fusion matrix, distributing the weight of each hierarchical data field in the fusion matrix and extracting the principal component space, calculating the deviation rate of each data field in the principal component space in the fusion matrix, and obtaining a data quality evaluation result. The application realizes accurate evaluation of data quality through a large model.

Images