AI-based data full life cycle quality evaluation method and system

Abstract

This invention relates to the field of AI-based data quality assessment technology, specifically to an AI-based method and system for assessing data quality throughout its entire lifecycle. The method includes: acquiring multi-stage data samples throughout the data lifecycle; decoupling data content and metadata structure; separating the core content field from the process trajectory field; automatically constructing a multi-dimensional quality feature space in the core content field; extracting native feature points and mapping them into the space; tracking data transfer events in the process trajectory field to generate dynamic transfer trajectories; injecting the process information carried by the trajectory back into the feature space; applying process constraint offsets to the native feature points to form quality state points; and finally generating a data quality assessment value. This solution achieves dynamic quality representation throughout the data lifecycle through field decoupling and process constraint perturbation, making the assessment results more closely reflect the actual data flow state.

Description

Technical Field

[0001] This invention relates to the field of AI-based data quality assessment technology, and in particular to an AI-based method and system for assessing the quality of data throughout its entire lifecycle. Background Technology

[0002] Existing AI-based data lifecycle quality assessment methods mostly focus on extracting and analyzing quality features from the data itself. The accompanying metadata is only used as simple traceability aids, without any dedicated structural decoupling of data content and metadata. Data transfer behavior at each stage of the lifecycle is recorded only as basic log information. Process parameters such as timestamps, operation entities, and interface types are only used for flow recording and are not integrated into the data quality feature construction process. Data content analysis and lifecycle process recording use independent processing logics, and the assessment process focuses only on the static content features of the data.

[0003] Current assessment methods rely solely on the intrinsic features of the core data content to construct a quality feature space, failing to integrate the dynamic data flow process with static content features. The state transitions of data at different stages are not reflected in the quality feature system. The intrinsic feature points are determined only by the data content itself; information about the flow process cannot constrain the feature parameters within the feature space, resulting in quality assessment results that cannot match the dynamic changes throughout the data's entire lifecycle. It is necessary to structurally decouple the data content and metadata and delineate them into independent fields. Simultaneously, the process information carried by the dynamic transfer trajectory should be applied inversely to the multi-dimensional quality feature space, imposing process constraint perturbations on the intrinsic feature points. This will create a quality state representation method that adapts to the entire data lifecycle. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an AI-based method and system for quality assessment of the entire data lifecycle.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an AI-based data lifecycle quality assessment method, comprising: Acquire data samples from multiple stages throughout the entire data lifecycle, wherein the data samples contain data content and its accompanying metadata; The data content and its accompanying metadata are structurally decoupled to separate the core content field of the data from the process trajectory field describing the flow of the core content field. In the core content field, a multi-dimensional quality feature space is automatically constructed based on the data type. The original feature points corresponding to each dimension of quality features are extracted from the core content field, and the original feature points are mapped to the multi-dimensional quality feature space. In the process trajectory field, the transfer events of data are tracked between each stage of the life cycle. Based on the timestamp of the transfer event, the operating subject and the interface type, a dynamic transfer trajectory representing the data state transition is generated in the process trajectory field. The process information carried by the dynamic transfer trajectory is injected into the multidimensional quality feature space in reverse. Process constraint perturbation is applied to the original feature points of the corresponding stage in the multidimensional quality feature space, so that the coordinates of the original feature points shift with the change of process information, forming quality state points constrained by the process. Based on the process-constrained quality state points, a data quality assessment value is generated.

[0006] As a further aspect of the present invention, the automatic construction of a multi-dimensional quality feature space based on data type within the core content field includes: Identify the data structure of data samples in the core content field, wherein the data structure includes structured tables, unstructured text, semi-structured logs, and binary streams; For each data structure type, a set of basic quality dimensions are preset, including completeness, consistency, timeliness and uniqueness; Based on the basic quality dimensions, and considering the specific content of the data samples, derivative quality dimensions related to the business scenarios are extended through pattern matching. The basic quality dimension and the derived quality dimension are combined to form an abstract mathematical space with a variable number of dimensions. This abstract mathematical space is a multidimensional quality feature space used to quantitatively represent the quality status of data samples.

[0007] As a further aspect of the present invention, the step of extracting native feature points corresponding to each dimension of quality features from the core content field and mapping the native feature points to a multidimensional quality feature space includes: For each quality dimension in the multidimensional quality feature space, define its value calculation logic; According to the value calculation logic, the data samples in the core content field are calculated to obtain the specific quantitative values ​​of the data samples in each quality dimension. The quantified values ​​of each quality dimension are combined into an ordered numerical vector, and the endpoints of the ordered numerical vector are located in the multi-dimensional quality feature space. The endpoints are the original feature points that characterize the original quality features of the data sample.

[0008] As a further aspect of the present invention, the tracking data transfer events between different stages of its lifecycle generate dynamic transfer trajectories representing data state transitions in the process trajectory field based on the timestamps, operating entities, and interface types of the transfer events, including: In the process trajectory field, a directed process graph is constructed with life cycle stages as nodes and data flow between stages as edges; Monitor each instance of data flow from one lifecycle stage node to the next, record the timestamp of the event, the identifier of the entity that triggered the event, and the interface type used for data outflow and inflow. This recorded information constitutes a transfer event. In the directed process graph, starting from the data generation stage node, the life cycle stage nodes are connected sequentially according to the sequence of transfer events to form a path that runs through the directed process graph. The path is a dynamic transfer trajectory with attached time, subject and interface type information.

[0009] As a further aspect of the present invention, the process information carried by the dynamic transfer trajectory is injected in reverse into the multidimensional quality feature space, and process constraint perturbations are applied to the original feature points of the corresponding stages in the multidimensional quality feature space, including: From the dynamic transfer trajectory, the current life cycle stage of the data sample is extracted, as well as the sequence of operation subjects and interface types of all transfer events experienced to reach the stage; The operation subject sequence and interface type sequence are input into a pre-trained perturbation evaluation model, and the perturbation evaluation model outputs the correction amount for each dimension coordinate of the multidimensional quality feature space. The correction amount is superimposed on the coordinates of the original feature points corresponding to the data samples, so that the original feature points are displaced along each dimension of the multidimensional quality feature space. The points after the displacement are the new coordinates that have been integrated with the process constraint information. The points corresponding to the new coordinates are the quality state points constrained by the process.

[0010] As a further aspect of the present invention, a data quality assessment value is generated based on the process-constrained quality state point, including: Field modeling is performed on all process-constrained quality state points in the multidimensional quality feature space to generate a probabilistic quality field that can cover all quality state points. The field strength at any point in the probabilistic quality field represents the probability density of occurrence of the quality feature combination corresponding to the quality state point throughout the entire data lifecycle. Based on the probabilistic mass field, a dynamically evolving mass contour surface is constructed, which divides the probabilistic mass field into high-mass regions and low-mass regions. The process-constrained quality state point corresponding to the newly entered data sample in the multi-dimensional quality feature space is calculated in real time, and the positional relationship of the quality state point relative to the quality contour surface is calculated to generate the original quality deviation. The original quality deviation is corrected based on the stage weight of the life cycle stage. The stage weight is dynamically adjusted according to the stage of the data's life cycle to generate a data quality assessment value.

[0011] As a further aspect of the present invention, the step of performing field modeling on all process-constrained quality state points in the multidimensional quality feature space to generate a probabilistic mass field that can cover all quality state points specifically includes: Collect all process-constrained quality state points generated at each stage of the entire lifecycle from historical data to form a training point set; Using the training point set, a probability field model based on kernel density estimation is trained. The probability field model outputs the probability density estimate of the training point set near the input coordinate point. The continuous probability density distribution defined by the probability field model on the complete multidimensional mass feature space is the probability mass field, and the probability density value at any position in the probability mass field is the field strength at that position. Based on the probabilistic mass field, a dynamically evolving mass contour surface is constructed, specifically including: In a probability mass field, a probability density threshold is set; Find all points in the probability mass field whose probability density values ​​are equal to the probability density threshold, and the set of these points forms an isosurface; As the training point set is updated over time, the probability mass field is updated accordingly, and the shape and position of the isosurface also change dynamically. The dynamically changing isosurface is the mass contour surface that divides the probability mass field into high-quality, high-probability-density regions and low-quality, low-probability-density regions.

[0012] As a further aspect of the present invention, the real-time calculation of the process-constrained quality state point corresponding to the newly entered data sample in the multi-dimensional quality feature space, and the calculation of the positional relationship of the quality state point relative to the quality contour surface to generate the original quality deviation, includes: For newly entered data samples, calculate the coordinates of the corresponding process-constrained quality state points in the multidimensional quality feature space; Calculate the shortest geometric distance from the coordinates of the process-constrained mass state point to the current mass contour surface; The shortest geometric distance is used as the original mass deviation. When the mass state point is located on one side of the high-quality region divided by the mass contour surface, the original mass deviation is negative, and when it is located on the side of the low-quality region, it is positive.

[0013] As a further aspect of the present invention, the original quality deviation is corrected based on the stage weight of the lifecycle stage. The stage weight is dynamically adjusted according to the stage of the data's lifecycle to generate a data quality assessment value, including: Identify the specific lifecycle stage of the new data sample; Query the preset weight configuration table to obtain the quality impact weight coefficient corresponding to the specific life cycle stage; Multiply the original quality deviation by the quality impact weighting coefficient to obtain the data quality assessment value after stage importance weighting.

[0014] As a further aspect of the present invention, the present invention also includes an AI-based data lifecycle quality assessment system, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the steps of the AI-based data lifecycle quality assessment method described above.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The data content and its accompanying metadata are structurally decoupled, separating the core content field that carries the information of the data itself from the process trajectory field that describes the flow of the core content. The core content information and the life cycle flow process information are divided into different independent fields, and the representation logic of the two types of information is distinguished from each other. The inherent attributes of the data ontology and the process flow attributes are clearly separated, and the information composition within the entire life cycle of the data is refined and sorted out. The original content information and the process-derived information no longer have mixed representations. The quality feature extraction can accurately match the information types corresponding to the two fields, and the feature extraction is more closely matched with the data information.

[0016] Based on the timestamp of the data transfer event, the operating entity, and the interface type, a dynamic transfer trajectory is generated in the process trajectory field. The process information carried by the dynamic transfer trajectory is injected back into the multidimensional quality feature space. The process information applies process constraint perturbation to the original feature points corresponding to the life cycle stage in the multidimensional quality feature space. The coordinates of the original feature points will shift accordingly with the changes in process information. The quality state points formed after the shift can simultaneously carry dual information of data content and transfer process. The static original quality features and dynamic process information form a direct parameter correlation. The feature parameters of the quality feature space can be adjusted synchronously with the data transfer event. The quality state points can fully reflect the actual state of the data in each stage of the entire life cycle. The data quality assessment value is generated based on the feature parameters of the fused process constraints. Attached Figure Description

[0017] Figure 1This is a flowchart of the AI-based data lifecycle quality assessment method described in this invention; Figure 2 A flowchart for generating dynamic transfer trajectories; Figure 3 A data lifecycle quality status trajectory map; Figure 4 A probabilistic mass field and contour map for multidimensional mass feature space; Figure 5 This is a comparison chart of data quality assessment values ​​after stage weighting. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] See Figure 1This invention provides an AI-based method for quality assessment of the entire data lifecycle. The method includes: acquiring data samples from multiple stages throughout the data lifecycle, each data sample containing data content and accompanying metadata; structurally decoupling the data content and accompanying metadata to separate a core content field and a process trajectory field describing the flow of the core content field; automatically constructing a multi-dimensional quality feature space based on data type within the core content field; extracting native feature points corresponding to each dimension of quality features from the core content field; and mapping these native feature points to the multi-dimensional quality feature space; tracking data transfer events between different stages of the lifecycle within the process trajectory field; and generating dynamic transfer trajectories representing data state transitions based on the timestamps, operating entities, and interface types of the transfer events; injecting the process information carried by the dynamic transfer trajectories back into the multi-dimensional quality feature space; applying process constraint perturbations to the native feature points of the corresponding stages in the multi-dimensional quality feature space, causing the coordinates of the native feature points to shift with changes in process information, forming process-constrained quality state points; and generating data quality assessment values ​​based on the process-constrained quality state points.

[0021] In one embodiment of the present invention, the data structure of data samples in the core content field is identified. The data structure includes structured tables, unstructured text, semi-structured logs, and binary streams. For each data structure type, a set of basic quality dimensions is preset, including completeness, consistency, timeliness, and uniqueness. Based on the specific content of the data sample, derived quality dimensions related to the business scenario are extended using pattern matching. The basic quality dimensions and derived quality dimensions are combined to form an abstract mathematical space with a variable number of dimensions. This abstract mathematical space is a multi-dimensional quality feature space used to quantitatively characterize the quality status of data samples.

[0022] For each quality dimension in the multidimensional quality feature space, a value calculation logic is defined. Following this logic, data samples in the core content field are processed to obtain specific quantized values ​​for each quality dimension. The quantized values ​​of each quality dimension are combined into an ordered numerical vector, and the endpoints of this ordered numerical vector are located in the multidimensional quality feature space. These endpoints are the original feature points representing the original quality characteristics of the data sample.

[0023] In practical implementation, the data structures of data samples in the core content field include structured tables, unstructured text, semi-structured logs, and binary streams. A set of basic quality dimensions is preset for each data structure type, including completeness, consistency, timeliness, and uniqueness. Based on the basic quality dimensions, derivative quality dimensions related to the business scenario are extended through pattern matching for the specific content of the data sample. The basic and derivative quality dimensions are combined to form an abstract mathematical space with a variable number of dimensions. This abstract mathematical space is the multi-dimensional quality feature space used to quantitatively characterize the quality status of the data sample. In some embodiments, the multi-dimensional quality feature space of structured table data samples includes completeness, consistency, timeliness, uniqueness, and a derivative quality dimension of "transaction amount verification"; the multi-dimensional quality feature space of unstructured text data samples includes completeness, consistency, timeliness, and a derivative quality dimension of "sentiment polarity consistency"; the multi-dimensional quality feature space of semi-structured log data samples includes completeness, consistency, timeliness, and a derivative quality dimension of "log level compliance"; and the multi-dimensional quality feature space of binary stream data samples includes the basic quality dimensions of completeness and consistency.

[0024] For each quality dimension in the multidimensional quality feature space, a calculation logic for the value of each quality dimension is defined. Following this logic, data samples in the core content field are processed to obtain specific quantified values ​​for each quality dimension. In specific implementation, for the integrity quality dimension, the calculation logic is based on the proportion of non-empty fields, and the quantified values ​​for the integrity dimension are... Calculated using the following formula: in: This indicates the number of non-empty fields in the data sample. This represents the total number of fields in the data sample. It can be understood that the calculation logic for the consistency quality dimension is based on the proportion of field values ​​conforming to predefined rules, the calculation logic for the timeliness quality dimension is based on the difference between the data timestamp and the current time, and the calculation logic for the uniqueness quality dimension is based on the proportion of duplicate records. The quantified values ​​of each quality dimension are combined into an ordered numerical vector, and the endpoints of this ordered numerical vector are located in a multi-dimensional quality feature space. These endpoints are the original feature points representing the original quality characteristics of the data sample. Optionally, in data comparison scenarios, the original feature points of structured tabular data samples and semi-structured log data samples are mapped to different coordinate positions in the multi-dimensional quality feature space, reflecting the differences in quality characteristics caused by differences in data structure.

[0025] In practical implementation, extending derived quality dimensions through pattern matching involves analyzing business rule patterns within the data sample content. For example, for structured tabular data samples of financial transactions, pattern matching identifies the business rule of "non-negative transaction amount" and generates the derived quality dimension of "amount validity." In some embodiments, for unstructured text data samples, pattern matching identifies the business rule of "text language type" and generates the derived quality dimension of "language consistency." It can be understood that the number of dimensions in the multidimensional quality feature space dynamically changes with the identification of derived quality dimensions. The dimensions of the ordered numerical vector correspond one-to-one with the dimensions of the multidimensional quality feature space, and the coordinate positions of the original feature points are determined by the quantized values ​​of each dimension in the ordered numerical vector.

[0026] In one embodiment of the present invention, see [reference] Figure 2 In the process trajectory field, a directed process graph is constructed with lifecycle stages as nodes and data flow between stages as edges. Each data flow from one lifecycle stage node to the next is monitored, recording the timestamp of the event, the identifier of the triggering entity, and the interface type used for data inflow and outflow. This recorded information constitutes a transfer event. In the directed process graph, starting from the data generation stage node, the lifecycle stage nodes are sequentially connected according to the sequence of transfer events, forming a path that traverses the directed process graph. This path is the dynamic transfer trajectory with attached time, entity, and interface type information.

[0027] In practical implementation, a directed process graph is constructed in the process trajectory field, with lifecycle stages as nodes and data flow between stages as edges. The nodes of the directed process graph include "data creation," "data validation," "data storage," "data usage," and "data archiving," and the edges represent the direction of data flow from upstream nodes to downstream nodes. Each data flow from one lifecycle stage node to the next is monitored, recording the timestamp of the action, the identifier of the triggering entity, and the interface type used for data inflow and outflow. This recorded information constitutes a transfer event. In some embodiments, the entity identifier is "User Service Module" or "ETL Job - Number 123," and the interface type is "RESTful API" or "Message Queue Kafka Topic."

[0028] In practical implementation, in a directed process graph, starting from the data generation stage node, the nodes of the lifecycle stages are sequentially connected according to the sequence of transition events to form a path that runs through the directed process graph. This path is a dynamic transition trajectory with attached time, subject, and interface type information. In some embodiments, for the "user registration" data sample, the node sequence of the dynamic transition trajectory is "data creation" -> "data verification" -> "data storage", and the sequence of transition events attached to the edges of the dynamic transition trajectory is [(t1,S1,I1),(t2,S2,I2)]. It can be understood that for the "order transaction" data sample, the node sequence of the dynamic transition trajectory may be "data creation" -> "data verification" -> "data storage" -> "data usage", and the sequence of transition events attached to the edges of the dynamic transition trajectory is [(t3,S3,I3),(t4,S4,I4),(t5,S5,I5)]. By comparing the differences in the length of the dynamic transition trajectory node sequence and the content of the transition event sequence for different data samples, the differences in the data flow path can be reflected.

[0029] In practice, the timestamp of the recorded action is used to calculate the inter-stage processing delay. The processing delay is calculated by the difference in timestamps of consecutive transfer events, and the formula for calculating the difference is as follows: in: This indicates the timestamp when data flows out from the current stage node. This indicates the timestamp of the data reaching the current stage node. Optionally, in a data comparison scenario, the timestamp of the "user registration" data sample from the "data creation" node to the "data verification" node... Between the same nodes and the "order transaction" data samples The numerical values ​​differ, and this difference is recorded in their respective dynamic transfer trajectory information. It can be understood that the dynamic transfer trajectory fully records the sequence of nodes the data passes through during the transfer process, as well as the timestamp, operation entity identifier, and interface type corresponding to each transfer step.

[0030] In one embodiment of the present invention, the current lifecycle stage of a data sample, as well as the sequence of operational entities and interface types of all transfer events experienced to reach the stage, are parsed from the dynamic transfer trajectory. The sequence of operational entities and interface types is input into a pre-trained perturbation evaluation model, which outputs corrections to the coordinates of each dimension of the multidimensional quality feature space. These corrections are then superimposed on the coordinates of the original feature point corresponding to the data sample, causing the original feature point to shift along each dimension of the multidimensional quality feature space. The shifted point becomes the new coordinates that incorporate process constraint information, and the point corresponding to these new coordinates is the process-constrained quality state point.

[0031] Field modeling is performed on all process-constrained quality state points in the multidimensional quality feature space to generate a probabilistic quality field that covers all quality state points. The field strength at any point in the probabilistic quality field represents the probability density of the occurrence of the quality feature combination corresponding to that quality state point throughout the entire data lifecycle. Based on the probabilistic quality field, a dynamically evolving quality contour surface is constructed, which divides the probabilistic quality field into high-quality and low-quality regions. The process-constrained quality state points corresponding to newly entering data samples in the multidimensional quality feature space are calculated in real time, and the positional relationship of the quality state points relative to the quality contour surface is calculated to generate the original quality deviation. The original quality deviation is corrected based on the stage weight of the lifecycle stage, which is dynamically adjusted according to the stage of the data's lifecycle to generate a data quality assessment value.

[0032] In practice, the current lifecycle stage of the data sample and the sequence of operation subjects and interface types of all transfer events experienced to reach the lifecycle stage are parsed from the dynamic transfer trajectory. The sequence of operation subjects and interface types is input into a pre-trained perturbation evaluation model. The perturbation evaluation model outputs the correction amount for each dimension of the multidimensional quality feature space. The correction amount is superimposed on the coordinates of the original feature point corresponding to the data sample, so that the original feature point is displaced along each dimension of the multidimensional quality feature space. The point after displacement is the new coordinate that integrates process constraint information, and the point corresponding to the new coordinate is the quality state point constrained by the process.

[0033] In practical implementation, a probabilistic mass field is generated by performing field modeling on all process-constrained quality state points in the multidimensional quality feature space. This probabilistic mass field covers all quality state points. The field strength at any point in the probabilistic mass field represents the probability density of the occurrence of the corresponding quality feature combination throughout the entire data lifecycle. It can be understood that the training point set used for field modeling includes all process-constrained quality state points generated from "user registration," "order transactions," and historical data. The probabilistic mass field forms a continuous probability density distribution across the entire multidimensional quality feature space. Based on the probabilistic mass field, a dynamically evolving quality contour surface is constructed, dividing the probabilistic mass field into high-quality and low-quality regions. In practical implementation, the construction of the quality contour surface relies on a set probability density threshold, which divides the probabilistic mass field into high-probability-density and low-probability-density regions. The high-probability-density region corresponds to the high-quality region. In some embodiments, the "user registration" business and the "order transaction" business have different data characteristics and flow processes, and the clustering areas of their process-constrained quality state points in the multidimensional quality feature space are different. As a result, the probability quality field and quality contour surface constructed based on the entire historical point set can simultaneously cover the quality distribution characteristics of the two business models.

[0034] The system calculates the process-constrained quality state points corresponding to newly entered data samples in the multidimensional quality feature space in real time, and calculates the positional relationship of the quality state points relative to the quality contour surface to generate the original quality deviation. The original quality deviation is then corrected based on lifecycle stage weights, which are dynamically adjusted according to the data's lifecycle stage, to generate a data quality assessment value. Optionally, the system calculates the shortest geometric distance from the quality state point to the quality contour surface. As the original quality deviation, among which This represents the minimum Euclidean distance from the coordinates of a quality state point to the quality contour surface in multidimensional space. It can be understood that when the quality state point of a newly entered "user registration" data sample lies within the high-quality region defined by the quality contour surface, its... A negative value indicates that the combination of quality characteristics is in a high probability density region. In practice, adjusting the stage weights of the original quality deviation involves identifying the specific lifecycle stage of the new data sample; for example, the "data validation" stage has a higher weighting coefficient for quality impact. The "data archiving" stage has a lower quality impact weighting coefficient. The original quality deviation The data quality assessment value is obtained by multiplying it by the corresponding quality impact weighting coefficient. .

[0035] See Figure 3 This is a data lifecycle quality status trajectory chart, intuitively reflecting the quality change trend of data during its flow. Data acquisition → data storage: quality declines (88→82), possibly due to data loss or format conversion during storage. Data storage → data verification: quality significantly improves (82→95), verification operations effectively correct quality issues. Data verification → data use: quality declines again (95→86), operations during use may introduce new quality disturbances. Data use → data archiving: quality further declines (86→78), long-term storage during the archiving stage may lead to data integrity and timeliness degradation. The rules and automation capabilities of data verification should be continuously strengthened to maximize its quality repair role. To address data loss during storage and quality degradation after archiving, regular quality inspections and integrity verification mechanisms can be introduced.

[0036] In one embodiment of the present invention, all process-constrained quality state points generated at each stage of the entire lifecycle are collected from historical data to form a training point set. Using this training point set, a probability field model based on kernel density estimation is trained. The probability field model outputs the probability density estimate of the training point set near any input coordinate point. The continuous probability density distribution defined by the probability field model on the complete multidimensional quality feature space is the probability mass field, and the probability density value at any location in the probability mass field is the field strength at that location. A probability density threshold is set in the probability mass field. All points in the probability mass field whose probability density values ​​are equal to the probability density threshold are found, and the set of these points constitutes an isosurface. Since the training point set is updated over time, the probability mass field is updated accordingly, and the shape and position of the isosurface also change dynamically. The dynamically changing isosurface is the quality contour surface that divides the probability mass field into high-quality, high-probability-density regions and low-quality, low-probability-density regions.

[0037] In practical implementation, a training point set is formed by collecting all process-constrained quality state points generated at each stage of the historical data lifecycle. This training point set includes quality state points of "user registration" data samples during the "data creation," "data verification," and "data storage" stages, as well as quality state points of "order transaction" data samples during the "data creation," "data verification," "data storage," and "data usage" stages. A probability field model based on kernel density estimation is trained using this training point set. The probability field model outputs a probability density estimate of the training point set near any input coordinate point. In some embodiments, the kernel density estimate of the probability field model is calculated using the following formula: in: This indicates that the probability field model is at the input coordinate points. The probability density estimate output at that point, This represents the total number of training point clusters. Indicates bandwidth parameter, The dimension represents the multidimensional quality feature space. Represents the kernel function. Represents the first training point in the training point set. A point. The continuous probability density distribution defined by the probability field model on the complete multidimensional quality feature space is the probability quality field, and the probability density value at any position in the probability quality field is the field strength at that position. In specific implementation, a training point set containing historical quality status points of "user registration" and "order transaction" is used to train the probability field model. The probability quality field learned by the probability field model can reflect the joint probability distribution of the two types of business data in dimensions such as completeness, consistency, and timeliness.

[0038] In practice, refer to Table 1, which shows a simplified example of a training point set, containing the coordinates of some process-constrained quality state points and their corresponding probability density estimates from historical data samples of "user registration" and "order transactions".

[0039] Table 1: Example Table of Training Point Sets In a probabilistic mass field, a probability density threshold is set, and all points in the probabilistic mass field whose probability density values ​​are equal to the probability density threshold are found. The set of points constitutes an isosurface. In some embodiments, the probability density threshold is set to the median of the field strength distribution of the training point set in the probabilistic mass field. Since the training point set is updated over time, the probabilistic mass field is updated accordingly, and the shape and position of the isosurface also change dynamically. The dynamically changing isosurface is the mass contour surface that divides the probabilistic mass field into high-quality, high-probability-density regions and low-quality, low-probability-density regions. It can be understood that as new "user registration" and "order transaction" data samples are continuously generated and transformed into quality state points added to the training point set, the probabilistic field model will recalculate the probability density distribution. The shape of the isosurface with a probability density threshold of 0.1 in three-dimensional coordinate space will evolve from an ellipsoid to a more complex curved surface. Optionally, in a data comparison scenario, the probabilistic mass field trained based on historical "user registration" data and the probabilistic mass field trained based on historical "order transaction" data have different probability density distributions, resulting in differences in the position and shape of the mass contour surface generated based on the same probability density threshold in space.

[0040] See Figure 4 This is a multidimensional quality feature space probability quality field and contour plot. The probability density peak is located in the region where integrity ≈ 0.925 and consistency ≈ 0.960, corresponding to the bright yellow core area. The contour lines are elliptical, indicating a positive correlation between integrity and consistency, with the data points being most densely concentrated in the peak region. This field is generated by estimating the kernel density of historical process-constrained quality state points. A higher field strength indicates that the quality combination is more common in actual data flow. The inner contour lines (bright yellow area) have the highest probability density, representing the typical high-quality data feature distribution; the outer contour lines (dark purple area) have a probability density close to 0, representing the abnormal or low-quality data feature distribution. The contour lines divide the space into "high-quality, high-probability regions" and "low-quality, low-probability regions," which can be used to determine the quality deviation of new data samples in real time.

[0041] In one embodiment of the present invention, for a newly entered data sample, the coordinates of its corresponding process-constrained quality state point in the multidimensional quality feature space are calculated. The shortest geometric distance from the coordinates of the process-constrained quality state point to the current quality contour surface is calculated. This shortest geometric distance is used as the original quality deviation, wherein the original quality deviation is negative when the quality state point is located on one side of the high-quality region defined by the quality contour surface, and positive when it is located on the side of the low-quality region. The specific lifecycle stage of the new data sample is identified. A preset weight configuration table is queried to obtain the quality influence weight coefficient corresponding to the specific lifecycle stage. The original quality deviation is multiplied by the quality influence weight coefficient to obtain a data quality assessment value weighted by stage importance.

[0042] In specific implementation, for newly entered data samples, the coordinates of the process-constrained quality state point corresponding to the newly entered data sample in the multidimensional quality feature space are calculated. The shortest geometric distance from the coordinates of the process-constrained quality state point to the current moment's quality contour surface is calculated, and the shortest geometric distance is used as the original quality deviation. The original quality deviation is negative when the quality state point is located on one side of the high-quality region divided by the quality contour surface, and positive when it is located on the side of the low-quality region. In some embodiments, the coordinates of the process-constrained quality state point of a newly entered "user registration" data sample are (0.94, 0.97, 0.89). The original quality deviation is obtained by calculating the shortest geometric distance from this coordinate point to the dynamic quality contour surface. A negative value indicates that the point is located within a high-quality, high-probability-density region. In practice, the coordinates of the process-constrained quality state point of a newly entered "order transaction" data sample are (0.82, 0.90, 0.75). The calculated shortest geometric distance to the surface of equal quality is... A positive value indicates that the point is located on one side of a low-quality, low-probability-density region. By comparison, it can be seen that the original quality deviation of the "user registration" data sample differs from that of the "order transaction" data sample in both value and sign.

[0043] The specific lifecycle stage of a new data sample is identified, and a pre-defined weighting table is consulted to obtain the quality impact weighting coefficient corresponding to that specific lifecycle stage. The original quality deviation is multiplied by this weighting coefficient to obtain a data quality assessment value weighted by stage importance. In some embodiments, the weighting table defines the quality impact weighting coefficients using the lifecycle stage as an index. For example, the quality impact weighting coefficient for the "data creation" stage is 0.8, for the "data verification" stage it is 1.2, for the "data storage" stage it is 1.0, for the "data usage" stage it is 1.1, and for the "data archiving" stage it is 0.6. It can be understood that the magnitude of the quality impact weighting coefficient reflects the degree of influence of different lifecycle stages on the final data quality assessment value; stages with higher importance have larger weighting coefficients. In specific implementations, the formula for calculating the data quality assessment value is: in: This represents the final data quality assessment value. Indicates the deviation from the original quality. This represents the quality impact weight coefficient corresponding to the current lifecycle stage, obtained by querying the weight configuration table. Optionally, in a data comparison scenario, assume the original quality deviation of a "user registration" data sample in the "data verification" stage is... The quality impact weight coefficients for the "data validation" phase are obtained by querying the weight configuration table. The data quality assessment value was calculated. Meanwhile, the original quality deviation of an "order transaction" data sample in the "data usage" stage is [missing information]. The quality impact weight coefficients for the "data usage" phase are obtained by querying the weight configuration table. The data quality assessment value was calculated. It is understandable that even if the absolute values ​​of the original quality deviations of two data samples are similar, the different lifecycle stages they are in lead to different weighting coefficients for the quality impact of the application, resulting in further differentiation in the final data quality assessment values ​​in terms of both numerical value and sign. This reflects the effect of weighted correction based on the lifecycle stage.

[0044] See Figure 5This is a comparison chart of data quality assessment values ​​after stage weighting. The quality status point of the user registration data sample is located in the high-quality, high-probability-density region, indicating better quality performance than the typical pattern. The quality status point of the order transaction data sample is located in the low-quality, low-probability-density region, indicating deviating from the typical pattern. The quality impact weight coefficients of both samples are >1, indicating that their lifecycle stages have high quality requirements. The final assessment value of the user registration sample is negative, representing quality better than the benchmark and indicating a high-quality sample. The final assessment value of the order transaction sample is positive, representing quality worse than the benchmark and requiring quality intervention. The quality importance of the user registration scenario stage is higher, hence the larger weight coefficient. The positive or negative final assessment value can quickly determine the sample quality, and the weight coefficient reflects the quality sensitivity of different businesses / stages, facilitating resource allocation.

[0045] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An AI-based method for quality assessment of data throughout its life cycle, characterized in that, The method comprises the following steps: acquiring data samples in multiple stages of the whole life cycle of data, the data samples containing data content and accompanying meta-information thereof; structurally decoupling the data content and accompanying meta-information thereof, separating a core content field of the data from a process track field describing the flow process of the core content field; in the core content field, automatically constructing a multi-dimensional quality feature space according to the data type, extracting original feature points corresponding to each dimension of the quality feature space from the core content field, and mapping the original feature points into the multi-dimensional quality feature space; in the process track field, tracking the transfer events of data between stages of the life cycle, generating a dynamic transfer track representing the state transition of data in the process track field according to the time stamp, operation subject and interface type of the transfer event; injecting the process information carried by the dynamic transfer track into the multi-dimensional quality feature space in a reverse direction, applying process constraint disturbance to the original feature points in the corresponding stage of the multi-dimensional quality feature space, so that the coordinates of the original feature points are offset with the change of the process information, forming quality state points constrained by the process; generating a data quality evaluation value according to the quality state points constrained by the process. 2.The AI-based data full life cycle quality evaluation method of claim 1, wherein, The method comprises the following steps: identifying the data structure of the data sample in the core content field, the data structure comprising a structured table, unstructured text, semi-structured log and binary stream; presetting a set of basic quality dimensions for each type of data structure, the basic quality dimensions comprising integrity, consistency, timeliness and uniqueness; extending the derived quality dimensions related to the business scenario on the basis of the basic quality dimensions according to the specific content of the data sample through pattern matching; combining the basic quality dimensions and the derived quality dimensions to form an abstract mathematical space with variable number of dimensions, which is the multi-dimensional quality feature space used to quantitatively represent the quality state of the data sample. 3.The AI-based data full life cycle quality evaluation method of claim 2, wherein, The method comprises the following steps: defining the value calculation logic of each quality dimension of the multi-dimensional quality feature space; performing calculation on the data sample in the core content field according to the value calculation logic to obtain the specific quantitative value of the data sample in each quality dimension; combining the quantitative values of each quality dimension into an ordered numerical vector, and positioning the end point of the ordered numerical vector in the multi-dimensional quality feature space, the end point being the original feature point representing the original quality feature of the data sample. 4.The AI-based data full life cycle quality evaluation method of claim 3, wherein, The method comprises the following steps: in the process track field, constructing a directed process graph with the stages of the life cycle as nodes and the data flow between stages as edges; Monitoring the behavior of each data flow from one life cycle stage node to the next life cycle stage node, recording the timestamp of the behavior, the operation subject identification triggering the behavior, and the interface type used by the data outflow and inflow, and the record information constitutes a transfer event; In the directed process graph, starting from the data generation stage node, according to the sequence of transfer events, the passing life cycle stage nodes are connected in turn to form a path through the directed process graph, and the path is a dynamic transfer trajectory attached with time, subject and interface type information. 5.The AI-based data full life cycle quality evaluation method of claim 4, wherein, Injecting the process information carried by the dynamic transfer trajectory into the multi-dimensional quality feature space in reverse, applying process constraint disturbance to the original feature points in the corresponding stage in the multi-dimensional quality feature space, including: From the dynamic transfer trajectory, analyze the life cycle stage where the data sample is currently located, and the operation subject sequence and interface type sequence of all transfer events experienced to reach the stage; Input the operation subject sequence and interface type sequence into a pre-trained disturbance evaluation model, and the disturbance evaluation model outputs the correction amount of each dimension coordinate in the multi-dimensional quality feature space; Superimpose the correction amount on the coordinate of the original feature point corresponding to the data sample, so that the original feature point is displaced along each dimension of the multi-dimensional quality feature space, and the point after displacement is a new coordinate fused with process constraint information, and the point corresponding to the new coordinate is a quality state point constrained by the process. 6.The AI-based data full life cycle quality evaluation method of claim 5, wherein, Generating a data quality evaluation value according to the quality state point constrained by the process, including: Field modeling is performed on all quality state points constrained by the process in the multi-dimensional quality feature space to generate a probability quality field that can cover all quality state points, and the field strength of any point in the probability quality field represents the occurrence probability density of the quality feature combination corresponding to the quality state point in the data full life cycle; Based on the probability quality field, a dynamically evolving quality contour surface is constructed, which divides the probability quality field into high-quality and low-quality regions; Real-time calculation of the corresponding quality state point of the newly entered data sample in the multi-dimensional quality feature space, and calculation of the position relationship of the quality state point relative to the quality contour surface to generate the original quality deviation degree; The original quality deviation degree is corrected based on the stage weight of the life cycle stage to generate a data quality evaluation value. 7.The AI-based data full life cycle quality evaluation method of claim 6, wherein, The field modeling of all quality state points constrained by the process in the multi-dimensional quality feature space to generate a probability quality field that can cover all quality state points, specifically includes: Collect all quality state points constrained by the process generated by historical data at each stage in the full life cycle to form a training point set; Using the training point set, a probability field model based on kernel density estimation is trained, which outputs the probability density estimation value of the training point set near the coordinate point according to the input of any coordinate point. The probability field model defines a continuous probability density distribution in a complete multi-dimensional quality feature space, i.e., the probability mass field, and the probability density value at any position in the probability mass field is the field strength at the position; Based on the probability mass field, a dynamic evolving quality contour surface is constructed, specifically including: In the probability mass field, a probability density threshold is set; Find all points in the probability mass field whose probability density values are equal to the probability density threshold, and the set of points forms an isosurface; Since the training point set is updated as time increases, the probability mass field is updated accordingly, and the shape and position of the isosurface also dynamically change. The dynamically changing isosurface is the quality contour surface that divides the probability mass field into a high-quality high-probability density region and a low-quality low-probability density region. 8.The AI-based data full life cycle quality evaluation method of claim 7, wherein, The real-time calculation of the new incoming data sample in the multi-dimensional quality feature space corresponding to the process-constrained quality state point, and the calculation of the position relationship of the quality state point relative to the quality contour surface, generate the original quality deviation, including: For the newly entered data sample, calculate the coordinates of the corresponding process-constrained quality state point in the multi-dimensional quality feature space; Calculate the shortest geometric distance from the coordinates of the process-constrained quality state point to the quality contour surface at the current time; Take the shortest geometric distance as the original quality deviation, wherein when the quality state point is located on one side of the high-quality region divided by the quality contour surface, the original quality deviation is negative, and when it is located on the low-quality region side, the original quality deviation is positive. 9.The AI-based data full life cycle quality evaluation method of claim 8, wherein, The original quality deviation is corrected based on the stage weight of the life cycle stage, and the stage weight is dynamically adjusted according to the life cycle stage of the data, to generate a data quality evaluation value, including: Identify the specific life cycle stage in which the new data sample is currently located; Query the pre-set weight configuration table to obtain the quality influence weight coefficient corresponding to the specific life cycle stage; Multiply the original quality deviation by the quality influence weight coefficient to obtain the data quality evaluation value weighted by the importance of the stage.

10. An AI-based data full life cycle quality assessment system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the AI-based data quality evaluation method throughout the life cycle as claimed in any one of claims 1 to 9.

Images