An AI-based data quality intelligent assessment and optimization system

Abstract

The present application provides an AI-based data quality intelligent evaluation and optimization system, including: a multi-source data acquisition module for collecting multi-source data, the sources of multi-source data include databases, network interfaces and data acquisition equipment; a multi-source data integration module for structuring multi-source data and processing it into structured data in a unified data storage format; a data quality assessment module with a built-in multi-dimensional data quality assessment model based on deep learning, which is used to evaluate the multi-dimensional quality score of structured data, and the evaluation dimensions of structured data include completeness, accuracy, consistency and timeliness; a data quality optimization module is used to analyze the data quality problems of structured data and generate prompt information based on the multi-dimensional quality score. This solution introduces artificial intelligence technology to perform intelligent evaluation and optimization of data quality, achieve accurate evaluation of data quality, significantly improve the reliability and efficiency of data quality evaluation, and provide a solid data foundation for enterprise decision-making.

Description

Technical Field

[0001] The present application relates to the field of data evaluation technology, and more specifically, to an AI-based intelligent data quality evaluation and optimization system. Background Art

[0002] In the digital age, data has become a critical basis for decision-making in businesses and organizations. Data quality can directly impact the accuracy of decisions, the smoothness of business processes, and the stability of system operations. However, with the explosive growth of data volumes and the diversification of data sources, traditional data quality assessment methods face numerous challenges.

[0003] Traditional data quality assessment relies primarily on manually defined rules and thresholds to check data integrity, accuracy, consistency, and timeliness. This approach has significant limitations: First, manually formulated rules struggle to fully cover complex and ever-changing data scenarios and lack the ability to adapt to emerging data patterns and anomalies. Second, manual assessments are inefficient and unable to meet the demands of real-time evaluation of massive amounts of data. This is particularly true in the production and sales processes of enterprises (particularly evident in the e-commerce sector), where data volumes are enormous and complex relationships exist. Assessing the quality of this data is a significant challenge in this field.

[0004] With the rapid development of artificial intelligence technology, especially the widespread application of machine learning, deep learning and other technologies in the field of data analysis, new ideas and methods have been provided for the optimization of data quality assessment systems: artificial intelligence technology can be used to automatically learn data features, discover potential patterns in the data, and achieve intelligent and accurate assessment of data quality, thereby making up for the shortcomings of traditional methods. Summary of the Invention

[0005] The purpose of the embodiments of this application is to provide an AI-based data quality intelligent assessment and optimization system, introduce artificial intelligence technology, perform intelligent assessment and optimization of data quality, achieve accurate assessment of data quality, significantly improve the reliability and efficiency of data quality assessment, and provide a solid and reliable data foundation for corporate decision-making.

[0006] In order to achieve the above objectives, the embodiments of the present application are implemented in the following manner:

[0007] In the first aspect, an embodiment of the present application provides an AI-based data quality intelligent assessment and optimization system, including: a multi-source data acquisition module for collecting multi-source data, wherein the sources of the multi-source data include databases, network interfaces and data acquisition equipment; a multi-source data integration module for structuring the multi-source data and processing it into structured data in a unified data storage format; a data quality assessment module with a built-in multi-dimensional data quality assessment model based on deep learning, which is used to evaluate the multi-dimensional quality score of structured data, wherein the assessment dimensions of structured data include completeness, accuracy, consistency, and timeliness; a data quality optimization module for analyzing data quality problems existing in structured data based on multi-dimensional quality scores and generating prompt information.

[0008] In combination with the first aspect, in a first possible implementation method of the first aspect, the multi-dimensional data quality assessment model in the data quality assessment module adopts a multi-task learning framework, including: an input layer for receiving input structured data; a shared feature layer for extracting shared features of structured data; a multi-task evaluation layer, including an integrity assessment head, an accuracy assessment head, a consistency assessment head, and a timeliness assessment head, the integrity assessment head is used to determine the integrity index of the structured data based on the shared features, the accuracy assessment head is used to determine the accuracy index of the structured data based on the shared features, the consistency assessment head is used to determine the consistency index of the structured data based on the shared features, and the timeliness assessment head is used to determine the timeliness index of the structured data based on the shared features, wherein the integrity index reflects the field missing status of the structured data, the accuracy index reflects the field abnormality of the structured data, the consistency index reflects the cross-field logical consistency of the structured data, and the timeliness index reflects the data delay of the structured data; an aggregation layer is used to determine the multi-dimensional quality score based on the integrity index, accuracy index, consistency index and timeliness index of the structured data; an output layer is used to output the multi-dimensional quality score of the structured data.

[0009] In combination with the first possible implementation method of the first aspect, in the second possible implementation method of the first aspect, the shared feature layer is specifically used to: perform field embedding on structured data to obtain a feature vector group of unified dimension, wherein the field embedding form includes numerical fields, categorical fields, and time fields; perform feature crossover on the feature vectors through a gated feature crossover network to obtain interactive feature vectors; and fuse the interactive feature vectors to generate shared features.

[0010] In combination with the second possible implementation of the first aspect, in a third possible implementation of the first aspect, the shared feature layer is specifically used to: fields, :Ruodi The fields are numeric fields and are embedded in the following way:

[0011] ,

[0012] in, Indicates the first When the field is a numeric field, the embedded feature vector is Indicates the The original value of the field, For the The mean of the fields, For the The standard deviation of the fields, is a learnable weight matrix with dimension , is a learnable bias vector with dimension ; If The fields are categorical fields and are embedded in the following way:

[0013] ,

[0014] in, Indicates the first When the field is a categorical field, the embedded feature vector is Indicates the The original value of the field, Indicates the One-hot encoding vector of fields, represents the learnable embedding matrix, The number of rows is equal to The number of categories of the field, that is length, The number of columns is equal to the target embedding dimension ; If The fields are time fields and are embedded in the following way:

[0015] ,

[0016] in, Indicates the first When the field is a time field, the embedded feature vector is Indicates the numerical representation of the timestamp, is the time period, is a learnable weight matrix with dimension ; After embedding each field of the structured data, a set of feature vectors is obtained ,in, The total number of fields in the structured data.

[0017] In combination with the second possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the shared feature layer is specifically used to: perform feature crossover on the feature vector through a gated feature crossover network:

[0018] ,

[0019] in, Indicates the The feature vector of the field With the The feature vector of the field The interaction feature vector between is the Sigmoid activation function, is a learnable weight matrix with dimension , Represents the feature vector With the eigenvector The splicing dimension is , Indicates the calculation of Hadamard product, is a learnable weight matrix with dimension ; and, based on the feature vector for each field and the interaction eigenvector between every two eigenvectors , generate shared features:

[0020] ,

[0021] in, For shared features.

[0022] In combination with the first possible implementation method of the first aspect, in the fifth possible implementation method of the first aspect, the multi-task evaluation layer is specifically used to: perform differentiated pre-processing on shared features to obtain an integrity assessment input vector, an accuracy assessment input vector, a consistency assessment input vector, and a timeliness assessment input vector, respectively; perform nonlinear transformation on the integrity assessment input vector, the accuracy assessment input vector, the consistency assessment input vector, and the timeliness assessment input vector; based on the integrity assessment input vector, the accuracy assessment input vector, the consistency assessment input vector, and the timeliness assessment input vector after the nonlinear transformation, calculate the integrity index, accuracy index, consistency index, and timeliness index of the structured data through the integrity assessment head, the accuracy assessment head, the consistency assessment head, and the timeliness assessment head, respectively.

[0023] In combination with the fifth possible implementation of the first aspect, in a sixth possible implementation of the first aspect, the multi-task evaluation layer is specifically used to: obtain the integrity task mask matrix :

[0024] ,

[0025] The integrity task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation:

[0026] ,

[0027] in, Input vector for integrity assessment, dimension is ; Get the accuracy task mask matrix :

[0028] ,

[0029] The accuracy task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation:

[0030] ,

[0031] in, Input vector for accuracy evaluation, dimension is ; Get the consistent task mask matrix :

[0032] ,

[0033] The consistency task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation:

[0034] ,

[0035] in, Input vector for consistency evaluation, dimension is ; Get the time-sensitive task mask matrix :

[0036] ,

[0037] The time-sensitive task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation:

[0038] ,

[0039] in, Input vector for timeliness evaluation, dimension is .

[0040] In conjunction with the sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, the multi-task evaluation layer is specifically configured to perform nonlinear transformation in the following manner:

[0041] ,

[0042] ,

[0043] ,

[0044] ,

[0045] in, 、 、 and are the integrity assessment input vector, accuracy assessment input vector, consistency assessment input vector and timeliness assessment input vector after nonlinear transformation, 、 、 and They are respectively the learnable weight matrices, and their dimensions are , 、 、 and Are bias vectors, the dimensions are .

[0046] In conjunction with the seventh possible implementation manner of the first aspect, in an eighth possible implementation manner of the first aspect, the multi-task evaluation layer is specifically configured to: the integrity assessment head calculates the integrity index of the structured data based on the integrity assessment input vector after nonlinear transformation using the following method:

[0047] ,

[0048] in, is the integrity index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the bias; the accuracy evaluation head is based on the accuracy evaluation input vector after nonlinear transformation, and calculates the accuracy index of structured data in the following way:

[0049] ,

[0050] in, is the accuracy index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the bias; the consistency assessment head calculates the consistency index of the structured data based on the consistency assessment input vector after nonlinear transformation in the following way:

[0051] ,

[0052] in, is the consistency index of structured data, is the Sigmoid function, for The number of dimensions, for No. values, for No. values, is a scalar weight, is the bias; the timeliness evaluation head calculates the timeliness index of structured data based on the timeliness evaluation input vector after nonlinear transformation in the following way:

[0053] ,

[0054] in, is the timeliness index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the time attenuation coefficient, is the aging threshold timestamp, The timestamp of the structured data. is the offset.

[0055] In conjunction with the eighth possible implementation manner of the first aspect, in a ninth possible implementation manner of the first aspect, the aggregation layer is specifically configured to calculate a multi-dimensional quality score based on the completeness index, accuracy index, consistency index, and timeliness index of the structured data using the following method:

[0056] ,

[0057] in, Represents the multi-dimensional quality score of structured data. 、 、 and is the weight.

[0058] Beneficial Effects: This solution collects multi-source data (derived from databases, network interfaces, data acquisition devices, and other multi-source data collection methods) and structures the data into structured data in a unified data storage format. It then utilizes a deep learning-based multidimensional data quality assessment model to evaluate the multidimensional quality of structured data from multiple dimensions, including completeness, accuracy, consistency, and timeliness. This multidimensional data quality assessment model employs a multi-task learning framework to extract shared features from structured data. It then deploys an integrity assessment head, an accuracy assessment head, a consistency assessment head, and a timeliness assessment head. Using these shared features, it determines the completeness index, accuracy index, consistency index, and timeliness index of the structured data, and comprehensively calculates a multidimensional quality score. This achieves a comprehensive multidimensional evaluation of structured data, breaking the isolation of traditional single-task evaluations, reducing repeated calculations, and enabling efficient collaborative evaluation. It also avoids the risk of information loss associated with different evaluation sequences in traditional pipelined evaluation schemes. This solution can also analyze data quality issues in structured data based on the multidimensional quality score and generate prompts, thereby achieving quality assessment of multi-source data. This solution uses artificial intelligence technology to conduct intelligent data quality assessment and optimization in multiple dimensions, achieving accurate assessment of data quality, significantly improving the evaluation efficiency of data quality, and ensuring the reliability of quality assessment, providing a solid and reliable data foundation for corporate decision-making.

[0059] Considering the characteristics of the structured data to be evaluated in this solution, a unified embedding mechanism for heterogeneous fields and gated feature crossover was designed to generate shared features with strong reflective capabilities. This mechanism addresses the two major issues in traditional data quality assessment: fragmented feature representation and inefficient interaction modeling. Field embedding is divided into numerical fields (using a standardized linear mapping), categorical fields (using a one-hot embedding matrix approach that dynamically adapts to the number of field categories, avoiding redundancy or information loss caused by a fixed embedding dimension), and time fields (using a periodic encoding + linear transformation approach to smooth distances between similar time points, which is superior to direct numerical timestamps). Maintaining a unified embedding dimension for each field, this adaptive field embedding approach addresses the issues of feature representation distortion and semantic loss in traditional data quality assessment systems. Furthermore, future optimizations will allow for the addition of more field types (such as state fields) to accommodate a wider range of structured data quality assessments. By performing feature crossover on feature vectors through a gated feature crossover network, we generate interaction feature vectors. This allows us to identify important interactions between different fields, strengthen logical interactions between valid fields, and weaken interactions between irrelevant fields. Compared to traditional fully connected crossover methods, this approach achieves gated compression and Hadamard product optimization, helping to eliminate irrelevant noise and addressing the problem of missing nonlinear relationships in high-order interaction modeling, which occurs when traditional statistical methods capture linear relationships (traditional statistical methods can only capture linear relationships). By fusing interaction feature vectors to generate shared features, we can preserve both the field's inherent characteristics and pairwise interaction features, thus adapting to tasks with different evaluation dimensions.

[0060] In the design of the multi-task evaluation layer, by performing differentiated pre-processing on shared features, the integrity evaluation input vector, accuracy evaluation input vector, consistency evaluation input vector, and timeliness evaluation input vector for different evaluation tasks (completeness evaluation, accuracy evaluation, consistency evaluation, and timeliness evaluation) are obtained respectively. For different evaluation tasks, a scheme of using mask matrices to perform differentiated pre-processing on shared features is creatively proposed. Through task-driven feature screening, the problems of feature redundancy interference and task conflict in traditional multi-task learning are solved. It can well take into account the emphasis of different evaluation tasks to determine the input vectors corresponding to different evaluation tasks (completeness evaluation input vector, accuracy evaluation input vector, consistency evaluation input vector, and timeliness evaluation input vector), which effectively reduces the computational complexity of each task evaluation head. Existing multi-task models typically directly input shared features into all task heads, resulting in task-irrelevant features interfering with the evaluation (for example, timeliness evaluation does not require field missing information) and diluting key features (for example, consistency evaluation requires cross-field interaction but is overwhelmed by the field's own features). This solution proposes a method of using a mask matrix to perform differentiated pre-processing on shared features, which allows each task head to focus on the features it needs, improving feature effectiveness while significantly reducing the amount of data processing. Next, nonlinear transformations are performed on the completeness, accuracy, consistency, and timeliness input vectors, imbuing each with nonlinear reflection capabilities. Appropriate calculation methods are then applied to the completeness, accuracy, consistency, and timeliness headers, respectively. Different assessment tasks focus on different aspects. For example, completeness and accuracy primarily examine the possibility of missing or abnormal fields and the importance of missing or abnormal fields, considering fields that are strongly correlated with missing or abnormal fields. These methods are primarily measured using linear mappings. Consistency, on the other hand, primarily considers cross-field interactions, thus requiring nonlinear mappings. Timeliness, based on linear mappings, incorporates the concept of time decay to measure the timeliness of data. Based on these methods, the completeness, accuracy, consistency, and timeliness indices of the structured data are calculated. Finally, when calculating the multidimensional quality score, different weights are assigned to different dimensions and a weighted sum is calculated to obtain the final multidimensional quality score. This effectively improves the accuracy and reliability of quality assessments for structured data, while reducing the overall computational complexity and increasing efficiency compared to traditional pipelined, single-task evaluations.

[0061] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 A framework diagram of the AI-based data quality intelligent assessment and optimization system provided in an embodiment of the present application.

[0064] Figure 2 Schematic diagram of the process for intelligent data quality assessment and optimization system.

[0065] Figure 3 This is a framework diagram of the multi-dimensional data quality assessment model based on deep learning.

[0066] Icons: 10-Data quality intelligent assessment and optimization system; 11-Multi-source data acquisition module; 12-Multi-source data integration module; 13-Data quality assessment module; 14-Data quality optimization module. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0068] See also Figure 1 and Figure 2 , Figure 1 This is a framework diagram of an AI-based data quality intelligent assessment and optimization system 10 provided in an embodiment of the present application. Figure 2 Schematic diagram of the process of data quality intelligent evaluation and optimization system 10.

[0069] In this embodiment, the AI-based data quality intelligent assessment and optimization system 10 may include a multi-source data acquisition module 11, a multi-source data integration module 12, a data quality assessment module 13 and a data quality optimization module 14.

[0070] The multi-source data acquisition module 11 is primarily used to collect multi-source data. The sources of multi-source data include databases, network interfaces, and data acquisition devices. The types of multi-source data mainly include structured data, semi-structured data, and unstructured data. Structured data and semi-structured data from databases are the majority. Structured data includes order tables (fields: order ID, user ID, amount, creation time), user information tables (fields: user ID, name, age, address), product information such as {product ID: "P1001", name: "mobile phone", price: 3999, inventory: 100}, key-value pair data (such as Redis cache: session:user123 = {last_login: "2023-06-10 09:30", status: "active"}), etc.; semi-structured data includes device parameters: <sensor> <id> S001< / id> <type> Temperature< / type> <range> 0-100℃< / range> < / sensor> , log files (such as Apache logs: 192.168.1.1 - - [10 / Jun / 2023:10:30:22 +0800]"GET / index.html HTTP / 1.1" 200 2326). Data from network interfaces is primarily semi-structured and unstructured data, such as message queues (such as Kafka): user behavior streams ({user_id:"U1001", action:"click", timestamp: 1686402000, page:" / product"}), IoT data streams (such as MQTT): real-time sensor reports (device ID: D001, temperature: 32.5°C, voltage: 5.2V, time:1686402010), and text or semi-structured data crawled by web crawlers. Data collected from data collection devices is primarily semi-structured data, such as industrial sensor temperature series ([timestamp: 1686402000, value: 25.3], [1686402005, 25.5], ...) and GPS device track points (latitude: 39.9042, longitude: 116.4074, speed: 60 km / h, time: 2023-06-10 10:00:00). Of course, data from the same enterprise doesn't actually span such a wide range and is typically focused on a specific area. For example, for a production-focused enterprise, data collected by equipment and stored in databases are predominantly semi-structured and structured data. For a marketing-focused enterprise, data from databases and network interfaces is predominant. Much of this valuable information is unstructured.

[0071] After collecting multi-source data, the multi-source data integration module 12 can perform structured processing on the multi-source data, processing it into structured data in a unified data storage format. The structured processing of data is not the focus of this solution and will not be discussed in detail here. However, it should be emphasized that in order to perform data quality assessment on a certain portion of data, it is necessary to maintain a unified structured storage format for that portion of data.

[0072] After completing the structured processing of multi-source data, the data quality assessment module 13 (with a built-in multi-dimensional data quality assessment model based on deep learning) can evaluate the multi-dimensional quality score of the structured data. The assessment dimensions of the structured data include completeness, accuracy, consistency, timeliness, etc.

[0073] To facilitate understanding of this solution, we first introduce the multi-dimensional data quality assessment model based on deep learning. Figure 3 ,The multi-dimensional data quality assessment model adopts a multi-task learning framework, including ,input layer, shared feature layer, multi-task evaluation layer, aggregation layer and ,output layer.

[0074] The input layer is mainly used to receive input structured data.

[0075] The shared feature layer is mainly used to extract shared features of structured data.

[0076] The multi-task evaluation layer includes an integrity assessment head, an accuracy assessment head, a consistency assessment head, and a timeliness assessment head. The integrity assessment head is used to determine the integrity index of structured data based on shared features, the accuracy assessment head is used to determine the accuracy index of structured data based on shared features, the consistency assessment head is used to determine the consistency index of structured data based on shared features, and the timeliness assessment head is used to determine the timeliness index of structured data based on shared features. Among them, the integrity index reflects the field missing status of structured data, the accuracy index reflects the field anomaly of structured data, the consistency index reflects the cross-field logical consistency of structured data, and the timeliness index reflects the data delay of structured data.

[0077] The aggregation layer is mainly used to determine multi-dimensional quality scores based on the completeness index, accuracy index, consistency index and timeliness index of structured data.

[0078] The output layer is used to output multi-dimensional quality scores of structured data.

[0079] In order to adapt the integrity assessment head, accuracy assessment head, consistency assessment head, and timeliness assessment head designed in the multi-task evaluation layer, a series of processing needs to be performed on the structured data at the shared feature layer to extract shared features that can provide underlying feature sharing for each task head.

[0080] In this embodiment, after the input layer passes the structured data to the shared feature layer, the shared feature layer can perform field embedding on the structured data to obtain a feature vector group of uniform dimensions. The field embedding form in this embodiment includes numerical fields, categorical fields, and time fields. For example, the structured data is { }, assuming The order amount , For the city , Order time (This is a timestamp format with a time zone. No time zone conversion is required. The time zone can be processed during structured processing.)

[0081] Then, for the first fields, , according to The type of each field is processed as follows:

[0082] Jordi The fields are numeric fields and are embedded in the following way:

[0083] , (1)

[0084] in, Indicates the first When the field is a numeric field, the embedded feature vector is Indicates the The original value of the field, For the The mean of the fields (obtained through training), For the The standard deviation of the fields (obtained through training), is a learnable weight matrix with dimension , is a learnable bias vector with dimension .

[0085] Jordi The fields are categorical fields and are embedded in the following way:

[0086] , (2)

[0087] in, Indicates the first When the field is a categorical field, the embedded feature vector is Indicates the The original value of the field, Indicates the One-hot encoding vector of fields, represents the learnable embedding matrix, The number of rows is equal to The number of categories of the field, that is length, The number of columns is equal to the target embedding dimension .

[0088] Jordi The fields are time fields and are embedded in the following way:

[0089] , (3)

[0090] in, Indicates the first When the field is a time field, the embedded feature vector is Indicates the numerical representation of the timestamp, is the time period (based on the data statistics period, for example, 24 hours), is a learnable weight matrix with dimension .

[0091] By embedding each field of the structured data into its corresponding field, we can get a set of feature vectors. ,in, The total number of fields in the structured data.

[0092] In field embedding, we embed numerical fields (using standardized linear mapping), categorical fields (a one-hot + embedding matrix solution that can dynamically adapt to the number of field categories and avoid redundancy or information loss caused by fixed embedding dimensions), and time fields (using a periodic encoding + linear transformation solution to smooth the distance between similar time points, which is better than the direct numericalization of timestamps). The unified dimension of each field embedding is maintained. This type of adaptive field embedding method solves the problems of feature representation distortion and semantic loss in traditional data quality assessment systems. In future solution optimization, more types of fields (such as state fields) can be added to accommodate a wider range of structured data quality assessments.

[0093] Get the feature vector group After that, the shared feature layer can perform feature crossover on the feature vector through the gated feature crossover network to obtain the interactive feature vector.

[0094] For example, the shared feature layer can perform feature crossover on the feature vector through a gated feature crossover network:

[0095] , (4)

[0096] in, Indicates the The feature vector of the field With the The feature vector of the field The interaction feature vector between is the Sigmoid activation function, is a learnable weight matrix with dimension , Represents the feature vector With the eigenvector The splicing dimension is , Indicates the calculation of Hadamard product, is a learnable weight matrix with dimension .

[0097] Based on this, the eigenvector group is determined The interaction feature vector between each two fields in Finally, the shared feature layer can fuse the interactive feature vectors to generate shared features: Specifically, it can be based on the feature vector of each field and the interaction eigenvector between every two eigenvectors , generate shared features:

[0098] , (5)

[0099] in, is a shared feature, and the principal axis element is a feature vector group The eigenvectors in are filled, while the non-principal axis elements are filled by the interaction eigenvectors Fill in the corresponding positions, thus forming a matrix of shared features .

[0100] Considering the characteristics of the structured data to be evaluated in this solution, a unified embedding and gated feature crossover mechanism for heterogeneous fields was designed to generate shared features with strong reflective power. This mechanism addresses the two major issues in traditional data quality assessment: fragmented feature representation and inefficient interaction modeling. By performing feature crossover on feature vectors through a gated feature crossover network, interaction feature vectors are generated. This can filter out important interactions between different fields, strengthen logical interactions between valid fields, and weaken interactions between irrelevant fields. Compared to traditional fully connected crossover methods, this method implements gated compression and Hadamard product optimization, which helps eliminate irrelevant noise and address the problem of missing nonlinear relationships in high-order interaction modeling, which is common with traditional statistical methods that capture linear relationships (traditional statistical methods can only capture linear relationships). By fusing interaction feature vectors to generate shared features, both the field's inherent characteristics and pairwise interaction characteristics are preserved, adapting to tasks with different evaluation dimensions.

[0101] After obtaining the shared features, the multi-task evaluation layer can perform differentiated pre-processing on the shared features to obtain the completeness evaluation input vector, accuracy evaluation input vector, consistency evaluation input vector, and timeliness evaluation input vector respectively.

[0102] In this embodiment, the multi-task evaluation layer can obtain the integrity task mask matrix :

[0103] , (6)

[0104] The completeness task mask matrix here , is determined by considering the characteristics of the integrity assessment task and can be directly called during operation. In the construction phase, we can construct a key field set (distinguish between key fields and non-key fields. Key fields consider the field itself and the interaction between other fields and this field, while non-key fields only consider the field itself and do not need to consider the interaction between other fields and this field). According to the key field set, we assign initial values ​​to each element in the matrix. For the parts that are not considered, we can assign zero to the corresponding element position (for the elements assigned zero, there is no need to train and update these parameters). For the parts that need to be considered, we assign certain non-zero initial values, continuously learn in the training phase, and update the integrity task mask matrix. The values ​​of each element in , then, after the training phase, the determined integrity task mask matrix can be obtained .

[0105] At this point, the multi-task evaluation layer can transform the completeness task mask matrix Shared features The corresponding position elements in are multiplied and weighted summed to obtain the integrity assessment input vector:

[0106] , (7)

[0107] in, Input vector for integrity assessment, dimension is .

[0108] In this embodiment, the multi-task evaluation layer can obtain the accuracy task mask matrix :

[0109] , (8)

[0110] The accuracy task mask matrix here , is determined by considering the characteristics of the accuracy assessment task and can be directly called during operation. In the construction phase, we can construct a Class I field type combination set (the Class I field type combination set mainly considers several field combinations that are more important for the accuracy assessment task, without considering all field combinations. The Class I field type combination mainly includes: numerical field-numerical field combination, numerical field-category field combination, category field-category field combination), and consider the field itself to form the accuracy task mask matrix. Assign initial values ​​(for the parts that are not considered, assign zero to the corresponding element position (the elements assigned to zero do not need to train and update these parameters), and assign certain non-zero initial values ​​to the parts that need to be considered. Continuously learn during the training phase and update the accuracy task mask matrix ), after training in the training phase, the accuracy task mask matrix is ​​determined .

[0111] At this point, the multi-task evaluation layer can convert the accuracy task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation:

[0112] , (9)

[0113] in, Input vector for accuracy evaluation, dimension is .

[0114] In this embodiment, the multi-task evaluation layer can obtain the consistent task mask matrix :

[0115] , (10)

[0116] Since the consistency assessment task mainly considers the cross-field consistency of structured data, it does not need to consider the field itself, but the interaction between different fields. Therefore, a global fusion method is adopted, with the main axis element as 0 and the other elements as 1. Matrix, as the consistency task mask matrix Of course, in other schemes, the main axis elements can be forced to be 0, and the remaining elements are assigned non-zero initial values. After the training phase, the consistent task mask matrix is ​​obtained. .

[0117] At this point, the multi-task evaluation layer can transform the consistent task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation:

[0118] , (11)

[0119] in, Input vector for consistency evaluation, dimension is .

[0120] In this embodiment, the multi-task evaluation layer can obtain the time-sensitive task mask matrix :

[0121] , (12)

[0122] The time-sensitive task mask matrix here , is determined by considering the characteristics of the timeliness evaluation task and can be directly called during operation. In the construction phase, we can construct a Class II field type combination set (Class II field type combination set mainly considers the field combination that is more important for the timeliness evaluation task, without considering all field combinations. Class II field type combinations mainly include: time type field-time type field combination, time type field-numeric field combination), and consider the field itself at the same time to form a mask matrix for the timeliness task. Assign initial values ​​(for the parts that are not considered, assign zero to the corresponding element position (the elements assigned to zero do not need to train and update these parameters), assign certain non-zero initial values ​​to the parts that need to be considered, continuously learn during the training phase, and update the time-sensitive task mask matrix ), after the training phase, the determined time-sensitive task mask matrix is ​​obtained .

[0123] Based on this, the multi-task evaluation layer can transform the time-sensitive task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation:

[0124] , (13)

[0125] in, Input vector for timeliness evaluation, dimension is .

[0126] In the design of the multi-task evaluation layer, by performing differentiated pre-processing on shared features, the integrity evaluation input vector, accuracy evaluation input vector, consistency evaluation input vector, and timeliness evaluation input vector for different evaluation tasks (completeness evaluation, accuracy evaluation, consistency evaluation, and timeliness evaluation) are obtained respectively. For different evaluation tasks, a scheme of using mask matrices to perform differentiated pre-processing on shared features is creatively proposed. Through task-driven feature screening, the problems of feature redundancy interference and task conflict in traditional multi-task learning are solved. It can well take into account the emphasis of different evaluation tasks to determine the input vectors corresponding to different evaluation tasks (completeness evaluation input vector, accuracy evaluation input vector, consistency evaluation input vector, and timeliness evaluation input vector), which effectively reduces the computational complexity of each task evaluation head. Existing multi-task models typically directly input shared features into all task heads, resulting in task-irrelevant features interfering with the evaluation (for example, timeliness evaluation does not require field missing information) and diluting key features (for example, consistency evaluation requires cross-field interaction but is overwhelmed by the field's own features). This solution proposes a method of using a mask matrix to perform differentiated pre-processing on shared features, which allows each task head to focus on the features it needs, improving feature effectiveness while significantly reducing the amount of data processing.

[0127] After pre-processing the shared features, the completeness assessment input vector, accuracy assessment input vector, consistency assessment input vector, and timeliness assessment input vector are obtained. To give each input vector nonlinear reflection capabilities, the multi-task evaluation layer can perform nonlinear transformations on the completeness assessment input vector, accuracy assessment input vector, consistency assessment input vector, and timeliness assessment input vector.

[0128] For example, the multi-task evaluation layer can perform nonlinear transformation in the following manner:

[0129] , (14)

[0130] , (15)

[0131] , (16)

[0132] , (17)

[0133] in, 、 、 and are the integrity assessment input vector, accuracy assessment input vector, consistency assessment input vector and timeliness assessment input vector after nonlinear transformation, 、 、 and They are respectively the learnable weight matrices, and their dimensions are , 、 、 and Are bias vectors, the dimensions are .

[0134] After completing the nonlinear mapping, the multi-task evaluation layer can calculate the integrity index, accuracy index, consistency index and timeliness index of the structured data through the integrity evaluation head, accuracy evaluation head, consistency evaluation head and timeliness evaluation head respectively based on the integrity evaluation input vector, accuracy evaluation input vector, consistency evaluation input vector and timeliness evaluation input vector after nonlinear transformation.

[0135] Specifically, the integrity assessment header calculates the integrity index of the structured data based on the integrity assessment input vector after nonlinear transformation in the following way:

[0136] , (18)

[0137] in, is the integrity index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the offset.

[0138] The accuracy evaluation head calculates the accuracy index of structured data based on the accuracy evaluation input vector after nonlinear transformation in the following way:

[0139] , (19)

[0140] in, is the accuracy index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the offset.

[0141] The consistency evaluation head calculates the consistency index of structured data based on the consistency evaluation input vector after nonlinear transformation in the following way:

[0142] , (20)

[0143] in, is the consistency index of structured data, is the Sigmoid function, for The number of dimensions, for No. values, for No. values, is a scalar weight, is the offset.

[0144] The timeliness evaluation head calculates the timeliness index of structured data based on the timeliness evaluation input vector after nonlinear transformation using the following method:

[0145] , (twenty one)

[0146] in, is the timeliness index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the time attenuation coefficient, is the aging threshold timestamp, The timestamp of the structured data. is the offset.

[0147] By performing nonlinear transformation on the completeness assessment input vector, accuracy assessment input vector, consistency assessment input vector and timeliness assessment input vector, each input vector is given nonlinear reflection ability, and then appropriate calculation methods are adopted through the completeness assessment head, accuracy assessment head, consistency assessment head and timeliness assessment head respectively: different assessment tasks have different focuses. For example, completeness and accuracy mainly examine whether there may be missing fields or field anomalies, whether the missing fields or abnormal fields are important, and fields that are strongly associated with missing fields or abnormal fields can be considered, mainly measured through linear mapping; consistency mainly considers cross-field interactions, so nonlinear mapping needs to be considered; and timeliness introduces the idea of ​​time decay on the basis of linear mapping to measure the timeliness of data, and accordingly calculates the completeness index, accuracy index, consistency index and timeliness index of structured data to achieve accurate evaluation of structured data in various dimensions.

[0148] After obtaining the completeness index, accuracy index, consistency index, and timeliness index of the structured data, the aggregation layer can further calculate the multi-dimensional quality score based on the completeness index, accuracy index, consistency index, and timeliness index of the structured data in the following way:

[0149] , (twenty two)

[0150] in, Represents the multi-dimensional quality score of structured data. 、 、 and is the weight.

[0151] The final multi-dimensional quality score is calculated by assigning different weights to different dimensions and performing a weighted summation. This effectively improves the accuracy and reliability of quality assessments for structured data, while also reducing the overall computational complexity and increasing efficiency compared to traditional pipelined, single-task evaluations.

[0152] After obtaining the multi-dimensional quality score of the structured data, the data quality optimization module 14 can analyze the structured data for any quality issues based on the multi-dimensional quality score and generate prompts. The data quality optimization module 14 can also generate analysis reports based on the multi-dimensional quality score of the structured data and further generate prompts to alert relevant personnel. Furthermore, the data quality optimization module 14 can analyze the root causes of quality issues within the multi-dimensional quality score (including each dimensional index) based on the multi-dimensional quality score of a batch of structured data through methods such as causal analysis and principal component analysis, and provide data quality optimization recommendations.

[0153] In summary, the embodiment of the present application provides an AI-based data quality intelligent assessment and optimization system 10. By collecting multi-source data (from multi-source data collection methods such as databases, network interfaces, and data acquisition devices), and structuring the multi-source data into structured data in a unified data storage format, a multi-dimensional data quality assessment model based on deep learning is used to evaluate the multi-dimensional quality score of the structured data from multiple dimensions such as completeness, accuracy, consistency, and timeliness. The multi-dimensional data quality assessment model adopts a multi-task learning framework to extract shared features of the structured data, deploy an integrity assessment head, an accuracy assessment head, a consistency assessment head, and a timeliness assessment head. Using the shared features, the integrity index, accuracy index, consistency index, and timeliness index of the structured data are determined, and a multi-dimensional quality score is comprehensively calculated to achieve a multi-dimensional comprehensive evaluation of the structured data. This breaks the isolation of traditional single-task evaluation, reduces repeated calculations, achieves efficient collaborative evaluation, and avoids the risk of information loss caused by different evaluation orders in traditional pipeline evaluation schemes. This solution can also analyze data quality issues existing in the structured data based on the multi-dimensional quality score and generate prompt information, thereby achieving quality evaluation of multi-source data. This solution uses artificial intelligence technology to conduct intelligent data quality assessment and optimization in multiple dimensions, achieving accurate assessment of data quality, significantly improving the evaluation efficiency of data quality, and ensuring the reliability of quality assessment, providing a solid and reliable data foundation for corporate decision-making.

[0154] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0155] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An AI-based data quality intelligent evaluation and optimization system, characterized by: include: A multi-source data acquisition module is used to collect multi-source data, wherein the sources of the multi-source data include databases, network interfaces and data acquisition devices; Multi-source data integration module, used to perform structured processing on multi-source data and process it into structured data in a unified data storage format; The data quality assessment module has a built-in multi-dimensional data quality assessment model based on deep learning, which is used to evaluate the multi-dimensional quality score of structured data. The evaluation dimensions of structured data include completeness, accuracy, consistency, and timeliness. The data quality optimization module is used to analyze the data quality issues of structured data based on multi-dimensional quality scores and generate prompt information; The multi-dimensional data quality assessment model in the data quality assessment module adopts a multi-task learning framework, including: Input layer, used to receive input structured data; Shared feature layer, used to extract shared features of structured data; The multi-task evaluation layer includes an integrity assessment head, an accuracy assessment head, a consistency assessment head, and a timeliness assessment head. The integrity assessment head is used to determine the integrity index of structured data based on shared features, the accuracy assessment head is used to determine the accuracy index of structured data based on shared features, the consistency assessment head is used to determine the consistency index of structured data based on shared features, and the timeliness assessment head is used to determine the timeliness index of structured data based on shared features. The integrity index reflects the field missing status of structured data, the accuracy index reflects the field anomaly status of structured data, the consistency index reflects the cross-field logical consistency of structured data, and the timeliness index reflects the data delay status of structured data. The aggregation layer is used to determine multi-dimensional quality scores based on the completeness index, accuracy index, consistency index, and timeliness index of structured data; The output layer is used to output multi-dimensional quality scores of structured data; Shared feature layer, specifically used for: Perform field embedding on structured data to obtain a feature vector group of unified dimension, where the field embedding forms include numerical fields, categorical fields, and time fields; Perform feature crossover on the feature vectors through the gated feature crossover network to obtain the interactive feature vectors; Fuse the interactive feature vectors to generate shared features; Shared feature layer, specifically used for: For the first fields, : Jordi The fields are numeric fields and are embedded in the following way: ， in, Indicates the first When the field is a numeric field, the embedded feature vector is Indicates the The original value of the field, For the The mean of the fields, For the The standard deviation of the fields, is a learnable weight matrix with dimension , is a learnable bias vector with dimension ; Jordi The fields are categorical fields and are embedded in the following way: ， in, Indicates the first When the field is a categorical field, the embedded feature vector is Indicates the The original value of the field, Indicates the One-hot encoding vector of fields, represents the learnable embedding matrix, The number of rows is equal to The number of categories of the field, that is length, The number of columns is equal to the target embedding dimension ; Jordi The fields are time fields and are embedded in the following way: ， in, Indicates the first When the field is a time field, the embedded feature vector is Indicates the numerical representation of the timestamp, is the time period, is a learnable weight matrix with dimension ; After embedding each field of the structured data, a set of feature vectors is obtained. ,in, is the total number of fields in the structured data; Shared feature layer, specifically used for: Perform feature crossover on the feature vector through the gated feature crossover network: ， in, Indicates the The feature vector of the field With the The feature vector of the field The interaction feature vector between is the Sigmoid activation function, is a learnable weight matrix with dimension , Represents the feature vector With the eigenvector The splicing dimension is , Indicates the calculation of Hadamard product, is a learnable weight matrix with dimension ; And, based on the feature vector of each field and the interaction eigenvector between every two eigenvectors , generate shared features: ， in, For shared features.

2. The AI-based data quality intelligent evaluation and optimization system according to claim 1 is characterized in that: Multi-task evaluation layer, specifically used for: Perform differential pre-processing on the shared features to obtain the completeness assessment input vector, accuracy assessment input vector, consistency assessment input vector, and timeliness assessment input vector respectively; Performing nonlinear transformation on the integrity assessment input vector, the accuracy assessment input vector, the consistency assessment input vector, and the timeliness assessment input vector; Based on the integrity assessment input vector, accuracy assessment input vector, consistency assessment input vector and timeliness assessment input vector after nonlinear transformation, the integrity index, accuracy index, consistency index and timeliness index of the structured data are calculated through the integrity assessment header, accuracy assessment header, consistency assessment header and timeliness assessment header respectively.

3. The AI-based data quality intelligent evaluation and optimization system according to claim 2 is characterized in that: Multi-task evaluation layer, specifically used for: Get the completeness task mask matrix : ， The integrity task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation: ， in, Input vector for integrity assessment, dimension is ; Get the accuracy task mask matrix : ， The accuracy task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation: ， in, Input vector for accuracy evaluation, dimension is ; Get the consistent task mask matrix : ， The consistency task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation: ， in, Input vector for consistency evaluation, dimension is ; Get the time-sensitive task mask matrix : ， The time-sensitive task mask matrix Shared features Multiply the corresponding position elements in and then perform weighted summation: ， in, Input vector for timeliness evaluation, dimension is .

4. The AI-based data quality intelligent evaluation and optimization system according to claim 3 is characterized in that: Multi-task evaluation layer, specifically used for: The nonlinear transformation is performed in the following way: ， ， ， ， in, 、 、 and are the integrity assessment input vector, accuracy assessment input vector, consistency assessment input vector and timeliness assessment input vector after nonlinear transformation, 、 、 and They are respectively the learnable weight matrices, and their dimensions are , 、 、 and Are bias vectors, the dimensions are .

5. The AI-based data quality intelligent evaluation and optimization system according to claim 4 is characterized in that: Multi-task evaluation layer, specifically used for: The integrity assessment header calculates the integrity index of the structured data based on the integrity assessment input vector after nonlinear transformation using the following method: ， in, is the integrity index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the offset; The accuracy evaluation head calculates the accuracy index of structured data based on the accuracy evaluation input vector after nonlinear transformation in the following way: ， in, is the accuracy index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the offset; The consistency assessment head calculates the consistency index of structured data based on the consistency assessment input vector after nonlinear transformation in the following way: ， in, is the consistency index of structured data, is the Sigmoid function, for The number of dimensions, for No. values, for No. values, is a scalar weight, is the offset; The timeliness evaluation head calculates the timeliness index of structured data based on the timeliness evaluation input vector after nonlinear transformation using the following method: ， in, is the timeliness index of structured data, is the Sigmoid function, for The transpose of for The weight vector of is the time attenuation coefficient, is the aging threshold timestamp, The timestamp of the structured data. is the offset.

6. The AI-based data quality intelligent evaluation and optimization system according to claim 5 is characterized in that: The aggregation layer is specifically used for: Based on the completeness index, accuracy index, consistency index, and timeliness index of structured data, a multi-dimensional quality score is calculated in the following way: ， in, Represents the multi-dimensional quality score of structured data. 、 、 and is the weight.

Images