Data quality monitoring system and method

Abstract

The invention relates to the technical field of data monitoring, and particularly discloses a data quality monitoring system and method, and the system comprises a data quality rule obtaining module which is used for defining data quality rules to obtain a data quality rule set; the data quality inspection report generation module is used for performing data exploration and analysis on the source data to obtain a data quality inspection report; and the source data quality grade evaluation module is used for carrying out quality evaluation on the data quality inspection report based on the set of the data quality rules to obtain a quality evaluation result, and the quality evaluation result is used for representing a quality grade label of the source data. Therefore, the data quality inspection report can be obtained by automatically acquiring and processing various data of different types and formats, and the data quality inspection report is analyzed by utilizing multiple data quality rules to realize quality evaluation of the source data, so that the quality grade label of the source data is determined.

Description

Technical Field

[0001] The present application relates to the field of data monitoring technology, and more specifically, to a data quality monitoring system and method. Background Art

[0002] In the era of big data, organizations and enterprises generate a large amount of data every day, and the quality of this data directly affects the effectiveness of business decisions. However, due to the wide and diverse sources of data, various problems exist in data collection, storage, transmission and other links, resulting in increasingly prominent data quality issues. Therefore, how to effectively monitor and manage data quality has become one of the current problems that need to be solved urgently.

[0003] However, traditional data quality monitoring and management methods often rely on manual review and empirical judgment, which is not only time-consuming and labor-intensive, inefficient, but also prone to misjudgment and missed detection. In addition, traditional data quality monitoring methods are usually periodic, that is, data quality checks are performed regularly. This means that even if there is a problem with the data, it may take some time to be discovered and resolved, which is unacceptable in an environment that requires rapid response. Not only that, existing data quality monitoring systems can often only perform quality assessments on data of a specific type or format, lacking versatility and flexibility; secondly, they usually require users to manually configure a large number of parameters and rules, which requires professional knowledge and technical support, making it difficult for non-technical personnel to participate, increasing the difficulty of implementation and the complexity of operation.

[0004] Therefore, an optimized data quality monitoring system is desired. Summary of the invention

[0005] The present application provides a data quality monitoring system and method, which can automatically acquire and process data of various types and formats to obtain a data quality inspection report, and use multiple data quality rules to analyze the data quality inspection report to achieve quality assessment of source data, thereby determining its quality grade label. In this way, data quality can be monitored and managed in real time in an automated manner, so that when problems are found, corresponding measures can be taken to ensure data quality and business continuity.

[0006] In a first aspect, a data quality monitoring system is provided, comprising: a data quality rule acquisition module, used to define data quality rules to obtain a set of data quality rules; a data quality check report generation module, used to perform data exploration and analysis on source data to obtain a data quality check report; a source data quality grade assessment module, used to perform quality assessment on the data quality check report based on the set of data quality rules to obtain a quality assessment result, wherein the quality assessment result is used to represent a quality grade label of the source data.

[0007] In a possible implementation, the source data quality level assessment module includes: a data quality rule semantic encoding unit, which is used to semantically encode each data quality rule in the set of data quality rules to obtain a set of data quality rule semantic encoding vectors; a data quality inspection report semantic encoding unit, which is used to semantically encode the data quality inspection report to obtain a data quality report semantic encoding vector; a data quality report-quality rule query response optimization encoding unit, which is used to perform semantic feature-significant guided query response processing on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain a data quality report-quality rule query response optimization encoding unit. ization coding features, wherein the data quality report-quality rule query response optimization coding unit includes: a data quality rule feature aggregation processing subunit, used to perform feature aggregation processing based on linear embedding coding on the set of data quality rule semantic coding vectors to obtain a prompt template; a cross-domain optimization query coding unit, used to perform cross-domain optimization query coding on the data quality report semantic coding vector and the set of data quality rule semantic coding vectors based on the prompt template to obtain the data quality report-quality rule query response coding features; a quality assessment result generation unit, used to determine the quality assessment result based on the data quality report-quality rule query response coding features.

[0008] In one possible implementation, the data quality rule feature aggregation processing subunit includes: a data quality rule linear embedding coding secondary subunit, which is used to use a key embedding matrix to perform linear embedding coding on each data quality rule semantic coding vector in the set of data quality rule semantic coding vectors to obtain a set of data quality rule semantic coding vectors after linear transformation; a data quality rule matrix arrangement processing secondary subunit, which is used to perform matrix arrangement on the set of data quality rule semantic coding vectors after linear transformation to obtain a data quality rule semantic key matrix; and a prompt template generation secondary subunit, which is used to extract the maximum value of each linearly transformed data quality rule semantic coding vector in the data quality rule semantic key matrix to obtain a significant feature vector of the data quality rule semantic key matrix as the prompt template.

[0009] In a possible implementation, the linear embedding coding secondary subunit is used to: multiply the data quality rule semantic coding vector by the key embedding matrix and then add it to the key embedding bias vector by position to obtain the linearly transformed data quality rule semantic coding vector.

[0010] In one possible implementation, the cross-domain optimization query encoding unit includes: a data quality report linear embedding encoding secondary sub-unit, which is used to use a query embedding matrix and a value embedding matrix to perform linear embedding encoding on the data quality report semantic encoding vector to obtain a data quality report semantic query vector and a data quality report semantic value vector; a template prompt optimization processing secondary sub-unit, which is used to input the data quality report semantic query vector, the data quality report semantic value vector, the data quality rule semantic encoding vectors after each linear transformation in the data quality rule semantic key matrix, and the prompt template into a heterogeneous converter structure based on template prompt optimization to obtain a sequence of data quality report-quality rule semantic cross-domain optimization query encoding vectors; a query response encoding secondary sub-unit, which is used to calculate the positional mean vector of the sequence of data quality report-quality rule semantic cross-domain optimization query encoding vectors to obtain a data quality report-quality rule query response encoding vector as the data quality report-quality rule query response encoding feature.

[0011] In one possible implementation, the template prompt optimization processing secondary sub-unit is used to: calculate the multiplication between the data quality report semantic query vector and the transposed vector of the data quality rule semantic encoding vector after the linear transformation, and then divide the obtained data quality report-quality rule semantic association feature matrix by the bi-norm of the significant feature vector of the data quality rule semantic key matrix by position to obtain the data quality report-quality rule semantic association representation matrix; input the data quality report-quality rule semantic association representation matrix into a function for processing to obtain the data quality report-quality rule semantic association weight matrix; multiply the data quality report-quality rule semantic association weight matrix by the significant feature vector of the data quality rule semantic key matrix, and then perform positional point multiplication of the obtained feature vector with the data quality report semantic value vector to obtain the data quality report-quality rule semantic cross-domain optimization query encoding vector.

[0012] In a possible implementation, the quality assessment result generating unit is used to: input the data quality report-quality rule query response encoding vector into a classifier-based data quality assessment module to obtain the quality assessment result.

[0013] In a second aspect, a data quality monitoring method is provided, comprising: defining data quality rules to obtain a set of data quality rules; performing data exploration and analysis on source data to obtain a data quality inspection report; based on the set of data quality rules, performing quality assessment on the data quality inspection report to obtain a quality assessment result, wherein the quality assessment result is used to represent a quality grade label of the source data.

[0014] In one possible implementation, based on the set of data quality rules, the data quality check report is quality evaluated to obtain a quality evaluation result, including: semantically encoding each data quality rule in the set of data quality rules to obtain a set of data quality rule semantic encoding vectors; semantically encoding the data quality check report to obtain a data quality report semantic encoding vector; performing semantic feature-significant guidance query response processing on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain data quality report-quality rule query response optimization coding features, including: performing feature aggregation processing based on linear embedding coding on the set of data quality rule semantic encoding vectors to obtain a prompt template; based on the prompt template, performing cross-domain optimized query encoding on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain the data quality report-quality rule query response coding features; based on the data quality report-quality rule query response coding features, determining the quality evaluation result.

[0015] In one possible implementation, the set of data quality rule semantic coding vectors is subjected to feature aggregation processing based on linear embedding coding to obtain a prompt template, including: using a key embedding matrix to perform linear embedding coding on each data quality rule semantic coding vector in the set of data quality rule semantic coding vectors to obtain a set of linearly transformed data quality rule semantic coding vectors; arranging the set of linearly transformed data quality rule semantic coding vectors in a matrix to obtain a data quality rule semantic key matrix; extracting the maximum value of each linearly transformed data quality rule semantic coding vector in the data quality rule semantic key matrix to obtain a significant feature vector of the data quality rule semantic key matrix as the prompt template.

[0016] The present application provides a data quality monitoring system and method, which includes: a data quality rule acquisition module, which is used to define data quality rules to obtain a set of data quality rules; a data quality inspection report generation module, which is used to perform data exploration and analysis on source data to obtain a data quality inspection report; a source data quality grade assessment module, which is used to perform quality assessment on the data quality inspection report based on the set of data quality rules to obtain a quality assessment result, and the quality assessment result is used to represent the quality grade label of the source data. It can automatically acquire and process data of various types and formats to obtain a data quality inspection report, and use multiple data quality rules to analyze the data quality inspection report to achieve quality assessment of the source data, thereby determining its quality grade label. In this way, data quality can be monitored and managed in real time in an automated manner, so that when problems are found, it can respond quickly and take corresponding measures to ensure data quality and business continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0018] Figure 1 A schematic block diagram of a data quality monitoring system according to an embodiment of the present application.

[0019] Figure 2 It is a schematic block diagram of a source data quality level evaluation module in a data quality monitoring system according to an embodiment of the present application.

[0020] Figure 3 Schematic diagram of data flow in a source data quality level assessment module in a data quality monitoring system according to an embodiment of the present application.

[0021] Figure 4 It is a schematic block diagram of a data quality report-quality rule query response optimization coding unit in a data quality monitoring system in an embodiment of the present application.

[0022] Figure 5 A schematic flowchart of a data quality monitoring method according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0024] Traditionally, data quality management relies on manual review and decision-making methods based on personal experience. This approach not only consumes a lot of time and human resources, but also may lead to misjudgment or omission of problems due to the limitations of human cognition. In addition, traditional data quality management usually runs in a periodic manner, that is, data quality inspection tasks are performed at predetermined time intervals. A significant disadvantage of this approach is that even if deviations or anomalies occur in the data, they need to wait for the next scheduled checkpoint to be detected and processed, which is obviously not timely enough for business scenarios that require immediate response. In addition, many existing data quality management tools are also limited in applicability, because they are often only applicable to specific types or structured data sets, and cannot flexibly respond to changes in multiple data formats.

[0025] In view of the above technical problems, in the technical solution of the present application, a data quality monitoring system is proposed, which includes: a data quality rule acquisition module, which is used to define data quality rules to obtain a set of data quality rules; a data quality inspection report generation module, which is used to perform data exploration and analysis on the source data to obtain a data quality inspection report; a source data quality grade evaluation module, which is used to perform quality evaluation on the data quality inspection report based on the set of data quality rules to obtain a quality evaluation result, and the quality evaluation result is used to represent the quality grade label of the source data. The data quality monitoring system can automatically acquire and process data of various types and formats to obtain a data quality inspection report, and use multiple data quality rules to analyze the data quality inspection report to achieve quality evaluation of the source data, thereby determining its quality grade label. In this way, the data quality can be monitored and managed in real time in an automated manner, so that when problems are found, it can respond quickly and take corresponding measures to ensure data quality and business continuity.

[0026] Figure 1 FIG. 1 is a schematic block diagram of a data quality monitoring system according to an embodiment of the present application. Figure 1 As shown, the data quality monitoring system 100 includes: a data quality rule acquisition module 110, used to define data quality rules to obtain a set of data quality rules; a data quality inspection report generation module 120, used to perform data exploration and analysis on source data to obtain a data quality inspection report; a source data quality grade assessment module 130, used to perform quality assessment on the data quality inspection report based on the set of data quality rules to obtain a quality assessment result, and the quality assessment result is used to represent the quality grade label of the source data.

[0027] In the above-mentioned data quality monitoring system, the data quality rule acquisition module 110 is used to define data quality rules to obtain a set of data quality rules. It should be understood that data quality rules are defined according to data processing and quality inspection requirements, and the data quality rules include null value verification, uniqueness verification, vector point position rationality verification, vector surface topology rule verification, value range verification, data format verification, field annotation verification, China longitude verification, China latitude verification, IP address verification, global longitude verification, global latitude verification, mailbox verification, date and time verification, comparison verification, regular verification, length verification, null value filtering, MAC address verification, ID card verification, mobile phone number verification, integer verification, unified social credit code verification, organization code verification, landline number verification, coordinate system standardization and value range rationality, and data quality inspection services are performed through data quality rules. Among them, the data quality rules are defined according to objective standards. Optionally, in one embodiment of the present application, the following data quality rule definition examples are provided. First, null value verification: there should be no unfilled fields in the data set. For example, in a customer information table, the "age" field of all customers must have a value. 2. Uniqueness check: Some fields should have unique values ​​in the entire data set. For example, in a user database, each entry in the "email address" field should be unique. 3. Vector point location rationality check: The location of the vector point should be within a reasonable geographical range. 4. Vector surface topology rule check: Ensure that the topological structure of the vector surface (polygon) is correct without overlap or gaps. 5. Value domain verification: The value in the field should conform to the preset range. For example, the value of the "age" field should be between one and one hundred and twenty years old. 6. Data format verification: The data in the field should conform to specific format requirements. For example, the "date time" field should follow the annotation format. 7. Field annotation verification: Ensure that each field has a clear and accurate description or annotation. For example, the annotation of the "height (cm)" field should indicate that the unit is centimeters. 8. China longitude / latitude verification: For geographic location data located in China, longitude and latitude should conform to objective facts. 9. Global longitude / latitude verification: Longitude should be between -180 and 180, and latitude should be between -90 and 90. 10. Email verification: The email address should conform to the standard format of email. 11. Date and time validation: Confirm that the date and time format is correct and reasonable, such as no future dates or dates that could not have existed in the past. 12. Comparison validation: Compare two related fields to ensure logical consistency. For example, if a person's date of birth is after their date of death, the record is invalid. 13. Regular expression validation: Use regular expressions to verify that a string conforms to the expected pattern. For example, a phone number should match the corresponding national phone number pattern.

[0028] In the above data quality monitoring system, the data quality inspection report generation module 120 is used to perform data exploration and analysis on the source data to obtain a data quality inspection report. It should be understood that the business data in the form of aggregated database tables, files and interfaces is explored, and the source data storage location, provision method, total amount and update status, business meaning, field format semantics, data structure, data quality, etc. are analyzed in multiple dimensions through access method exploration, business exploration, field exploration, data set exploration, problem data exploration, etc., so as to achieve the purpose of understanding the data. Through data exploration, data that does not meet the specifications can be found, and a corresponding data quality inspection report can be issued, which will be fed back to the relevant data source and provide a basis for the formulation of subsequent data processing rules.

[0029] In the above-mentioned data quality monitoring system, the source data quality grade assessment module 130 is used to perform quality assessment on the data quality inspection report based on the set of data quality rules to obtain a quality assessment result, and the quality assessment result is used to represent the quality grade label of the source data. It should be understood that in the data quality monitoring system, evaluating the quality grade of the source data is a key link. The overall quality level of the source data is determined by comparing the data quality inspection report after data exploration and analysis with the pre-defined data quality rules. The result of the assessment is usually assigned a quality grade label to intuitively represent the quality status of the source data.

[0030] Specifically, the technical concept of the present application is to use data processing and semantic understanding algorithms based on artificial intelligence and deep learning to perform semantic analysis on a set of data quality rules and data quality inspection reports, so as to capture the semantics of data quality reports and multiple data quality rule semantics, and automatically evaluate the quality level of source data based on the query response encoding representation between the two semantics, thereby generating a quality evaluation result. In this way, data quality can be monitored and evaluated in an automated manner, and an evaluation result of the data quality level can be generated, so as to promptly discover and process data that does not meet quality standards, and take necessary measures to ensure data quality, thereby supporting more effective business decisions.

[0031] Figure 2 It is a schematic block diagram of a source data quality level evaluation module in a data quality monitoring system according to an embodiment of the present application. Figure 3 This is a schematic diagram of data flow in the source data quality level assessment module in the data quality monitoring system of the embodiment of the present application. Figure 2 and Figure 3As shown, the source data quality level assessment module 130 includes: a data quality rule semantic encoding unit 131, which is used to semantically encode each data quality rule in the set of data quality rules to obtain a set of data quality rule semantic encoding vectors; a data quality check report semantic encoding unit 132, which is used to semantically encode the data quality check report to obtain a data quality report semantic encoding vector; a data quality report-quality rule query response optimization encoding unit 133, which is used to perform semantic feature-significant guided query response processing on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain a data quality report-quality rule query response optimization encoding feature; a quality assessment result generation unit 134, which is used to determine the quality assessment result based on the data quality report-quality rule query response encoding feature.

[0032] In the above-mentioned data quality monitoring system, the data quality rule semantic encoding unit 131 is used to semantically encode each data quality rule in the set of data quality rules to obtain a set of data quality rule semantic encoding vectors. It should be understood that by means of semantic encoding, the semantic feature information in each data quality rule can be extracted, such as null value check, uniqueness check, vector point position rationality check, vector surface topology rule check, value range verification, data format verification and other rule semantics. In particular, semantic encoding can capture subtle differences between rules, thereby improving the accuracy of data quality assessment. For example, it can distinguish the subtle differences between "the value should be greater than zero" and "the value should be greater than or equal to zero", so as to adapt to the ever-changing data and business needs, and help to better perform data quality assessment.

[0033] Optionally, in one embodiment of the present application, each data quality rule in the set of data quality rules is semantically encoded to obtain a set of data quality rule semantic encoding vectors, including: cleaning each data quality rule in the set of data quality rules, removing irrelevant symbols, and unifying the format. The cleaned data quality rules are segmented to divide long sentences into words or phrases. A pre-trained word embedding model (such as Word2Vec, GloVe, etc.) is used to map each word to a vector in a high-dimensional space. A converter-based model (such as BERT) is used to perform global contextual semantic encoding on the word embedding sequence to obtain a data quality rule semantic encoding vector.

[0034] In the above-mentioned data quality monitoring system, the data quality inspection report semantic encoding unit 132 is used to semantically encode the data quality inspection report to obtain a data quality report semantic encoding vector. It should be understood that since the data quality inspection report is obtained by exploring the source data, it contains data that does not meet the specifications that are explored and analyzed from multiple different angles (such as storage location, provision method, total amount and update status, business meaning, field format semantics, and data structure). Therefore, semantic encoding of the data quality inspection report can further understand the data quality semantic features contained in the data quality inspection report, which is conducive to providing a basis for subsequent quality level evaluation.

[0035] In the above-mentioned data quality monitoring system, the data quality report-quality rule query response optimization coding unit 133 is used to perform semantic feature-significant guidance query response processing on the data quality report semantic coding vector and the set of data quality rule semantic coding vectors to obtain data quality report-quality rule query response optimization coding features. It should be understood that the data quality report semantic coding vector and the set of data quality rule semantic coding vectors respectively contain the semantic coding features of the data quality report and multiple data quality rule semantic coding features. In order to be able to use the semantics of each data quality rule to query and match the semantic feature information related to data quality in the data quality report semantic coding vector, thereby generating a data quality assessment semantic representation that conforms to these data quality rules, so as to more comprehensively and accurately judge the quality level of the source data, in the technical solution of the present application, the data quality report semantic coding vector and the set of data quality rule semantic coding vectors are further subjected to semantic feature-significant guidance query response processing to obtain data quality report-quality rule query response optimization coding features. By subjecting the semantic coding vector of the data quality report and the set of semantic coding vectors of the data quality rules to semantic feature-significant guidance query response processing, more accurate matching and feature query can be achieved. This method uses the semantics of various data quality rules to constrain and optimize the semantic expression of data quality reports, thereby helping the system to identify and query feature expressions related to the semantics of data quality rules from data quality inspection reports, so as to provide a basis for subsequent data quality assessment.

[0036] Optionally, in one embodiment of the present application, Figure 4 Schematic block diagram of a data quality report-quality rule query response optimization coding unit in a data quality monitoring system of an embodiment of the present application. Figure 4As shown, the data quality report-quality rule query response optimization coding unit 133 includes: a data quality rule feature aggregation processing subunit 1331, which is used to perform feature aggregation processing based on linear embedding coding on the set of data quality rule semantic coding vectors to obtain a prompt template; a cross-domain optimization query coding unit 1332, which is used to perform cross-domain optimization query coding on the set of the data quality report semantic coding vector and the data quality rule semantic coding vector based on the prompt template to obtain the data quality report-quality rule query response coding feature.

[0037] Optionally, in one embodiment of the present application, the data quality rule feature aggregation processing subunit includes: a data quality rule linear embedding coding secondary subunit, which is used to use a key embedding matrix to perform linear embedding coding on each data quality rule semantic coding vector in the set of data quality rule semantic coding vectors to obtain a set of data quality rule semantic coding vectors after linear transformation; a data quality rule matrix arrangement processing secondary subunit, which is used to perform matrix arrangement on the set of data quality rule semantic coding vectors after linear transformation to obtain a data quality rule semantic key matrix; and a prompt template generation secondary subunit, which is used to extract the maximum value of each linearly transformed data quality rule semantic coding vector in the data quality rule semantic key matrix to obtain a significant feature vector of the data quality rule semantic key matrix as the prompt template.

[0038] Optionally, in one embodiment of the present application, the linear embedding coding secondary sub-unit is used to: multiply the data quality rule semantic coding vector by the key embedding matrix and then add it positionally with the key embedding bias vector to obtain the linearly transformed data quality rule semantic coding vector.

[0039] Optionally, in one embodiment of the present application, the cross-domain optimization query encoding unit includes: a data quality report linear embedding encoding secondary sub-unit, which is used to use a query embedding matrix and a value embedding matrix to perform linear embedding encoding on the data quality report semantic encoding vector to obtain a data quality report semantic query vector and a data quality report semantic value vector; a template prompt optimization processing secondary sub-unit, which is used to input the data quality report semantic query vector, the data quality report semantic value vector, the data quality rule semantic encoding vectors after each linear transformation in the data quality rule semantic key matrix, and the prompt template into a heterogeneous converter structure based on template prompt optimization to obtain a sequence of data quality report-quality rule semantic cross-domain optimization query encoding vectors; a query response encoding secondary sub-unit, which is used to calculate the positional mean vector of the sequence of data quality report-quality rule semantic cross-domain optimization query encoding vectors to obtain a data quality report-quality rule query response encoding vector as the data quality report-quality rule query response encoding feature.

[0040] Optionally, in one embodiment of the present application, the template prompt optimization processing secondary subunit is used to: calculate the multiplication between the data quality report semantic query vector and the transposed vector of the linearly transformed data quality rule semantic encoding vector, and then divide the obtained data quality report-quality rule semantic association feature matrix by the bi-norm of the significant feature vector of the data quality rule semantic key matrix by position to obtain a data quality report-quality rule semantic association representation matrix; input the data quality report-quality rule semantic association representation matrix The data quality report - quality rule semantic association weight matrix is ​​processed in the function; after multiplying the data quality report - quality rule semantic association weight matrix with the significant feature vector of the data quality rule semantic key matrix, the obtained feature vector is multiplied by the data quality report semantic value vector at the position point to obtain the data quality report - quality rule semantic cross-domain optimization query encoding vector.

[0041] In summary, in the embodiment of the present application, the set of the data quality report semantic coding vector and the data quality rule semantic coding vector is subjected to semantic feature inter-query response optimization coding using the following feature response interaction formula to obtain the data quality report-quality rule query response coding feature; wherein the feature response interaction formula is: ;in, is the set of semantic encoding vectors of the data quality rules, are the first, second and third vectors in the set of semantic encoding vectors of the data quality rules. data quality rule semantic encoding vector, is the first in the set of semantic encoding vectors of the data quality rules data quality rule semantic encoding vector, and are the key embedding matrix and key embedding bias vector respectively, is the first in the set of semantic encoding vectors of data quality rules after linear transformation The semantic encoding vector of data quality rules after linear transformation, is the data quality rule semantic key matrix, To extract the maximum value in a vector, is the significant feature vector of the semantic key matrix of data quality rules, a semantic encoding vector for said data quality report, and are the query embedding matrix and the query bias vector respectively, and are the value embedding matrix and value bias vector respectively, Semantic query vectors for data quality reporting, Report semantic value vectors for data quality, is the two-norm of the vector, for function, is matrix multiplication, For point-by-position multiplication, The first in the sequence of query encoding vectors for data quality report-quality rule semantic cross-domain optimization Data quality report - quality rule semantic cross-domain optimization query encoding vector, the number of vectors in the sequence of query encoding vectors for the data quality report-quality rule semantic cross-domain optimization, A vector is encoded for the data quality report-quality rule query response.

[0042] Specifically, the process of significantly guiding query response processing between semantic features includes: first, linearly embedding and encoding each data quality rule semantic encoding vector in the set of data quality rule semantic encoding vectors to perform a linear transformation operation. This process maps the original data quality rule semantic feature space to a new latent space, in order to capture the relationship between different data quality rule semantics and better represent the complex patterns in these data semantics. Next, these linearly transformed data quality rule semantic encoding vectors are arranged in order to form a new data structure called a key matrix for parallel processing, thereby improving computational efficiency. Next, the maximum value of each column (i.e., each linearly transformed data quality rule semantic encoding vector) is selected from the data quality rule semantic key matrix to obtain the significant feature vector of the data quality rule semantic key matrix. Then, the significant feature vector of the key matrix is ​​used as a prompt template, representing the most prominent or representative part of the input data quality rule semantic sequence. The significant feature vector of the data quality rule semantic key matrix is ​​introduced as an additional guiding signal, which helps to reduce noise interference and strengthen the model's learning of important patterns. At the same time, it can also help the attention mechanism focus more on the key part of the data quality rule semantic data, thereby improving the model's attention to important details. Further, the query embedding matrix and the value embedding matrix are respectively applied to the data quality report semantic encoding vector to generate corresponding query vector and value vector. The two vectors each play different but complementary roles: the query vector is used to determine which data quality report semantic information needs to be paid attention to, while the value vector carries the actual data quality report semantic content. The combination of the two can accurately select relevant information while maintaining the context, thereby enhancing the generalization ability and accuracy of the model. Subsequently, the data quality report semantic query vector, the data quality report semantic value vector, the data quality rule semantic key matrix and the prompt template are jointly fed into a specially designed heterogeneous transformer structure, which usually includes a heterogeneous multi-head attention mechanism component, which can automatically adjust the weight allocation strategy according to the provided prompt template, so as to better capture the complex relationship between data. This method can enhance the model's ability to handle long-distance dependency problems and ensure that global information is taken into account when evaluating data quality. The last step is to average the sequence of the data quality report-quality rule semantic cross-domain optimization query encoding vector according to the position, and finally obtain the data quality report-quality rule query response encoding vector as the data quality report-quality rule query response encoding feature. This step not only simplifies the form of the final result, making it more suitable for further analysis and decision-making, but also retains the comprehensive characteristics of all position information in the sequence, ensuring the comprehensiveness and accuracy of the data quality assessment results and quality grade determination results.

[0043] In the above-mentioned data quality monitoring system, the quality assessment result generating unit 134 is used to determine the quality assessment result based on the data quality report-quality rule query response coding feature. Optionally, in one embodiment of the present application, the quality assessment result generating unit is used to: input the data quality report-quality rule query response coding vector into a classifier-based data quality assessment module to obtain the quality assessment result. That is to say, the query response optimization coding feature information between the data quality report semantic feature and the quality rule semantic feature is used for classification processing, so as to automatically evaluate the quality level of the source data, thereby generating a quality assessment result. In this way, the data quality can be monitored and evaluated in an automated manner, and the evaluation result of the data quality level can be generated, so as to timely discover and process the data that does not meet the quality standards, and take necessary measures to ensure the data quality, thereby supporting more effective business decisions.

[0044] Here, in the case where the data quality report semantic coding vector and the set of data quality rule semantic coding vectors respectively represent the coded semantic features of the data quality check report and the set of data quality rules, when performing query coding optimization using the key matrix significant features as prompt templates, the semantic feature correspondence gap will also cause the key matrix significant distribution to become sparse, resulting in large-scale simple repetition of the distribution prompts of the coded semantic features, affecting the query coding semantic logical dependency and reducing the accuracy of the classification results based on the semantic distribution.

[0045] Therefore, the present application hopes to improve the overall query encoding semantic consistency aggregation expression effect of the data quality report-quality rule query response encoding vector when the data quality report-quality rule query response encoding vector is input into a classifier-based data quality assessment module to obtain the quality assessment result, thereby optimizing it.

[0046] In a preferred example, the data quality report-quality rule query response encoding vector is input into a classifier-based data quality assessment module to obtain the quality assessment result, including: performing feature clustering on all feature values ​​of the data quality report-quality rule query response encoding vector to obtain a data quality report-quality rule query response intra-class feature set and Data Quality Report - Quality Rule Query Response Out-of-Class Feature Set ,in, Represents the feature set within the data quality report-quality rule query response class, The first feature set representing the data quality report-quality rule query response class eigenvalues, The first one represents the out-of-class feature set of data quality report-quality rule query response feature values.

[0047] Calculate the ratio of the number of feature values ​​in the feature set within the data quality report-quality rule query response class to the number of feature values ​​of all features of the data quality report-quality rule query response encoding vector ,in, represents the number of feature values ​​in the feature set within the data quality report-quality rule query response class, The number of feature values ​​representing all features of the data quality report-quality rule query response encoding vector, Indicates a ratio.

[0048] Calculate the sum of the absolute values ​​of all feature values ​​in the feature set within the data quality report-quality rule query response class The sum of the absolute values ​​of all eigenvalues ​​in the data quality report-quality rule query response encoding vector The ratio of the first and second powers is used to obtain the first data quality report-quality rule query response weight.

[0049] Calculate the sum of squares of all feature values ​​in the feature set of the data quality report-quality rule query response class The power of the sum of the squares of all eigenvalues ​​in the data quality report-quality rule query response encoding vector The ratio of the two powers is used to obtain the second data quality report-quality rule query response weight.

[0050] That is, the first data quality report-quality rule query response weight and the second data quality report-quality rule query response weight It is expressed as: ;in, represents each characteristic value of the data quality report-quality rule query response encoding vector, represents the first data quality report-quality rule query response weight, Represents the second data quality report-quality rule query response weight.

[0051] Then, for each feature value in the feature set within the data quality report-quality rule query response class, the product of the feature value and the second data quality report-quality rule query response weight is calculated, and then the product of the feature value and the first data quality report-quality rule query response weight is added to obtain the optimized feature value. ,in, Indicates the first feature set in the query response class based on the data quality report-quality rule Optimize the eigenvalues.

[0052] For each feature value in the data quality report-quality rule query response out-of-class feature set, calculate the product of the feature value and the first data quality report-quality rule query response weight to obtain an optimized feature value ,in, represents the first out-of-class feature set based on the data quality report-quality rule query response Optimize the eigenvalues.

[0053] Finally, the optimized feature values ​​based on the feature set within the data quality report-quality rule query response class and the feature set outside the data quality report-quality rule query response class are used to form an optimized data quality report-quality rule query response encoding vector, and the optimized data quality report-quality rule query response encoding vector is input into a classifier-based data quality assessment module to obtain the quality assessment result.

[0054] That is, while the data quality report-quality rule query response coding vector is subjected to feature clustering, the interaction description of the key feature information of the data quality report-quality rule query response coding vector in the clustering process is carried out, and the geometric equivariant topology of the feature is constructed by low-rank harmonic modulation of the features on the equivariant basis of the clustering features of the data quality report-quality rule query response coding vector and the features as a whole, so as to obtain the graphical distribution translation and rotation symmetry of the clustering features of the data quality report-quality rule query response coding vector relative to the features as a whole, thereby introducing geometric message passing in the feature expression of the data quality report-quality rule query response coding vector and realizing the cluster mapping symmetry of the data quality report-quality rule query response coding vector through manipulation of irreducible low-rank coefficients, thereby improving the consistency of the clustering-based feature representation of the data quality report-quality rule query response coding vector, thereby improving the accuracy of the quality assessment result obtained by inputting the data quality report-quality rule query response coding vector into the classifier-based data quality assessment module. In this way, data quality can be monitored and evaluated in an automated manner, and evaluation results of data quality levels can be generated, so that data that does not meet quality standards can be discovered and processed in a timely manner, and necessary measures can be taken to ensure data quality, thereby supporting more effective business decisions.

[0055] In summary, the data quality monitoring method according to the embodiment of the present application is explained, which includes: a data quality rule acquisition module, which is used to define data quality rules to obtain a set of data quality rules; a data quality inspection report generation module, which is used to perform data exploration and analysis on the source data to obtain a data quality inspection report; a source data quality grade assessment module, which is used to perform quality assessment on the data quality inspection report based on the set of data quality rules to obtain a quality assessment result, and the quality assessment result is used to represent the quality grade label of the source data. It can automatically acquire and process data of various types and formats to obtain a data quality inspection report, and use multiple data quality rules to analyze the data quality inspection report to achieve quality assessment of the source data, thereby determining its quality grade label. In this way, data quality can be monitored and managed in real time in an automated manner, so that when problems are found, it can respond quickly and take corresponding measures to ensure data quality and business continuity.

[0056] Figure 5 Schematic flow chart of the data quality monitoring method of the embodiment of the present application. Figure 5 As shown, the data quality monitoring method includes: S1, defining data quality rules to obtain a set of data quality rules; S2, performing data exploration and analysis on source data to obtain a data quality inspection report; S3, based on the set of data quality rules, performing quality assessment on the data quality inspection report to obtain a quality assessment result, and the quality assessment result is used to represent the quality grade label of the source data.

[0057] Optionally, in one embodiment of the present application, based on the set of data quality rules, the data quality check report is quality evaluated to obtain a quality evaluation result, including: semantically encoding each data quality rule in the set of data quality rules to obtain a set of data quality rule semantic encoding vectors; semantically encoding the data quality check report to obtain a data quality report semantic encoding vector; performing semantic feature-significant guidance query response processing on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain a data quality report-quality rule query response optimization coding feature, including: performing feature aggregation processing based on linear embedding coding on the set of data quality rule semantic encoding vectors to obtain a prompt template; based on the prompt template, performing cross-domain optimized query encoding on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain the data quality report-quality rule query response coding feature; based on the data quality report-quality rule query response coding feature, determining the quality evaluation result.

[0058] Optionally, in one embodiment of the present application, the set of data quality rule semantic coding vectors is subjected to feature aggregation processing based on linear embedding coding to obtain a prompt template, including: using a key embedding matrix to perform linear embedding coding on each data quality rule semantic coding vector in the set of data quality rule semantic coding vectors to obtain a set of linearly transformed data quality rule semantic coding vectors; arranging the set of linearly transformed data quality rule semantic coding vectors in a matrix to obtain a data quality rule semantic key matrix; extracting the maximum value of each linearly transformed data quality rule semantic coding vector in the data quality rule semantic key matrix to obtain a significant feature vector of the data quality rule semantic key matrix as the prompt template.

[0059] Here, those skilled in the art can understand that the specific operations of each step in the above data quality monitoring method have been referred to above. Figures 1 to 4 The data quality monitoring system has been described in detail, and therefore, its repeated description will be omitted.

[0060] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0061] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the module division is only a logical function division, and there may be other division methods in actual implementation. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0062] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0063] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.

[0064] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units stated in the device claim can also be implemented by one unit through software or hardware.

[0065] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A data quality monitoring system, characterized in that: include: A data quality rule acquisition module, used for defining data quality rules to obtain a set of data quality rules; A data quality inspection report generation module is used to perform data exploration and analysis on source data to obtain a data quality inspection report; The source data quality grade assessment module is used to perform quality assessment on the data quality inspection report based on the set of data quality rules to obtain a quality assessment result, and the quality assessment result is used to represent the quality grade label of the source data.

2. The data quality monitoring system according to claim 1, characterized in that: The source data quality level assessment module includes: a data quality rule semantic encoding unit, which is used to semantically encode each data quality rule in the set of data quality rules to obtain a set of data quality rule semantic encoding vectors; a data quality inspection report semantic encoding unit, which is used to semantically encode the data quality inspection report to obtain a data quality report semantic encoding vector; a data quality report-quality rule query response optimization encoding unit, which is used to perform semantic feature-significant guidance query response processing on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain a data quality report-quality rule query response optimization encoding feature, Among them, the data quality report-quality rule query response optimization coding unit includes: a data quality rule feature aggregation processing subunit, which is used to perform feature aggregation processing based on linear embedding coding on the set of data quality rule semantic coding vectors to obtain a prompt template; a cross-domain optimization query coding unit, which is used to perform cross-domain optimization query coding on the data quality report semantic coding vector and the set of data quality rule semantic coding vectors based on the prompt template to obtain the data quality report-quality rule query response coding characteristics; a quality assessment result generation unit, which is used to determine the quality assessment result based on the data quality report-quality rule query response coding characteristics.

3. The data quality monitoring system according to claim 2, characterized in that: The data quality rule feature aggregation processing subunit includes: a data quality rule linear embedding coding secondary subunit, which is used to use a key embedding matrix to perform linear embedding coding on each data quality rule semantic coding vector in the set of data quality rule semantic coding vectors to obtain a set of data quality rule semantic coding vectors after linear transformation; a data quality rule matrix arrangement processing secondary subunit, which is used to perform matrix arrangement on the set of data quality rule semantic coding vectors after linear transformation to obtain a data quality rule semantic key matrix; and a prompt template generation secondary subunit, which is used to extract the maximum value of each linearly transformed data quality rule semantic coding vector in the data quality rule semantic key matrix to obtain a significant feature vector of the data quality rule semantic key matrix as the prompt template.

4. The data quality monitoring system according to claim 3, characterized in that: The linear embedding coding secondary subunit is used to: multiply the data quality rule semantic coding vector by the key embedding matrix and then add the key embedding bias vector by position to obtain the linearly transformed data quality rule semantic coding vector.

5. The data quality monitoring system according to claim 4, characterized in that: The cross-domain optimization query encoding unit includes: a data quality report linear embedding encoding secondary sub-unit, which is used to use a query embedding matrix and a value embedding matrix to perform linear embedding encoding on the data quality report semantic encoding vector to obtain a data quality report semantic query vector and a data quality report semantic value vector; a template prompt optimization processing secondary sub-unit, which is used to input the data quality report semantic query vector, the data quality report semantic value vector, the data quality rule semantic encoding vectors after each linear transformation in the data quality rule semantic key matrix, and the prompt template into a heterogeneous converter structure based on template prompt optimization to obtain a sequence of data quality report-quality rule semantic cross-domain optimization query encoding vectors; a query response encoding secondary sub-unit, which is used to calculate the positional mean vector of the sequence of data quality report-quality rule semantic cross-domain optimization query encoding vectors to obtain a data quality report-quality rule query response encoding vector as the data quality report-quality rule query response encoding feature.

6. The data quality monitoring system according to claim 5, characterized in that: The template prompt optimization processing secondary subunit is used to: calculate the multiplication between the data quality report semantic query vector and the transposed vector of the linearly transformed data quality rule semantic encoding vector, and then divide the obtained data quality report-quality rule semantic association feature matrix by the bi-norm of the significant feature vector of the data quality rule semantic key matrix to obtain a data quality report-quality rule semantic association representation matrix; input the data quality report-quality rule semantic association representation matrix The data quality report - quality rule semantic association weight matrix is ​​processed in the function; after multiplying the data quality report - quality rule semantic association weight matrix with the significant feature vector of the data quality rule semantic key matrix, the obtained feature vector is multiplied by the data quality report semantic value vector at the position point to obtain the data quality report - quality rule semantic cross-domain optimization query encoding vector.

7. The data quality monitoring system according to claim 6, characterized in that: The quality assessment result generating unit is used to: input the data quality report-quality rule query response encoding vector into a classifier-based data quality assessment module to obtain the quality assessment result.

8. A data quality monitoring method, characterized in that: include: defining data quality rules to obtain a set of data quality rules; Conduct data exploration and analysis on source data to obtain data quality inspection reports; Based on the set of data quality rules, a quality assessment is performed on the data quality check report to obtain a quality assessment result, and the quality assessment result is used to represent a quality grade label of the source data.

9. The data quality monitoring method according to claim 8, characterized in that: Based on the set of data quality rules, the data quality check report is quality evaluated to obtain a quality evaluation result, including: semantically encoding each data quality rule in the set of data quality rules to obtain a set of data quality rule semantic encoding vectors; semantically encoding the data quality check report to obtain a data quality report semantic encoding vector; performing semantic feature-significant guidance query response processing on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain data quality report-quality rule query response optimization coding features, including: performing feature aggregation processing based on linear embedding coding on the set of data quality rule semantic encoding vectors to obtain a prompt template; based on the prompt template, performing cross-domain optimized query encoding on the data quality report semantic encoding vector and the set of data quality rule semantic encoding vectors to obtain the data quality report-quality rule query response coding features; based on the data quality report-quality rule query response coding features, the quality evaluation result is determined.

10. The data quality monitoring method according to claim 9, characterized in that: The set of data quality rule semantic coding vectors is subjected to feature aggregation processing based on linear embedding coding to obtain a prompt template, including: using a key embedding matrix to perform linear embedding coding on each data quality rule semantic coding vector in the set of data quality rule semantic coding vectors to obtain a set of linearly transformed data quality rule semantic coding vectors; arranging the set of linearly transformed data quality rule semantic coding vectors into a matrix to obtain a data quality rule semantic key matrix; extracting the maximum value of each linearly transformed data quality rule semantic coding vector in the data quality rule semantic key matrix to obtain a significant feature vector of the data quality rule semantic key matrix as the prompt template.

Images