A data quality monitoring method based on time series large model

Abstract

The present invention relates to the technical field of data quality monitoring, and discloses a data quality monitoring method based on a time series big model, comprising: dividing data elements of a data source layer and a data warehouse processing layer into classified data elements and data elements to be classified; executing a multi-way matching strategy to obtain data element classification of data elements to be classified; the classified data elements are data elements that match the data element classification; matching data element classification and data quality assessment indicators for each data element to be classified; performing data quality assessment on newly added data elements in key focus tables and regular tables; executing a new indicator value prediction strategy based on historical data elements to predict the value of the data quality assessment indicator of the newly added data elements; executing an anomaly detection strategy on the newly added data elements based on the result of the new indicator value prediction strategy to detect whether the data quality assessment indicator value of the newly added data elements is abnormal; executing a graded alarm strategy to alarm for abnormal data quality assessment indicator values.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality monitoring, and in particular to a data quality monitoring method based on a time series large model. Background Art

[0002] Nowadays, the application of data elements is becoming more and more extensive and important. With the continuous growth of business, the intensification of market volatility, and the continuous increase in data volume and complexity, traditional data quality management faces various problems in terms of coverage, management costs, flexibility, etc. 1. Over-reliance on manual labor: Traditional data quality monitoring usually requires a lot of manual intervention, which not only increases labor costs, but also may lead to low monitoring efficiency and be susceptible to errors caused by subjective factors. It is impossible to achieve efficient use of operation and maintenance data and unable to respond accurately and quickly. 2. Limited coverage; 3. Weak ability to deal with complex problems: Traditional data quality monitoring methods mainly rely on rules and thresholds, which are difficult to deal with complex data quality problems and are not comprehensive enough.

[0003] This solution proposes a data quality monitoring method based on a time series big model. It uses a combination of statistical methods, machine learning methods and time series big models to cope with various forms of anomaly monitoring such as absolute value anomalies, fluctuation anomalies, correlation anomalies and time series context anomalies, which is more efficient and flexible. Summary of the invention

[0004] The present invention provides a data quality monitoring method based on a time series large model, which helps to solve the problems mentioned in the above background technology.

[0005] In a first aspect, the present application provides a data quality monitoring method based on a time series large model, which adopts the following technical solution: A data quality monitoring method based on a time series large model, comprising:

[0006] The data elements of the data source layer and the data warehouse processing layer are divided into classified data elements and data elements to be classified;

[0007] By classifying data elements, data can be managed and organized more clearly, reducing the complexity of subsequent data processing. The manageability of data is ensured, and the classified data elements are matched with evaluation indicators according to the data element type, while the data elements to be classified can be classified according to specific strategies, thereby improving the accuracy of data quality monitoring.

[0008] The classified data element is a data element that matches the data element classification;

[0009] Match data element classification and data quality assessment indicators for each data element to be classified, specifically:

[0010] S1. Obtain any data element to be classified from the data source layer and the data warehouse processing layer;

[0011] S2, executing a multi-way matching strategy to obtain data element classification of the data element to be classified;

[0012] S3, recording the data element classification of the data element to be classified as the target classification, obtaining the data quality assessment index corresponding to the target classification as the data quality assessment index of the data element to be classified, and recording the data element to be classified as the classified data element;

[0013] By classifying each data element to be classified and then matching the evaluation indicators according to the classification, it can ensure that the monitoring indicators are consistent with the data element classification, making the data quality assessment more specific and effective. This process provides a standardized framework for subsequent quality assessment and monitoring, reducing the possibility of manual intervention.

[0014] Set frequency threshold;

[0015] By setting the frequency threshold to select the key focus tables and giving priority to them, we can focus resources and energy on those tables that have a greater impact on data quality monitoring, avoiding the waste of a large number of irrelevant tables. The selection of key focus tables ensures the priority of monitoring, avoids the dispersion of resources, and improves the efficiency and effectiveness of data quality assessment.

[0016] Get the tables in the data warehouse development requirements, filter the tables whose query times in the data warehouse query records are greater than the frequency threshold, and get the focus tables;

[0017] Filter the tables whose query times in the data warehouse query records are less than or equal to the frequency threshold to obtain regular tables;

[0018] The data elements stored in the focus table and regular table include two categories, namely historical data elements and newly added data elements;

[0019] Obtain the results of anomaly detection of historical data elements;

[0020] Based on the historical data elements, a new indicator value prediction strategy is executed to predict the value of the data quality assessment indicator of the new data element;

[0021] According to the result of the new indicator value prediction strategy, an anomaly detection strategy is executed on the new data element to detect whether the data quality assessment indicator value of the new data element is abnormal;

[0022] By performing anomaly detection on newly added data elements, potential data quality issues can be identified and responded to in real time to ensure that the quality of newly added data meets the predetermined standards. This real-time anomaly monitoring mechanism provides a guarantee for the dynamic management of data quality, ensuring that newly added data can be integrated into the system in a timely manner and meet the expected quality standards.

[0023] If there is an abnormality, the hierarchical alarm strategy is executed to issue an alarm for the abnormal data quality assessment indicator value.

[0024] Preferably, the classified data element is a data element that matches the data element classification, including:

[0025] Divide data elements into different data element categories according to their business attributes;

[0026] For any data element in a data element classification, obtain the data quality assessment index corresponding to the data element classification;

[0027] Verify the six data quality standards of the data element according to the data quality assessment indicators;

[0028] The six data quality standards are completeness, accuracy, standardization, consistency, uniqueness and timeliness, wherein each data quality standard combines one or more data quality assessment indicators.

[0029] Preferably, the executing a multi-way matching strategy to obtain the data element classification of the data element to be classified includes:

[0030] Using a large model algorithm to match the data element classification of the data element to be classified, and recording the matched data element classification as the first data element classification;

[0031] For any classified data element, the 12 attributes of the classified data element are divided into two categories, namely numerical attributes and non-numerical attributes;

[0032] Representing numeric attributes as vectors ,in, Represents the value of each numeric attribute, 0≦i≦b, b is the number of numeric attributes;

[0033] Use the word embedding method Word2Vec to represent each non-numeric attribute as a vector ,vector ,…,vector , where c-1 is the number of non-numeric attributes;

[0034] The classified data elements are represented as vectors ;

[0035] Represent the data element to be classified as a vector ;

[0036] The similarity between the classified data element and the data element to be classified is calculated, and the similarity is calculated as follows:

[0037] ,in, , , , are the weights of each component vector set respectively;

[0038] Set similarity threshold;

[0039] Comparing the similarity with a similarity threshold, if the similarity≧the similarity threshold, recording the data element classification of the classified data element as a first data element classification;

[0040] If the similarity is less than the similarity threshold, the first data element is classified as empty.

[0041] The use of machine learning algorithms can achieve an automated and efficient classification process, avoiding the errors or deviations that may be caused by manual classification. The application of machine learning algorithms improves the accuracy and flexibility of classification, and can handle complex data structures and adapt to the diversity of data.

[0042] Preferably, the executing a multi-way matching strategy to obtain the data element classification of the data element to be classified includes:

[0043] Use the data element blood relationship to match the data element classification of the data element to be classified, and record the matched data element classification as the second data element classification:

[0044] Use the lineage analysis tool to obtain the upstream dependent data elements of the data element to be classified and form a collection ,in, is a vector of any classified data element, 0≦i≦d, d is the number of upstream dependent data elements;

[0045] calculate ,in, To represent the weight of the relationship between each upstream dependent data element and the data element to be classified, 0≦i≦d;

[0046] Obtaining a classified data element that has the strongest blood relationship with the data element to be classified, and using the data element classification of the classified data element as the second data element classification;

[0047] Data element lineage analysis can deeply explore the dependencies between data and provide more accurate classification basis, especially in complex data flows and dependencies. This method helps to better understand the upstream and downstream relationships of data and improve the accuracy of data classification, especially in scenarios with complex data sources and many dependencies.

[0048] Match the data element classification of the data element to be classified according to the custom matching rule, and record the matched data element classification as the third data element classification:

[0049] Obtain the data element classification for which the custom matching rule is set, match the data element to be classified with each custom matching rule, and determine whether the match is successful;

[0050] If the match is successful, the successfully matched data element classification is used as the third data element classification;

[0051] If all custom matching rules fail to match successfully, the third data element category is empty;

[0052] By adding custom rules, we can ensure that data classification is based on the diversity of business needs, improving the accuracy and compliance of data classification. Custom rules provide flexibility, allowing the data classification process to adapt to specific business needs or special scenarios, increasing the adaptability and universality of the method.

[0053] Obtaining a first data element classification, a second data element classification, and a third data element classification, and determining whether the three data element classifications are equal;

[0054] If they are equal, any data element classification is used as the data element classification of the data element to be classified;

[0055] If they are not equal, the first data element classification, the second data element classification and the third data element classification are prioritized from low to high, and data element classifications are selected from high to low priorities as data element classifications for the data elements to be classified.

[0056] By integrating multiple matching strategies, it is possible to classify classified data elements from multiple perspectives, ensuring more accurate classification results. By setting priorities, classification conflicts can be handled reasonably, ensuring that classification decisions in complex situations are still efficient and consistent.

[0057] Preferably, executing a new indicator value prediction strategy based on historical data elements to predict the value of a data quality assessment indicator of the new data elements includes:

[0058] Obtain all data quality evaluation indicators for each historical data element, and the standard attributes corresponding to the value of any data quality evaluation indicator;

[0059] Use statistical methods to predict the interval of the value of the standard attribute at future moments, and record the predicted interval as the statistical prediction interval;

[0060] Get each value and corresponding time point of the standard attribute in the focus table and the general table, expressed as , … ,in, Indicates a point in time, It's time point The value of the corresponding standard attribute, 0≦i≦f, f is the number of values ​​of the standard attribute;

[0061] Calculate the mean ;

[0062] Calculate standard deviation ;

[0063] Calculating statistical prediction intervals .

[0064] Preferably, executing a new indicator value prediction strategy based on historical data elements to predict the value of a data quality assessment indicator of the new data elements includes:

[0065] Use the time series big model to predict the interval of the value of the standard attribute at the future time, and record the predicted interval as the model prediction interval;

[0066] Obtain each value and corresponding time point of the standard attributes in the focus table and the regular table, and use the time series big model to extract trend features and seasonal features;

[0067] Extract trend features: ,in, It's time The hidden state of represents the model’s understanding of the current input. is the weight matrix, is the bias, is the activation function;

[0068] Extract seasonal features: ,in, is the amplitude of the cycle and T is the cycle length.

[0069] Preferably, executing a new indicator value prediction strategy based on historical data elements to predict the value of the data quality assessment indicator of the new data elements includes:

[0070] For each value of the standard attribute in the focus table and the regular table and the data at the corresponding time point, a generative adversarial network is used to generate new sample data;

[0071] The generative adversarial network includes a generator G and a discriminator D, using the loss function Optimize new sample data, ,in, Represents input noise z, generating false data , use the discriminator D to determine whether the data is fake data generated by the generator, and output the probability value, The average value used to calculate the input noise loss, represents z from the probability distribution The noise vector sampled in ;

[0072] The new sample data generated by the adversarial network is used to train the time series large model to obtain the predicted value output by the time series large model ;

[0073] The prediction interval of the calculation model is .

[0074] Prediction strategies can be used to predict future data quality indicators based on the trends and patterns of historical data, providing a scientific basis for monitoring new data. Statistical methods or time series models provide powerful prediction tools that can reduce errors and uncertainties while ensuring the accuracy of data quality monitoring.

[0075] Preferably, the method of executing an anomaly detection strategy on the newly added data element according to the result of the newly added index value prediction strategy to detect whether the data quality assessment index value of the newly added data element is abnormal includes:

[0076] Obtain the maximum attribute value and the minimum attribute value of the statistical prediction interval, calculate the data quality assessment index value of the maximum attribute value and the minimum attribute value respectively, obtain the maximum statistical index value and the minimum statistical index value, and obtain the statistical prediction interval of the index value;

[0077] Obtain the maximum attribute value and the minimum attribute value of the model prediction interval, calculate the data quality assessment index value of the maximum attribute value and the minimum attribute value respectively, obtain the maximum model index value and the minimum model index value, and obtain the model prediction interval of the index value;

[0078] Calculate the data quality assessment values ​​of the standard attributes of the newly added data elements, and use the statistical prediction intervals and model prediction intervals of the calculated data quality assessment values ​​for anomaly detection.

[0079] Preferably, the step of executing a graded alarm strategy to issue an alarm for abnormal data quality assessment index values ​​includes:

[0080] When an abnormal value of a standard attribute of a newly added data element is detected, the abnormality is matched to an alarm condition;

[0081] Set different alarm levels for different alarm conditions;

[0082] The alarm levels are classified from low to high into normal, other, suspicious, warning and severe.

[0083] The hierarchical alarm strategy can trigger different response measures according to different abnormal levels, avoiding false alarms and missed alarms, and improving the response speed and accuracy of the monitoring system. Through the alarm system, relevant personnel can be notified in time to handle abnormal data, prevent potential problems from affecting business processes, and ensure the reliability of data use.

[0084] The present invention has the following beneficial effects:

[0085] 1. This data quality monitoring method based on the time series big model uses big models and other technologies to build an indicator anomaly monitoring model, monitors data anomalies for four types of anomalies: absolute value anomalies, fluctuation anomalies, correlation anomalies, and time series context anomalies, and combines statistics, machine learning, deep learning, and time series big models and other algorithms to achieve comprehensive intelligent monitoring of indicator anomalies. Through the application of time series big models in data quality monitoring, we get rid of the traditional machine learning paradigm, and there is no need to train corresponding models for specific tasks of specific data sets, which reduces the cost of model development and maintenance.

[0086] 2. This data quality monitoring method based on the time series big model realizes the automatic matching of evaluation indicators and data elements based on the big model and lineage analysis. By using machine learning, data element lineage analysis and custom rules, the automatic classification of data elements and the matching of data elements and evaluation indicators are realized. The automatic matching of this process helps to comprehensively reduce manual intervention, improve efficiency and realize true automation of the whole process.

[0087] 3. This data quality monitoring method based on the time series big model adds distribution indicators and correlation monitoring indicators based on traditional data quality indicators. It also proposes data distribution indicators such as kurtosis, skewness, characteristic stability CSI, etc. and correlation monitoring indicators such as the correlation between user data and the market index, which improves the comprehensiveness of data quality detection, avoids the limitations of traditional manual work, and improves the effect of data quality detection.

[0088] 4. This data quality monitoring method based on the time series big model can effectively improve the accuracy and stability of data prediction. By integrating long-term, medium-term and short-term trends, this method captures the changes in data in different time dimensions and enhances the comprehensiveness of the prediction. The regression model and polynomial regression model in the time series big model provide flexible analysis capabilities for long-term and medium-term trends, while the moving average rule reduces the noise impact of short-term fluctuations. This method realizes the forward-looking identification of data quality issues by predicting the quality assessment indicators of newly added data elements, and improves the efficiency of the monitoring process by automatically predicting the standard attribute interval.

[0089] 5. The data quality monitoring method based on the time series large model obtains the statistical prediction interval and model prediction interval of the data quality evaluation index by calculating the prediction interval of the data evaluation index, and then calculates the data quality evaluation index value for the boundary value in the statistical prediction interval of the attribute value to obtain the statistical prediction interval of the data quality evaluation index value. Similarly, the model prediction interval of the data quality evaluation index value is obtained, instead of directly predicting the statistical interval and the model interval of the data quality evaluation index value. The value of the predicted standard attribute can be combined with the regularity of the standard attribute value of the securities industry to more truly reflect the possible value range of the standard attribute value in the future, and then calculate the data quality evaluation index value of the predicted standard attribute value, which can more truly reflect the changes in data and provide a prediction method that is more in line with industry characteristics for data quality detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 Schematic diagram of the method of the present invention.

[0091] Figure 2 Schematic diagram of the corresponding relationship between data element classification and evaluation index of the present invention.

[0092] Figure 3 It is the technical roadmap of the present invention.

[0093] Figure 4 This is the data flow diagram of the present invention.

[0094] Figure 5 Schematic diagram of data quality assessment indicators of the present invention.

[0095] Figure 6 It is a table diagram of the alarm levels and conditions of the data quality assessment indicators of the present invention. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0097] Embodiment 1, refer to Figure 1 , a data quality monitoring method based on a time series large model, comprising:

[0098] The data elements of the data source layer and the data warehouse processing layer are divided into classified data elements and data elements to be classified;

[0099] The classified data element is a data element that matches the data element classification;

[0100] Match data element classification and data quality assessment indicators for each data element to be classified, specifically:

[0101] S1. Obtain any data element to be classified from the data source layer and the data warehouse processing layer;

[0102] S2, executing a multi-way matching strategy to obtain data element classification of the data element to be classified;

[0103] S3, recording the data element classification of the data element to be classified as the target classification, obtaining the data quality assessment index corresponding to the target classification as the data quality assessment index of the data element to be classified, and recording the data element to be classified as the classified data element;

[0104] Set frequency threshold;

[0105] By setting a frequency threshold to select key focus tables, data quality monitoring can be focused on key data, reducing resource waste. This step increases the priority of data monitoring by focusing on tables with higher query frequencies, making the monitoring system more accurate and flexible. Selecting key focus tables can also effectively avoid monitoring a large number of low-frequency tables, ensuring efficient use of monitoring resources.

[0106] Get the tables in the data warehouse development requirements, filter the tables whose query times in the data warehouse query records are greater than the frequency threshold, and get the focus tables;

[0107] Filter the tables whose query times in the data warehouse query records are less than or equal to the frequency threshold to obtain regular tables;

[0108] The data elements stored in the focus table and regular table include two categories, namely historical data elements and newly added data elements;

[0109] Obtain the results of anomaly detection of historical data elements;

[0110] Based on the historical data elements, a new indicator value prediction strategy is executed to predict the value of the data quality assessment indicator of the new data element;

[0111] According to the result of the new indicator value prediction strategy, an anomaly detection strategy is executed on the new data element to detect whether the data quality assessment indicator value of the new data element is abnormal;

[0112] If there is an abnormality, the hierarchical alarm strategy is executed to issue an alarm for the abnormal data quality assessment indicator value. Figure 4 For the data flow diagram.

[0113] Classified data elements are data elements that match the data element classification, including:

[0114] Divide data elements into different data element categories according to their business attributes;

[0115] For any data element in a data element classification, obtain the data quality assessment index corresponding to the data element classification;

[0116] Verify the six data quality standards of the data element according to the data quality assessment indicators;

[0117] The six data quality standards are completeness, accuracy, standardization, consistency, uniqueness and timeliness, wherein each data quality standard combines one or more data quality assessment indicators.

[0118] In this embodiment, refer to Figure 5 ,Schematic diagram of data quality assessment indicators.

[0119] Based on the research theories of data quality at home and abroad, combined with the industry standards of financial data in the securities industry, the data quality standards are divided into six categories: completeness, uniqueness, accuracy, standardization, consistency, and timeliness. More than 40 detection rules are formulated: line number detection, null value detection, duplicate number detection, value range (maximum and minimum value), character length, data distribution characteristics (kurtosis and skewness, etc.), coefficient of variation (CV), characteristic stability (CSI), etc. Referring to the "Basic Data Element Specifications for Securities and Futures Industry", data elements are divided into 9 data domains: subject, variety, account, behavior, asset, agreement, finance, supervision and disclosure. 614 major data element standards for the securities industry have been formulated. Each data element standard includes 12 standard attributes such as data element number, data domain, Chinese name, English name, business meaning, data type and length, and code number. Among them, data type and length can be used as the detection standard of character length index in standardization detection, and code number can be used as the detection standard of enumeration value index in accuracy detection. According to the business caliber of data indicators, data elements are classified according to their business attributes. Each classification is bound to multiple evaluation indicators, which can be customized and adjusted. For the specific relationship between data element classification and evaluation indicators, refer to Figure 2 .

[0120] Execute a multi-way matching strategy to obtain the data element classification of the data element to be classified, including:

[0121] Using the large model to match the data element classification of the data element to be classified, and recording the matched data element classification as the first data element classification;

[0122] For any classified data element, the 12 attributes of the classified data element are divided into two categories, namely numerical attributes and non-numerical attributes;

[0123] Representing numeric attributes as vectors ,in, Represents the value of each numeric attribute, 0≦i≦b, b is the number of numeric attributes;

[0124] Use the existing word embedding method Word2Vec to represent each non-numeric attribute as a vector ,vector ,…,vector , where c-1 is the number of non-numeric attributes;

[0125] The classified data elements are represented as vectors ;

[0126] Represent the data element to be classified as a vector ;

[0127] Calculate the similarity between the classified data element and the data element to be classified. The similarity is calculated as follows:

[0128] ,in, , , , are the weights of each component vector set respectively;

[0129] Although data elements have 12 standard attributes, in practical applications, the actual value of each standard attribute varies greatly. Therefore, it is considered to set different weights for each attribute to accurately express the value of the data element. At the same time, it is applied to calculate the similarity. The similarity between the classified data element and the data element to be classified is calculated based on the value of the attribute, which can better identify the true similarity between data elements.

[0130] Set similarity threshold;

[0131] In this embodiment, the similarity threshold is 0.8;

[0132] The similarity is compared with the similarity threshold, and if the similarity ≧ the similarity threshold, the data element classification of the classified data element is recorded as the first data element classification;

[0133] If the similarity is less than the similarity threshold, the first data element is classified as empty.

[0134] Use machine learning for classification recommendation. Based on vector retrieval technology, the classified data of the data elements are vectorized using bge-m3 and stored in the faiss vector database to form the vector data of the data elements. The names of the data elements to be classified are vectorized, and the distance between the data elements to be classified and the data elements in the vector database is calculated through the vector matching algorithm. The closest data element classification with a similarity greater than 0.8 is used as the recommended classification.

[0135] Execute a multi-way matching strategy to obtain the data element classification of the data element to be classified, including:

[0136] Use the data element blood relationship to match the data element classification of the data element to be classified, and record the matched data element classification as the second data element classification:

[0137] Use existing lineage analysis tools to obtain upstream dependent data elements of the data element to be classified and form a collection ,in, is a vector of any classified data element, 0≦i≦d, d is the number of upstream dependent data elements;

[0138] calculate ,in, To represent the weight of the relationship between each upstream dependent data element and the data element to be classified, 0≦i≦d;

[0139] Obtaining a classified data element that has the strongest blood relationship with the data element to be classified, and using the data element classification of the classified data element as the second data element classification;

[0140] In this embodiment, since there is a situation where the weights of the data element to be classified and multiple upstream dependent data elements are slightly different, the strength of the blood relationship cannot be judged only by the weight. Therefore, it is necessary to calculate the weighted average of the weight and the upstream dependent data elements again to most intuitively and accurately obtain the upstream dependent data element with the strongest blood relationship with the data element to be classified.

[0141] Classification recommendation is made based on the data element lineage. The open source lineage analysis tool sqllineage is used to analyze the hsql data element level lineage in the data warehouse and store it in the datahub data element management platform. The upstream dependent data elements are obtained from the datahub data element lineage through the table and data element name to be classified, and the data element classification of the upstream dependent data element is used as the recommended classification of the data element to be classified.

[0142] Match the data element classification of the data element to be classified according to the custom matching rule, and record the matched data element classification as the third data element classification:

[0143] Obtain the data element classification for which the custom matching rule is set, match the data element to be classified with each custom matching rule, and determine whether the match is successful;

[0144] If the match is successful, the successfully matched data element classification is used as the third data element classification;

[0145] If all custom matching rules fail to match successfully, the third data element category is empty;

[0146] Classification recommendation based on custom matching rules. By configuring custom regular expression matching rules for the classification, the data sampling is matched. If the match is successful, the classification is used as the recommended classification for matching data elements, such as email addresses, mobile phone numbers, etc., which can be more accurate with custom classification rules. In the end, each data element corresponds to a data element classification, and is matched to the quality assessment indicator corresponding to the data element classification.

[0147] Obtaining a first data element classification, a second data element classification, and a third data element classification, and determining whether the three data element classifications are equal;

[0148] If they are equal, any data element classification is used as the data element classification of the data element to be classified;

[0149] If they are not equal, the first data element classification, the second data element classification and the third data element classification are prioritized from low to high, and data element classifications are selected from high to low priorities as data element classifications for the data elements to be classified.

[0150] Match the unclassified data elements in the data warehouse with the corresponding data element classifications, and study the use of machine learning for classification recommendations, classification recommendations based on data element lineage, and classification recommendations based on custom matching rules, and automatically match different evaluation indicators for each type. If multiple recommendation types conflict, the priority is selected as follows: custom classification recommendation > data element lineage recommendation > machine learning classification recommendation.

[0151] By combining machine learning, blood relationship analysis and custom rules for multi-way matching, the classification results of the data elements to be classified are ensured to be more accurate. Machine learning improves the automation and accuracy of classification, blood relationship analysis helps understand the dependencies between data, and custom rules enhance flexibility. This method can fully consider the characteristics of the data and improve the scientificity and applicability of classification.

[0152] In this embodiment, the methods that can be used include 3 Methods, box plots, ARIMA models and Pearson correlation coefficients; machine learning and deep learning methods, including Prophet models, LightGBM models, HMM, LSTM, etc.; large model methods: including Chronos, Moirai, TimesFM, etc.

[0153] Based on the creation of data metadata quality assessment indicators, abnormal monitoring is performed on the indicator results to identify abnormal values ​​or abnormal patterns of the indicators. The abnormalities of indicators in the project are divided into absolute value abnormalities, fluctuation abnormalities, correlation abnormalities, and time series context abnormalities. Absolute value abnormalities refer to the value of the indicator exceeding the normal range, fluctuation abnormalities refer to the fluctuation of the indicator value exceeding the normal range, and correlation abnormalities usually occur when the relationship between two or more indicators changes. If the correlation of strongly correlated indicators suddenly weakens or disappears, it indicates that there is a problem with one or more of the indicators. In the time series context abnormality, the indicator value itself is not abnormal, but there is a large difference in the context environment compared with the previous and next time periods, which may appear as a change in trend or periodic abnormalities.

[0154] Through four types of anomaly detection strategies (absolute value anomaly, fluctuation anomaly, correlation anomaly, and time series context anomaly), various data quality issues can be captured in detail. The prediction strategy of statistical methods and time series models can accurately predict the quality indicators of newly added data elements and discover potential problems in a timely manner. Through these strategies, dynamic monitoring and early warning of data quality can be achieved, reducing the occurrence of problems and improving the foresight of data management.

[0155] Based on the historical data elements, the new indicator value prediction strategy is executed to predict the value of the data quality assessment indicator of the new data element, including:

[0156] Obtain all data quality evaluation indicators for each historical data element, and the standard attributes corresponding to the value of any data quality evaluation indicator;

[0157] In this embodiment, since the newly added data element will become a historical data element, the anomaly detection of the historical data element only targets the standard attributes that match the data quality assessment index. Therefore, when performing anomaly detection on the newly added data element, it is only necessary to predict and detect the standard attributes that may be used for subsequent data quality assessment index value detection, saving time and improving efficiency.

[0158] Use statistical methods to predict the interval of the value of the standard attribute at future moments, and record the predicted interval as the statistical prediction interval;

[0159] Get each value and corresponding time point of the standard attribute in the focus table and the general table, expressed as , … ,in, Indicates a point in time, It's time point The value of the corresponding standard attribute, 0≦i≦f, f is the number of values ​​of the standard attribute;

[0160] Calculate the mean ;

[0161] Calculate standard deviation ;

[0162] Calculating statistical prediction intervals .

[0163] Since the data of the securities industry has certain regularities, while knowing the data in the focus table, we can calculate the slope of a certain standard attribute in the table, and assume that the value of the standard attribute at future moments will continue this regularity. Therefore, we calculate the slope of the standard attribute and combine it with the standard deviation of 3 The model estimates the interval of the value of the standard attribute at future moments, improves the accuracy of the prediction, and improves the efficiency of data detection.

[0164] Based on the historical data elements, the new indicator value prediction strategy is executed to predict the value of the data quality assessment indicator of the new data element, including:

[0165] Use the time series big model to predict the interval of the value of the standard attribute at the future time, and record the predicted interval as the model prediction interval;

[0166] Obtain each value and corresponding time point of the standard attributes in the focus table and the regular table, and use the time series big model to extract trend features and seasonal features;

[0167] Extract trend features: ,in, It's time The hidden state of represents the model’s understanding of the current input. is the weight matrix, is the bias, is the activation function;

[0168] Extract seasonal features: ,in, is the amplitude of the cycle and T is the cycle length.

[0169] This data quality monitoring method based on a large time series model can effectively improve the accuracy and stability of data forecasting. By integrating long-term, medium-term and short-term trends, this method captures changes in data in different time dimensions and enhances the comprehensiveness of forecasting. The application of regression models and polynomial regression models provides flexible analysis capabilities for long-term and medium-term trends, while the moving average rule reduces the noise impact of short-term fluctuations. This method achieves forward-looking identification of data quality issues by predicting the quality assessment indicators of newly added data elements, and improves the efficiency of the monitoring process by automatically predicting standard attribute intervals.

[0170] Based on the historical data elements, the new indicator value prediction strategy is executed to predict the value of the data quality assessment indicator of the new data element, including:

[0171] For each value of the standard attribute in the focus table and the regular table and the data at the corresponding time point, a generative adversarial network is used to generate new sample data;

[0172] The generative adversarial network includes a generator G and a discriminator D, using the loss function Optimize new sample data, ,in, Represents input noise z, generating false data , use the discriminator D to determine whether the data is fake data generated by the generator, and output the probability value, The average value used to calculate the input noise loss, represents z from the probability distribution The noise vector sampled in ;

[0173] The new sample data generated by the adversarial network is used to train the time series large model to obtain the predicted value output by the time series large model ;

[0174] The prediction interval of the calculation model is .

[0175] In this embodiment, refer to Figure 3 For the technology roadmap.

[0176] According to the result of the new indicator value prediction strategy, the anomaly detection strategy is executed on the new data element to detect whether the data quality assessment indicator value of the new data element is abnormal, including:

[0177] Obtain the maximum attribute value and the minimum attribute value of the statistical prediction interval, calculate the data quality assessment index value of the maximum attribute value and the minimum attribute value respectively, obtain the maximum statistical index value and the minimum statistical index value, and obtain the statistical prediction interval of the index value;

[0178] Obtain the maximum attribute value and the minimum attribute value of the model prediction interval, calculate the data quality assessment index value of the maximum attribute value and the minimum attribute value respectively, obtain the maximum model index value and the minimum model index value, and obtain the model prediction interval of the index value;

[0179] Calculate the data quality evaluation value of the standard attributes of the newly added data element and report the calculated data quality

[0180] The evaluation value uses the statistical prediction interval of the indicator value and the model prediction interval for anomaly detection.

[0181] By calculating the prediction interval of the value of the standard attribute corresponding to the data evaluation indicator, the statistical prediction interval and model prediction interval of the standard attribute value are obtained, and then the data quality evaluation indicator value is calculated for the boundary value in the statistical prediction interval of the attribute value to obtain the statistical prediction interval of the data quality evaluation indicator value. Similarly, the model prediction interval of the data quality evaluation indicator value is obtained, instead of directly predicting the statistical interval and model interval of the data quality evaluation indicator value. The value of the predicted standard attribute can be combined with the regularity of the standard attribute value of the securities industry to more truly reflect the possible value range of the standard attribute value in the future, and then the data quality evaluation indicator value of the predicted standard attribute value is calculated, which can more truly reflect the changes in data and provide a prediction method that is more in line with industry characteristics for data quality detection.

[0182] Execute the hierarchical alarm strategy to issue alarms for abnormal data quality assessment indicator values, including:

[0183] When an abnormal value of a standard attribute of a newly added data element is detected, the abnormality is matched to an alarm condition;

[0184] Set different alarm levels for different alarm conditions;

[0185] Alarm levels are classified from low to high into normal, other, suspicious, warning and severe.

[0186] In this embodiment, refer to Figure 6 , data quality assessment indicator warning level and condition table.

[0187] Through the hierarchical alarm strategy, accurate alarms are issued for abnormal data quality indicators to ensure that data problems of different abnormal levels are responded to in a timely manner. After setting the alarm conditions and alarm levels, the response measures can be flexibly adjusted according to the actual situation to avoid excessive alarms or missed reports, ensuring the accuracy and reliability of the monitoring system. Through hierarchical alarms, relevant personnel can take appropriate countermeasures for problems of different severity and improve processing efficiency.

[0188] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0189] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data quality monitoring method based on a time series large model, characterized in that: include: The data elements of the data source layer and the data warehouse processing layer are divided into classified data elements and data elements to be classified; The classified data element is a data element that matches the data element classification; Match data element classification and data quality assessment indicators for each data element to be classified, specifically: S1. Obtain any data element to be classified from the data source layer and the data warehouse processing layer; S2, executing a multi-way matching strategy to obtain data element classification of the data element to be classified; S3, recording the data element classification of the data element to be classified as the target classification, obtaining the data quality assessment index corresponding to the target classification as the data quality assessment index of the data element to be classified, and recording the data element to be classified as the classified data element; Set frequency threshold; Get the tables in the data warehouse development requirements, filter the tables whose query times in the data warehouse query records are greater than the frequency threshold, and get the focus tables; Filter the tables whose query times in the data warehouse query records are less than or equal to the frequency threshold to obtain regular tables; The data elements stored in the focus table and regular table include two categories, namely historical data elements and newly added data elements; Based on the historical data elements, a new indicator value prediction strategy is executed to predict the value of the data quality assessment indicator of the new data element; According to the result of the new index value prediction strategy, an anomaly detection strategy is executed on the new data element to detect whether the data quality assessment index value of the new data element is abnormal; Among them, the focus table takes precedence over the regular table for anomaly detection; If there is an abnormality, the hierarchical alarm strategy is executed to issue an alarm for the abnormal data quality assessment indicator value.

2. The data quality monitoring method based on the time series big model according to claim 1 is characterized in that: The classified data element is a data element that matches the data element classification, including: Divide data elements into different data element categories according to their business attributes; For any data element in a data element classification, obtain the data quality assessment index corresponding to the data element classification; Verify the six data quality standards of the data element according to the data quality assessment indicators; The six data quality standards are completeness, accuracy, standardization, consistency, uniqueness and timeliness, wherein each data quality standard combines one or more data quality assessment indicators.

3. The data quality monitoring method based on time series big model according to claim 1 is characterized in that: The executing of the multi-way matching strategy to obtain the data element classification of the data element to be classified includes: Using the large model to classify the data elements of the data element to be classified, and recording the matching data element classification as the first data element classification; For any classified data element, the 12 standard attributes of the classified data element are divided into two categories, namely, numerical attributes and non-numerical attributes; Representing numeric attributes as vectors ,in, Represents the value of each numeric attribute, 0≦i≦b, b is the number of numeric attributes; Use the word embedding method Word2Vec to represent each non-numeric attribute as a vector ,vector ,…,vector , where c-1 is the number of non-numeric attributes; The classified data elements are represented as vectors ; Represent the data element to be classified as a vector ; The similarity between the classified data element and the data element to be classified is calculated, and the similarity is calculated as follows: ,in, , , , are the weights of each component vector set respectively; Set similarity threshold; Compare the similarity with the similarity threshold, and if there exists a similarity between the classified data element and the data element to be classified≧the similarity threshold, record the data element classification of the classified data element as the first data element classification; If the similarity between all classified data elements and the data element to be classified is less than the similarity threshold, the first data element classification is empty.

4. The data quality monitoring method based on the time series big model according to claim 3 is characterized in that: The executing of the multi-way matching strategy to obtain the data element classification of the data element to be classified includes: Use the data element blood relationship to match the data element classification of the data element to be classified, and record the matched data element classification as the second data element classification: Use the lineage analysis tool to obtain the upstream dependent data elements of the data element to be classified and form a collection ,in, is a vector of any classified data element, 0≦i≦d, d is the number of upstream dependent data elements; calculate ,in, To represent the weight of the relationship between each upstream dependent data element and the data element to be classified, 0≦i≦d; Obtaining a classified data element that has the strongest blood relationship with the data element to be classified, and using the data element classification of the classified data element as the second data element classification; Match the data element classification of the data element to be classified according to the custom matching rule, and record the matched data element classification as the third data element classification: Obtain the data element classification for which the custom matching rule is set, match the data element to be classified with each custom matching rule, and determine whether the match is successful; If the match is successful, the successfully matched data element classification is used as the third data element classification; If all custom matching rules fail to match successfully, the third data element category is empty; Obtaining a first data element classification, a second data element classification, and a third data element classification, and determining whether the three data element classifications are equal; If they are equal, any data element classification is used as the data element classification of the data element to be classified; If they are not equal, the first data element classification, the second data element classification and the third data element classification are prioritized from low to high, and data element classifications are selected from high to low priorities as data element classifications for the data elements to be classified.

5. The data quality monitoring method based on time series big model according to claim 1 is characterized in that: The step of executing a new indicator value prediction strategy based on the historical data element to predict the value of the data quality assessment indicator of the new data element includes: Obtain all data quality evaluation indicators for each historical data element, and the standard attributes corresponding to the value of any data quality evaluation indicator; Use statistical methods to predict the interval of the value of the standard attribute at future moments, and record the predicted interval as the statistical prediction interval; Get each value and corresponding time point of the standard attribute in the focus table and the general table, expressed as , … ,in, Indicates a point in time, It's time point The value of the corresponding standard attribute, 0≦i≦f, f is the number of values ​​of the standard attribute; Calculate the mean ; Calculate standard deviation ; Calculating statistical prediction intervals .

6. The data quality monitoring method based on time series big model according to claim 5 is characterized in that: The step of executing a new indicator value prediction strategy based on the historical data element to predict the value of the data quality assessment indicator of the new data element includes: Use the time series big model to predict the interval of the value of the standard attribute at the future time, and record the predicted interval as the model prediction interval; Obtain each value and corresponding time point of the standard attributes in the focus table and the regular table, and use the time series big model to extract trend features and seasonal features; Extract trend features: ,in, It's time The hidden state of represents the model’s understanding of the current input. is the weight matrix, is the bias, is the activation function; Extract seasonal features: ,in, is the amplitude of the cycle and T is the cycle length.

7. The data quality monitoring method based on time series big model according to claim 6 is characterized in that: The step of executing a new indicator value prediction strategy based on the historical data element to predict the value of the data quality assessment indicator of the new data element includes: For each value of the standard attribute in the focus table and the regular table and the data at the corresponding time point, a generative adversarial network is used to generate new sample data; The generative adversarial network includes a generator G and a discriminator D, using the loss function Optimize new sample data, ,in, Represents input noise z, generating false data , use the discriminator D to determine whether the data is fake data generated by the generator, and output the probability value, The average value used to calculate the input noise loss, represents z from the probability distribution The noise vector sampled in ; The new sample data generated by the adversarial network is used to train the time series large model to obtain the predicted value output by the time series large model ; The prediction interval of the calculation model is .

8. The data quality monitoring method based on time series big model according to claim 7 is characterized in that: The method of executing an anomaly detection strategy on the newly added data element according to the result of the newly added index value prediction strategy to detect whether the data quality assessment index value of the newly added data element is abnormal includes: Obtain the maximum attribute value and the minimum attribute value of the statistical prediction interval, calculate the data quality assessment index value of the maximum attribute value and the minimum attribute value respectively, obtain the maximum statistical index value and the minimum statistical index value, and obtain the statistical prediction interval of the index value; Obtain the maximum attribute value and the minimum attribute value of the model prediction interval, calculate the data quality assessment index value of the maximum attribute value and the minimum attribute value respectively, obtain the maximum model index value and the minimum model index value, and obtain the model prediction interval of the index value; Calculate the data quality assessment values ​​of the standard attributes of the newly added data elements, and use the statistical prediction intervals and model prediction intervals of the calculated data quality assessment values ​​for anomaly detection.

9. The data quality monitoring method based on time series big model according to claim 1 is characterized in that: The implementation of the hierarchical alarm strategy to issue an alarm for abnormal data quality assessment index values ​​includes: When an abnormal value of a standard attribute of a newly added data element is detected, the abnormality is matched to an alarm condition; Set different alarm levels for different alarm conditions; The alarm levels are classified from low to high into normal, other, suspicious, warning and severe.

Images