Multi-dimensional data quality detection method based on data medium table

Abstract

The invention is suitable for the technical field of data processing, and particularly relates to a multi-dimensional data quality detection method based on a data-in-data station, which establishes a relation between actual data and predicted data through a time sequence prediction model, facilitates the data quality inspection from the dynamic angle of a time sequence, and improves the accuracy of data quality inspection. The degree that the initial time sequence data deviates from the expected time sequence change rule is judged by predicting the deviation degree, data with small deviation from the expected time sequence change rule is screened out as initial reference data according to the comparison of the predicted deviation degree and a deviation degree threshold value, and data with obvious deviation from the expected time sequence change rule is screened out as target data; and obtaining a correlation probability value to quantify a correlation degree between the target data and the target reference data, so as to comprehensively consider the correlation between the target data and the target reference data and the influence of the quality condition of the target reference data on quality evaluation, and analyze to obtain a data quality score of each piece of initial time sequence data. And the detection accuracy of the data quality is improved.

Description

Technical Field

[0001] The present invention is applicable to the field of data processing technology, and in particular relates to a multidimensional data quality detection method based on a data center. Background Art

[0002] In today's digital age, data has become a key element in corporate decision-making, business operations and other activities. With the explosive growth of data volume and the diversification of data sources, data quality issues have become increasingly prominent. Users often integrate and manage data by building a data middle platform, but in the data middle platform environment, the complexity and large scale of data make the data middle platform face huge challenges in managing and utilizing high-quality data.

[0003] Traditional data quality detection methods generally use statistical principles to analyze data, which can identify outliers or data that does not conform to normal distribution, or compare the data to be detected with sample data or historical data to find data changes that do not conform to abnormal data patterns. However, the above data quality detection methods focus on the data itself in each dimension, and cannot use the relevant information between multi-dimensional data. It is difficult to fully evaluate data quality, resulting in poor accuracy of data quality detection results, which cannot meet the management and utilization needs of high-quality data in the data center.

[0004] Therefore, in the field of data processing technology, how to improve the accuracy of data quality detection results has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a multi-dimensional data quality detection method based on a data center to solve the problem of low accuracy of existing data quality detection methods.

[0006] An embodiment of the present invention provides a multidimensional data quality detection method based on a data middle station. The multidimensional data quality detection method based on a data middle station is applied to the data middle station. The multidimensional data quality detection method based on the data middle station includes:

[0007] S1, obtaining a trained time series prediction model, a trained classification model, N groups of initial time series data, and the correlation between the initial time series data, wherein the correlation includes correlation and irrelevance, and N is an integer greater than 1.

[0008] S2, input each set of initial time series data into the trained time series prediction model to obtain the predicted time series data corresponding to each set of initial time series data.

[0009] S3, according to each group of initial time series data and the predicted time series data corresponding to each group of initial time series data, obtain the deviation degree threshold corresponding to the N groups of initial time series data and the predicted deviation degree corresponding to each group of initial time series data.

[0010] S4, classifying N groups of initial time series data into target data and initial reference data according to the prediction deviation degree and the deviation degree threshold, and obtaining a first data quality score of the corresponding initial reference data according to the prediction deviation degree corresponding to each initial reference data.

[0011] S5, for any target data, according to the correlation between the initial time series data, the target reference data corresponding to the current target data is obtained from all the initial reference data to obtain a data group to be classified consisting of the current target data and each target reference data.

[0012] S6, inputting each to-be-classified data group corresponding to the current target data into the trained classification model respectively, and obtaining the correlation probability value between the current target data and the corresponding target reference data.

[0013] S7, obtaining a second data quality score corresponding to the current target data according to the correlation probability value between the current target data and each corresponding target reference data, the prediction deviation degree corresponding to each target reference data corresponding to the current target data, and the prediction deviation degree corresponding to the current target data.

[0014] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by using the laws and patterns of time series data learned by the time series prediction model during the training process, a connection between actual data and predicted data is established, so as to facilitate the examination of data quality from the dynamic perspective of the time series, and according to each group of initial time series data and the predicted time series data corresponding to each group of initial time series data, the deviation degree threshold corresponding to N groups of initial time series data and the predicted deviation degree corresponding to each group of initial time series data are obtained, so as to accurately judge whether the initial time series data conforms to the expected time series change law and the degree of deviation by predicting the degree of deviation, and according to the comparison between the predicted degree of deviation and the degree of deviation threshold, relatively stable data with a small deviation from the expected value are selected as initial reference data, data with a more obvious deviation from the expected value are selected as target data, and the initial reference data is used as a reference benchmark for subsequent evaluation of the quality of the target data, and the strength of the correlation between the target data and the target reference data is quantified by obtaining the relevant probability value, and then the correlation between the target data and each target reference data and the influence of the quality status of the target reference data itself on the target data quality evaluation are comprehensively considered, and the second data quality score corresponding to the current target data is calculated according to the set correlation rule, thereby improving the accuracy of data quality analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 This is a schematic diagram of an application environment of a multi-dimensional data quality detection method based on a data middle platform provided in the first embodiment of the present invention;

[0017] Figure 2 It is a flow chart of a multi-dimensional data quality detection method based on a data middle platform provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0019] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0020] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0022] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0024] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0025] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0026] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0027] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0028] A multi-dimensional data quality detection method based on a data center is provided in the first embodiment of the present invention and can be applied to Figure 1 In an application environment, a client communicates with a server. The client includes but is not limited to a PDA, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cloud computer device, a personal digital assistant (PDA), and other computer devices. The server can be implemented by an independent server or a server cluster consisting of multiple servers.

[0029] Among them, the multi-dimensional data quality detection method based on the data middle platform can be applied to the data middle platform. The data middle platform is a data processing and service platform located between the business front desk and the data back end. It can extract, clean, convert, store and process the data generated by various business systems within the enterprise (such as sales systems, financial systems, customer relationship management systems, etc.), and then build a unified data asset system based on the processed data to provide standardized and reusable data services for different departments and different business scenarios within the enterprise, so as to facilitate the enterprise to fully and accurately understand the overall business picture and realize data interconnection.

[0030] Since enterprise data are of various types and from a wide range of sources, how to reasonably and accurately test data quality, so as to provide reliable input for users' decision support systems, data analysis applications and other tasks based on data quality test results, reduce the risk of wrong decisions caused by data quality issues, and thereby improve users' overall operational efficiency and competitiveness, has become the primary challenge facing the data middle platform.

[0031] See also Figure 2 , is a flow chart of a multi-dimensional data quality detection method based on a data middle station provided in the first embodiment of the present invention. The multi-dimensional data quality detection method based on a data middle station can be applied to Figure 1 The client in the data center can include the following steps:

[0032] S1, obtaining a trained time series prediction model, a trained classification model, N groups of initial time series data, and the correlation between the initial time series data, wherein the correlation includes correlation and irrelevance, and N is an integer greater than 1.

[0033] Among them, the initial time series data is the object to be tested for quality, representing a series of data records generated over time in a specific scenario. By analyzing and processing the initial time series data, we can fully understand the quality of the data in the time dimension, and then find possible problems, such as abnormal data fluctuations, data missing, etc., which can provide reliable input for users' decision support systems, data analysis applications and other tasks, reduce the risk of wrong decisions caused by data quality problems, and thus improve users' overall operational efficiency and competitiveness.

[0034] In practical applications, many data do not exist in isolation, but are related to each other. For example, the sales volume of a product may be related to factors such as marketing activities and seasonal changes. Therefore, by clarifying the correlation between the initial time series data, this embodiment can more accurately combine the correlation to evaluate the quality of the data in subsequent steps, avoiding misjudgment of data quality due to viewing the data in isolation.

[0035] Among them, the correlation between the initial time series data can be determined by the implementer through statistical analysis methods, for example, by calculating the correlation coefficient between the initial time series data (such as Pearson correlation coefficient, Spearman correlation coefficient, etc.) to determine the correlation between the initial time series data.

[0036] The time series prediction model is a tool for predicting and analyzing time series data. Under the multi-dimensional data quality detection task of the data center, the trained time series prediction model is used to learn the rules of time series data to predict the possible values ​​of the initial time series data at a certain time point or time period in the future, and then the data quality status is evaluated by comparing the predicted value with the actual value. Among them, the trained time series prediction model in this embodiment can be a long short-term memory artificial neural network (Long-Short Term Memory, LSTM), a recurrent neural network (Recurrent Neural Network, RNN) and other models for predicting and analyzing time series data. Those skilled in the art know that any time series prediction model in the prior art falls within the protection scope of the present invention and will not be repeated here.

[0037] The classification model is mainly used to classify the data. In this embodiment, the trained classification model is used to classify the data according to factors such as the relationship between the data, and output the confidence of the classification result as the probability value of the correlation between the data to further evaluate the data quality.

[0038] The above steps of obtaining the trained time series prediction model, the trained classification model, N groups of initial time series data, and the correlation between the initial time series data provide a data basis and model support for the subsequent calculation of the quality score of the initial time series data.

[0039] S2, input each set of initial time series data into the trained time series prediction model to obtain the predicted time series data corresponding to each set of initial time series data.

[0040] Among them, this embodiment uses the trained time series prediction model to perform prediction analysis on each group of initial time series data, so as to obtain the predicted time series data corresponding to each group of initial time series data. By comparing the actual acquired initial time series data with the predicted time series data obtained by model prediction, it can provide a basic basis for subsequent evaluation of data quality deviation and other operations, so as to accurately determine whether the initial time series data conforms to the expected time series change law, and then discover potential data quality problems.

[0041] Optionally, the initial time series data includes actual data corresponding to the M1th preset time point to the M2th preset time point, and S2 includes the following steps:

[0042] S21, for any initial time series data, input the current initial time series data into the trained time series prediction model to obtain the predicted time series data corresponding to the current initial time series data, wherein the predicted time series data includes the predicted data corresponding to the M1+1th preset time point to the M2+1th preset time point.

[0043] S22, traverse all the initial time series data to obtain the predicted time series data corresponding to each group of initial time series data.

[0044] Among them, the initial time series data covers the actual data corresponding to the M1th preset time point to the M2th preset time point. After each set of initial time series data is input into the time series prediction model, the time series prediction model will predict the data at the next time point based on the rules and patterns of the time series data learned during the training process, thereby outputting the predicted time series data corresponding to each set of initial time series data, that is, the predicted data corresponding to the M1+1th preset time point to the M2+1th preset time point, providing comparative reference data for the subsequent analysis of the quality of the initial time series data.

[0045] The above steps of inputting each set of initial time series data into the trained time series prediction model to obtain the predicted time series data corresponding to each set of initial time series data establish the connection between the actual data and the predicted data based on the rules and patterns of the time series data learned by the time series prediction model during the training process, which facilitates the examination of data quality from the dynamic perspective of the time series and provides a basis for data quality detection.

[0046] S3, according to each group of initial time series data and the predicted time series data corresponding to each group of initial time series data, obtain the deviation degree threshold corresponding to the N groups of initial time series data and the predicted deviation degree corresponding to each group of initial time series data.

[0047] Among them, by comparing and analyzing each group of initial time series data and its corresponding predicted time series data, the deviation degree threshold corresponding to N groups of initial time series data and the predicted deviation degree corresponding to each group of initial time series data are quantified. The predicted deviation degree is used to accurately judge whether the initial time series data conforms to the expected time series change law and the degree of deviation, thereby providing a data basis for subsequent operations such as accurately judging the data quality status, classifying the data, and finally giving the data quality score.

[0048] Optionally, S3 includes the following steps:

[0049] S31, according to the actual data and predicted data corresponding to each group of initial time series data at the Kth preset time point, obtain the Kth prediction deviation value corresponding to each group of initial time series data, where K=M1+1, M1+2, ..., M2.

[0050] S32, traverse K=M1+1, M1+2, ..., M2, and obtain M2-M1 prediction deviation values ​​corresponding to each group of initial time series data.

[0051] S33, determining the sum of the M2-M1 prediction deviation values ​​corresponding to each group of initial time series data as the total prediction deviation value corresponding to each group of initial time series data.

[0052] S34, determining the average value of the actual data corresponding to each group of initial time series data from the M1th preset time point to the M2th preset time point as the average data value corresponding to each group of initial time series data.

[0053] S35, determining the difference between the total value of the prediction deviation corresponding to each group of initial time series data and the corresponding average data value as the prediction deviation degree corresponding to each group of initial time series data.

[0054] Among them, the deviation of each group of initial time series data at each specific time point is quantitatively analyzed, and the deviation of each group of data at each time point is summarized through a summation operation to obtain the total value of the predicted deviation to comprehensively reflect the quantitative indicator of the degree of deviation of the group of data in the entire time period, and then the predicted deviation degree is obtained to characterize the deviation of each group of initial time series data relative to its own average level, which provides a key analysis dimension for subsequent operations such as data classification and quality scoring.

[0055] Optionally, S3 further includes the following steps:

[0056] S36, according to the prediction deviation degrees corresponding to the N groups of initial time series data, obtain the prediction deviation degree probability distribution function corresponding to the N groups of initial time series data.

[0057] S37, determining the third quartile corresponding to the prediction deviation degree probability distribution function as the deviation degree threshold corresponding to the N groups of initial time series data.

[0058] Among them, according to the prediction deviation degree corresponding to N groups of initial time series data, statistical methods are used to construct and obtain the prediction deviation degree probability distribution function corresponding to N groups of initial time series data, which is used to describe the probability of different prediction deviation degrees appearing in N groups of initial time series data. It helps to have a deeper understanding of the overall distribution characteristics of the data deviation degree, and provides a more scientific basis for the subsequent determination of the deviation degree threshold.

[0059] The quartile is a statistic that divides the data distribution into four equal parts. By determining the third quartile as the deviation degree threshold, N groups of initial time series data can be divided into data with smaller deviations and data with larger deviations based on the deviation degree threshold, which facilitates further data quality assessment and processing.

[0060] The above steps of obtaining the deviation degree thresholds corresponding to N groups of initial time series data and the predicted deviation degree corresponding to each group of initial time series data based on each group of initial time series data and the predicted time series data corresponding to each group of initial time series data, accurately judge whether the initial time series data conforms to the expected time series change law and the degree of deviation through the predicted deviation degree, thereby providing a data basis for subsequent operations such as accurately judging the data quality status, classifying the data and finally giving the data quality score.

[0061] S4, classifying N groups of initial time series data into target data and initial reference data according to the prediction deviation degree and the deviation degree threshold, and obtaining a first data quality score of the corresponding initial reference data according to the prediction deviation degree corresponding to each initial reference data.

[0062] Among them, according to the comparison between the prediction deviation degree and the deviation degree threshold, N groups of initial time series data are classified into target data and initial reference data. The target data is the data that deviates greatly from the expected degree and needs further in-depth exploration of its quality status. The initial reference data is relatively closer to the expected and can be used as a reference standard for evaluating the quality of the target data. The corresponding first data quality score is determined according to the prediction deviation degree of the initial reference data, thereby providing more detailed distinction and quantitative indicators for comprehensive evaluation of data quality status.

[0063] Optionally, S4 includes the following steps:

[0064] S41, for any initial time series data, if the prediction deviation degree corresponding to the current initial time series data is less than the deviation degree threshold, the current initial time series data is classified as initial reference data.

[0065] S42: If the prediction deviation degree corresponding to the current initial time series data is greater than or equal to the deviation degree threshold, the current initial time series data is classified as target data.

[0066] S43, obtaining a first data quality score corresponding to each initial reference data according to the prediction deviation degree corresponding to each initial reference data, wherein the first data quality score is negatively correlated with the corresponding prediction deviation degree.

[0067] Among them, relatively stable data with small deviation from expectations are selected as initial reference data, and the smaller the prediction deviation degree corresponding to the initial reference data, the higher the corresponding first data quality score. Conversely, the greater the prediction deviation degree corresponding to the initial reference data, the lower the corresponding first data quality score.

[0068] Data that deviates significantly from expectations is selected as target data, and the initial reference data is used as a reference benchmark for subsequent evaluation of the target data quality, so that the data quality of the target data can be analyzed in more dimensions in combination with the correlation between the data, thereby improving the accuracy of data quality analysis.

[0069] In the above step, N groups of initial time series data are classified into target data and initial reference data according to the prediction deviation degree and the deviation degree threshold, and a first data quality score of the corresponding initial reference data is obtained according to the prediction deviation degree corresponding to each initial reference data. Relatively stable data with a smaller deviation from expectations are screened out as initial reference data, and data with a more obvious deviation from expectations are screened out as target data. The initial reference data is used as a reference benchmark for subsequent evaluation of the target data quality, so as to conduct a multi-dimensional analysis of the data quality of the target data in combination with the correlation between the data, thereby improving the accuracy of data quality analysis.

[0070] S5, for any target data, according to the correlation between the initial time series data, the target reference data corresponding to the current target data is obtained from all the initial reference data to obtain a data group to be classified consisting of the current target data and each target reference data.

[0071] Among them, for target data that may have major quality problems or deviate greatly from expectations, based on the predetermined correlation between the initial time series data, the relevant target reference data are screened out from the initial reference data with relatively better quality, and the target data and each target reference data are combined into data groups to be classified, providing a more targeted and relevant analysis basis for subsequent further evaluation of the quality status of the target data.

[0072] Optionally, S5 includes the following steps:

[0073] S51, for any target data, according to the correlation between the initial time series data, obtain the correlation between the current target data and each initial reference data.

[0074] S52: For any initial reference data, if the correlation between the current target data and the current initial reference data is correlation, the current initial reference data is determined as the target reference data corresponding to the current target data.

[0075] S53, combining the current target data and the current target reference data to obtain a data group to be classified consisting of the current target data and the current target reference data.

[0076] S54, traversing all the initial reference data to obtain all the data groups to be classified corresponding to the current target data.

[0077] S55, traverse all target data to obtain all to-be-classified data groups corresponding to each target data.

[0078] The above-mentioned step of obtaining, for any target data, the target reference data corresponding to the current target data from all the initial reference data according to the correlation between the initial time series data, and obtaining the data group to be classified consisting of the current target data and each target reference data, and combining the target data and each target reference data into the data group to be classified, provides a more targeted and relevant analysis basis for further evaluation of the quality status of the target data, can make full use of the inherent correlation between the data, avoid viewing the target data in isolation, and thus improve the accuracy of data quality analysis.

[0079] S6, inputting each to-be-classified data group corresponding to the current target data into the trained classification model respectively, and obtaining the correlation probability value between the current target data and the corresponding target reference data.

[0080] Among them, the correlation probability value between the target data and the corresponding target reference data is obtained according to the trained classification model, which is used to quantitatively represent the strength of the correlation between the target data and the target reference data, providing an important basis for subsequent more accurate evaluation of the quality status of the target data.

[0081] The correlation probability value is a value between 0 and 1, where 0 indicates that there is almost no correlation between the two and 1 indicates that there is a strong correlation between the two.

[0082] The above-mentioned step of inputting each data group to be classified corresponding to the current target data into the trained classification model respectively, and obtaining the correlation probability value between the current target data and the corresponding target reference data, obtains the correlation probability value to quantify the strength of the correlation between the target data and the target reference data, and provides an important basis for subsequent more accurate evaluation of the quality status of the target data.

[0083] S7, obtaining a second data quality score corresponding to the current target data according to the correlation probability value between the current target data and each corresponding target reference data, the prediction deviation degree corresponding to each target reference data corresponding to the current target data, and the prediction deviation degree corresponding to the current target data.

[0084] Optionally, S7 includes the following steps:

[0085] S71, based on any target reference data corresponding to the current target data, based on the correlation probability value between the current reference data and the current target reference data, and the prediction deviation degree corresponding to the current target reference data, obtain the first scoring influence parameter corresponding to the current target data, wherein the first scoring influence parameter is positively correlated with the corresponding correlation probability value, and the first scoring influence parameter is negatively correlated with the corresponding prediction deviation degree.

[0086] S72, traverse all target reference data corresponding to the current target data, obtain all first scoring influence parameters corresponding to the current target data, and determine the average value of all first scoring influence parameters corresponding to the current target data as the second scoring influence parameter corresponding to the current target data.

[0087] S73, obtaining a second data quality score corresponding to the current target data according to the second score influencing parameter and the prediction deviation degree corresponding to the current target data, wherein the second data quality score is negatively correlated with the corresponding second score influencing parameter and the prediction deviation degree.

[0088] Among them, for the target data and target reference data with a correlation, if the correlation probability value given by the classification model indicates that the correlation between the two is weak, it can be characterized that the data error of the target data is large and the data quality is poor. Therefore, the first scoring influencing parameter is positively correlated with the correlation probability value, that is, it is negatively correlated with the second data quality score. And the smaller the prediction deviation of the target reference data, the higher the reliability of the target reference data, and the higher the reliability of the data quality of the target data characterized by the correlation probability value. Therefore, the first scoring influencing parameter is negatively correlated with the prediction deviation of the corresponding target reference data, that is, it is positively correlated with the second data quality score.

[0089] The above step of obtaining the second data quality score corresponding to the current target data based on the correlation probability value between the current target data and each corresponding target reference data, the prediction deviation degree corresponding to each target reference data corresponding to the current target data, and the prediction deviation degree corresponding to the current target data, comprehensively considers the correlation between the target data and each target reference data and the impact of the quality status of the target reference data itself on the target data quality assessment, and calculates the second data quality score corresponding to the current target data according to the set correlation rules, thereby improving the accuracy of data quality analysis.

[0090] The embodiment of the present invention establishes a connection between actual data and predicted data based on the rules and patterns of time series data learned by the time series prediction model during the training process, so as to examine the data quality from the dynamic perspective of the time series, and obtains the deviation degree threshold corresponding to N groups of initial time series data and the predicted deviation degree corresponding to each group of initial time series data according to each group of initial time series data and the predicted time series data corresponding to each group of initial time series data, so as to accurately judge whether the initial time series data conforms to the expected time series change law and the degree of deviation by predicting the degree of deviation, and screens out relatively stable data with a smaller deviation from the expected as initial reference data according to the comparison between the predicted degree of deviation and the degree of deviation threshold according to the comparison between the predicted degree of deviation and the degree of deviation threshold as the target data, and screens out data with a more obvious deviation from the expected situation as the target data, and uses the initial reference data as a reference benchmark for subsequent evaluation of the target data quality, and obtains the relevant probability value to quantify the strength of the correlation between the target data and the target reference data, and then comprehensively considers the correlation between the target data and each target reference data and the influence of the quality status of the target reference data itself on the target data quality evaluation, and calculates the second data quality score corresponding to the current target data according to the set correlation rule, thereby improving the accuracy of data quality analysis.

[0091] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0092] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A multidimensional data quality detection method based on a data middle station, wherein the multidimensional data quality detection method based on a data middle station is applied to a data middle station, and is characterized in that: The multi-dimensional data quality detection method based on the data center station includes: S1, obtaining a trained time series prediction model, a trained classification model, N groups of initial time series data, and a correlation between the initial time series data, wherein the correlation includes correlation and irrelevance, and N is an integer greater than 1; S2, inputting each set of initial time series data into the trained time series prediction model to obtain predicted time series data corresponding to each set of initial time series data; S3, according to each group of initial time series data and the predicted time series data corresponding to each group of initial time series data, obtaining the deviation degree threshold corresponding to the N groups of initial time series data and the predicted deviation degree corresponding to each group of initial time series data; S4, classifying the N groups of initial time series data into target data and initial reference data according to the prediction deviation degree and the deviation degree threshold, and obtaining a first data quality score of the corresponding initial reference data according to the prediction deviation degree corresponding to each initial reference data; S5, for any target data, according to the correlation between the initial time series data, the target reference data corresponding to the current target data is obtained from all the initial reference data, and a data group to be classified consisting of the current target data and each target reference data is obtained; S6, inputting each to-be-classified data group corresponding to the current target data into the trained classification model, and obtaining the correlation probability value between the current target data and the corresponding target reference data; S7, obtaining a second data quality score corresponding to the current target data according to the correlation probability value between the current target data and each corresponding target reference data, the prediction deviation degree corresponding to each target reference data corresponding to the current target data, and the prediction deviation degree corresponding to the current target data.

2. The multidimensional data quality detection method based on the data center according to claim 1 is characterized in that: The initial time series data includes actual data corresponding to the M1th preset time point to the M2th preset time point, and S2 includes the following steps: S21, for any initial time series data, input the current initial time series data into the trained time series prediction model to obtain the predicted time series data corresponding to the current initial time series data, wherein the predicted time series data includes the predicted data corresponding to the M1+1th preset time point to the M2+1th preset time point; S22, traverse all the initial time series data to obtain the predicted time series data corresponding to each group of initial time series data.

3. The multidimensional data quality detection method based on the data center according to claim 2 is characterized in that: S3 includes the following steps: S31, according to the actual data and the predicted data corresponding to each group of initial time series data at the Kth preset time point, obtaining the Kth prediction deviation value corresponding to each group of initial time series data, where K=M1+1, M1+2, ..., M2; S32, traverse K=M1+1, M1+2, ..., M2, and obtain M2-M1 prediction deviation values ​​corresponding to each group of initial time series data; S33, determining the sum of M2-M1 prediction deviation values ​​corresponding to each group of initial time series data as the total prediction deviation value corresponding to each group of initial time series data; S34, determining the average value of the actual data corresponding to each set of initial time series data from the M1th preset time point to the M2th preset time point as the average data value corresponding to each set of initial time series data; S35, determining the difference between the total value of the prediction deviation corresponding to each group of initial time series data and the corresponding average data value as the prediction deviation degree corresponding to each group of initial time series data.

4. The multi-dimensional data quality detection method based on the data center according to claim 3 is characterized in that: S3 also includes the following steps: S36, obtaining a probability distribution function of the prediction deviation degree corresponding to the N groups of initial time series data according to the prediction deviation degree corresponding to the N groups of initial time series data; S37, determining the third quartile corresponding to the prediction deviation degree probability distribution function as the deviation degree threshold corresponding to the N groups of initial time series data.

5. The multi-dimensional data quality detection method based on the data center according to claim 1 is characterized in that: S4 includes the following steps: S41, for any initial time series data, if the prediction deviation degree corresponding to the current initial time series data is less than the deviation degree threshold, classify the current initial time series data as initial reference data; S42, if the prediction deviation degree corresponding to the current initial time series data is greater than or equal to the deviation degree threshold, classifying the current initial time series data as target data; S43, obtaining a first data quality score corresponding to each initial reference data according to the prediction deviation degree corresponding to each initial reference data, wherein the first data quality score is negatively correlated with the corresponding prediction deviation degree.

6. The multi-dimensional data quality detection method based on the data center according to claim 1 is characterized in that: S5 includes the following steps: S51, for any target data, according to the correlation between the initial time series data, obtain the correlation between the current target data and each initial reference data; S52, for any initial reference data, if the correlation between the current target data and the current initial reference data is correlation, determining the current initial reference data as the target reference data corresponding to the current target data; S53, combining the current target data and the current target reference data to obtain a data group to be classified consisting of the current target data and the current target reference data; S54, traversing all initial reference data to obtain all data groups to be classified corresponding to the current target data; S55, traverse all target data to obtain all to-be-classified data groups corresponding to each target data.

7. The multi-dimensional data quality detection method based on the data center according to claim 6 is characterized in that: S7 includes the following steps: S71, according to any target reference data corresponding to the current target data, according to the correlation probability value between the current reference data and the current target reference data, and the prediction deviation degree corresponding to the current target reference data, obtain a first scoring influence parameter corresponding to the current target data, wherein the first scoring influence parameter is positively correlated with the corresponding correlation probability value, and the first scoring influence parameter is negatively correlated with the corresponding prediction deviation degree; S72, traversing all target reference data corresponding to the current target data, obtaining all first scoring influencing parameters corresponding to the current target data, and determining the average value of all first scoring influencing parameters corresponding to the current target data as the second scoring influencing parameter corresponding to the current target data; S73, obtaining a second data quality score corresponding to the current target data according to the second score influencing parameter and the prediction deviation degree corresponding to the current target data, wherein the second data quality score is negatively correlated with the corresponding second score influencing parameter and the prediction deviation degree.

Images