Data quality management method and system for distributed data
By adopting a multi-dimensional analysis method based on life cycle in data quality management, the problem of single-dimensional management and ignoring data life cycle changes in the existing technology is solved, and more accurate and reliable data quality management is achieved.
Patent Information
- Application Number
- CN202510142548.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-10
- Publication Date
- 2025-03-11
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing data quality management methods focus on a single data attribute, ignore the multi-dimensional characteristics of data quality, and only focus on the accuracy of the final result, and ignore the process of change in data throughout the life cycle.
By determining the sub-data processing branch cycle based on the data processing life cycle, collecting dimensional change characteristics, determining the single-dimensional change characteristic cluster, and performing multiple analysis and calculations, constructing a multi-dimensional change characteristic factor curve, and finally calculating the multi-dimensional data quality management coefficient.
It realizes an integrated analysis of the multi-dimensional change characteristics of data throughout the life cycle, reflects the overall quality status of the data more comprehensively, and improves the accuracy and reliability of data quality management.
Smart Images
Figure CN119598135A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and in particular to a data quality management method and system for distributed data. Background Art
[0002] In today's big data era, data has become an important driving force for corporate decision-making, scientific research and social development. With the explosive growth of data volume, data quality issues have become increasingly prominent and have become one of the key factors restricting the value of data. Therefore, it is particularly important to develop an efficient and accurate data quality management method.
[0003] Existing data quality management methods often focus on a single attribute of data (such as accuracy), ignoring the multi-dimensional characteristics of data quality. This single-dimensional management method is difficult to fully reflect the overall quality of data. In addition, in existing technologies, data quality management is usually focused on the final result, that is, it is managed only based on indicators such as the accuracy of the data at the last moment. Although this method can provide a certain degree of data quality feedback, it has obvious limitations and ignores the change process of data throughout its life cycle, resulting in relatively single data quality management. Summary of the invention
[0004] The embodiments of the present invention provide a data quality management method and system for distributed data. The present invention can obtain the multi-dimensional change characteristics of data throughout its entire life cycle, perform integrated analysis of the multi-dimensional change characteristics, and more comprehensively reflect the quality status of the data, thereby improving the accuracy and reliability of data quality management.
[0005] In order to achieve the above object, the present invention provides a data quality management method for distributed data, comprising: Determine multiple sub-data processing branch cycles of the data to be quality managed based on the data processing life cycle, collect dimension change characteristics of the data to be quality managed corresponding to each sub-data processing branch cycle, and determine multiple single-dimensional change characteristic clusters according to the dimension change characteristics; Performing a first analysis on the single-dimensional change characteristic cluster to determine a single-dimensional change characteristic sector, and calculating a single-dimensional change characteristic factor of the single-dimensional change characteristic sector, wherein the single-dimensional change characteristic sector includes a superior single-dimensional change characteristic sector and an inferior single-dimensional change characteristic sector, and the single-dimensional change characteristic factor includes a superior single-dimensional change characteristic factor and an inferior single-dimensional change characteristic factor; Calculate a first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor; Performing a second analysis on the single-dimensional change characteristic cluster, and calculating a second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector based on the second analysis result; Determine curve construction points according to the first comprehensive single-dimensional change characteristic factor and the second comprehensive single-dimensional change characteristic factor, construct a multi-dimensional change characteristic factor curve, and calculate the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve.
[0006] Further, when performing a first analysis on the single-dimensional variation characteristic cluster, determining the single-dimensional variation characteristic sector, and calculating the single-dimensional variation characteristic factor of the single-dimensional variation characteristic sector, it includes: Acquire an ideal single-dimensional variation characteristic corresponding to the single-dimensional variation characteristic cluster; Performing a first analysis on the single-dimensional change characteristic cluster according to the ideal single-dimensional change characteristic, when the dimension change characteristic in the single-dimensional change characteristic cluster is greater than the ideal single-dimensional change characteristic, dividing the corresponding dimension change characteristic into the optimal single-dimensional change characteristic sector; When the dimension variation characteristic in the single-dimensional variation characteristic cluster is less than or equal to the ideal single-dimensional variation characteristic, the corresponding dimension variation characteristic is divided into the inferior single-dimensional variation characteristic sector; Calculating the superior single dimension change characteristic factor based on the superior single dimension change characteristic sector; The inferior single dimension variation characteristic factor is calculated based on the inferior single dimension variation characteristic sector.
[0007] Further, when calculating the superior single dimension change characteristic factor based on the superior single dimension change characteristic sector, it includes: Determine a sub-data processing branch cycle corresponding to each dimensional change characteristic in the superior single dimensional change characteristic sector; According to the sub-data processing branch cycle, a weight is configured for each dimension change characteristic; The superior single dimension change characteristic factor is calculated according to the following formula: ; ; ; ; Among them, g is the optimal single dimension change characteristic factor, e is a constant, h1 is the number of dimensional change characteristics in the optimal single dimension change characteristic sector, k j is the j-th dimension change characteristic in the optimal single-dimensional change characteristic sector, q j is the weight corresponding to the j-th dimension change characteristic, It is an ideal single-dimensional variation characteristic.
[0008] Further, when calculating the inferior single dimension change characteristic factor based on the inferior single dimension change characteristic sector, it includes: Calculating the sector mean and sector variance corresponding to the inferior single-dimensional variation characteristic sector; Constructing a sector interval according to the sector mean and the sector variance, analyzing the inferior single dimension change characteristic sector, and determining the number of dimension change characteristics falling into the sector interval; The inferior single dimension change characteristic factor is calculated according to the following formula: ; Among them, a is the inferior single dimension change characteristic factor, s1 is the number of dimension change characteristics in the inferior single dimension change characteristic sector, d f is the fth dimension change characteristic of the inferior single dimension change characteristic sector, is the sector mean, For all s2 is the maximum value in , and s2 is the number of dimensional change characteristics falling into the sector interval.
[0009] Further, when calculating the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor, it includes: Setting a first calculation factor for the superior single dimension change characteristic factor, and setting a second calculation factor for the inferior single dimension change characteristic factor; The first comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic cluster is calculated according to the following formula: ; Among them, r is the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster, t1 is the first calculation factor, and t2 is the second calculation factor.
[0010] Further, when performing a second analysis on the single-dimensional variation characteristic cluster and calculating a second comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic sector based on the second analysis result, it includes: Extracting the same dimension change characteristics from the single dimension change characteristic clusters, and constructing the same dimension change characteristic sequence; Count the number of identical sequences of characteristic sequences with the same dimension variation; Calculate the same score and sum value of the dimension change characteristics in each sequence of the same dimension change characteristics, and calculate the same sum value of all the same score and sum values; Extracting different dimensional change characteristics from the single dimensional change characteristic cluster, and calculating different sum values of all different dimensional change characteristics; A second comprehensive unidimensional variation characteristic factor of the unidimensional variation characteristic sector is obtained according to the same sequence quantity, the same sum value and the different sum value.
[0011] Further, when calculating the second comprehensive unidimensional variation characteristic factor of the unidimensional variation characteristic sector according to the number of identical sequences, the identical sum value and the different sum value, it includes: The second comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic sector is calculated according to the following formula: ; Among them, y is the second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector, u is the number of identical sequences, p1 is the same sum value, and p2 is the different sum value.
[0012] Further, when calculating the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve, it includes: Determine an initial curve construction point corresponding to an initial sub-data processing branch cycle; Determine the end curve construction point corresponding to the end sub-data processing branch cycle; Determine the median curve construction point corresponding to the median sub-data processing branch cycle; Determining an initial Euclidean distance between the initial curve construction point and the median curve construction point; Determining a terminal Euclidean distance between the terminal curve construction point and the median curve construction point; Extracting all first curve construction points between the initial curve construction point and the median curve construction point, respectively determining first Euclidean distances between the first curve construction points and the median curve construction point, and extracting the maximum first Euclidean distance; Extracting all second curve construction points between the end curve construction point and the median curve construction point, respectively determining second Euclidean distances between the second curve construction points and the end curve construction point, and extracting the maximum second Euclidean distance; The multi-dimensional data quality management coefficient of the data to be quality managed is calculated according to the initial Euclidean distance, the final Euclidean distance, the maximum first Euclidean distance and the maximum second Euclidean distance.
[0013] Further, when calculating the multi-dimensional data quality management coefficient of the data to be quality managed according to the initial Euclidean distance, the final Euclidean distance, the first maximum Euclidean distance and the second maximum Euclidean distance, it includes: ; Wherein, c is the multi-dimensional data quality management coefficient of the data to be quality managed, v1 is the initial Euclidean distance, v2 is the final Euclidean distance, v3 is the maximum first Euclidean distance, v4 is the maximum second Euclidean distance, b1 is the first calculation coefficient, b2 is the second calculation coefficient, b3 is the third calculation coefficient, b4 is the fourth calculation coefficient, b2>b4>b3>b1, b2>0, b4>0, b3>0, b1>0, n is The variance of .
[0014] In order to achieve the above object, the present invention also provides a data quality management system for distributed data, comprising: A data collection module, used to determine multiple sub-data processing branch cycles of the data to be quality managed based on the data processing life cycle, collect the dimension change characteristics of the data to be quality managed corresponding to each sub-data processing branch cycle, and determine multiple single-dimensional change characteristic clusters according to the dimension change characteristics; a first calculation module, configured to perform a first analysis on the single-dimensional change characteristic cluster, determine a single-dimensional change characteristic sector, and calculate a single-dimensional change characteristic factor of the single-dimensional change characteristic sector, wherein the single-dimensional change characteristic sector includes a superior single-dimensional change characteristic sector and an inferior single-dimensional change characteristic sector, and the single-dimensional change characteristic factor includes a superior single-dimensional change characteristic factor and an inferior single-dimensional change characteristic factor; A second calculation module, configured to calculate a first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor; a third calculation module, configured to perform a second analysis on the single-dimensional change characteristic cluster, and calculate a second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector based on the second analysis result; A data management module is used to determine the curve construction points according to the first comprehensive single-dimensional change characteristic factor and the second comprehensive single-dimensional change characteristic factor, construct a multi-dimensional change characteristic factor curve, and calculate the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve.
[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a data quality management method and system for distributed data. The method comprises the following steps: determining a sub-data processing branch cycle based on a life cycle, collecting dimension change characteristics, and determining a single-dimensional change characteristic cluster; performing a first analysis on the single-dimensional change characteristic cluster, and calculating a single-dimensional change characteristic factor; calculating a first comprehensive single-dimensional change characteristic factor according to a superior single-dimensional change characteristic factor and an inferior single-dimensional change characteristic factor; performing a second analysis on the single-dimensional change characteristic cluster, and calculating a second comprehensive single-dimensional change characteristic factor; constructing a multi-dimensional change characteristic factor curve according to the first comprehensive single-dimensional change characteristic factor and the second comprehensive single-dimensional change characteristic factor, and calculating a multi-dimensional data quality management coefficient. An integrated analysis is performed on the dimension change characteristics corresponding to each sub-data processing branch cycle, so as to more comprehensively reflect the overall quality status of the data and improve the accuracy and reliability of data quality management. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 A schematic diagram of a process for managing data quality of distributed data in an embodiment of the present invention is shown; Figure 2 A structural schematic diagram of a data quality management system for distributed data in an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0017] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0018] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0019] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0020] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0021] The following is a description of preferred embodiments of the present invention with reference to the accompanying drawings.
[0022] like Figure 1 As shown, an embodiment of the present invention discloses a data quality management method for distributed data, comprising: S110: determining multiple sub-data processing branch cycles of the data to be quality managed based on the data processing life cycle, collecting dimension change characteristics of the data to be quality managed corresponding to each sub-data processing branch cycle, and determining multiple single-dimensional change characteristic clusters according to the dimension change characteristics; In this embodiment, the data processing life cycle refers to the data completion cycle of the quality management data. If it is 16 days, the corresponding sub-data processing branch cycle is the 2nd day, the 4th day, the 6th day,..., the 16th day, which is specifically an arithmetic progression. The sub-data processing branch cycle can also be adaptively adjusted according to actual needs, such as every three days as a sub-data processing branch cycle.
[0023] In this embodiment, the dimensional change characteristics refer to the degree of data standardization, completeness, accuracy, consistency, timeliness, etc. For example, the degree of data standardization can be obtained according to (number of compliant data records / total number of data records) × 100%, and the degree of accuracy can be obtained according to (number of correct data records / total number of data records) × 100%. The rest are not shown one by one.
[0024] In this embodiment, a cluster refers to a collection of a group of data. A dimensional change characteristic of the same type corresponding to each sub-data processing branch cycle is extracted to construct a single-dimensional change characteristic cluster. For example, the data standardization degree corresponding to each sub-data processing branch cycle is extracted to construct a single-dimensional change characteristic cluster, and the integrity degree corresponding to each sub-data processing branch cycle is extracted to construct a single-dimensional change characteristic cluster.
[0025] S120: performing a first analysis on the single-dimensional change characteristic cluster, determining a single-dimensional change characteristic sector, and calculating a single-dimensional change characteristic factor of the single-dimensional change characteristic sector, wherein the single-dimensional change characteristic sector includes a superior single-dimensional change characteristic sector and an inferior single-dimensional change characteristic sector, and the single-dimensional change characteristic factor includes a superior single-dimensional change characteristic factor and an inferior single-dimensional change characteristic factor; In some embodiments of the present application, when performing a first analysis on the single-dimensional variation characteristic cluster, determining a single-dimensional variation characteristic sector, and calculating a single-dimensional variation characteristic factor of the single-dimensional variation characteristic sector, the method includes: Acquire an ideal single-dimensional variation characteristic corresponding to the single-dimensional variation characteristic cluster; Performing a first analysis on the single-dimensional change characteristic cluster according to the ideal single-dimensional change characteristic, when the dimension change characteristic in the single-dimensional change characteristic cluster is greater than the ideal single-dimensional change characteristic, dividing the corresponding dimension change characteristic into the optimal single-dimensional change characteristic sector; When the dimension variation characteristic in the single-dimensional variation characteristic cluster is less than or equal to the ideal single-dimensional variation characteristic, the corresponding dimension variation characteristic is divided into the inferior single-dimensional variation characteristic sector; Calculating the superior single dimension change characteristic factor based on the superior single dimension change characteristic sector; The inferior single dimension variation characteristic factor is calculated based on the inferior single dimension variation characteristic sector.
[0026] In this embodiment, a cluster may include multiple sectors for distinguishing data.
[0027] In this embodiment, each single-dimensional change characteristic cluster corresponds to an ideal single-dimensional change characteristic. As mentioned above, the ideal single-dimensional change characteristic corresponding to the degree of data standardization is 85%, and the ideal single-dimensional change characteristic corresponding to the degree of accuracy is 90%. The rest are not given one by one.
[0028] The beneficial effect of the above technical solution is: the present invention divides the single-dimensional change characteristic cluster into superior single-dimensional change characteristic sectors and inferior single-dimensional change characteristic sectors according to the ideal single-dimensional change characteristic, thereby ensuring the calculation accuracy of the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor.
[0029] In some embodiments of the present application, when calculating the superior single dimension change characteristic factor based on the superior single dimension change characteristic sector, it includes: Determine a sub-data processing branch cycle corresponding to each dimensional change characteristic in the superior single dimensional change characteristic sector; According to the sub-data processing branch cycle, a weight is configured for each dimension change characteristic; The superior single dimension change characteristic factor is calculated according to the following formula: ; ; ; ; Among them, g is the optimal single dimension change characteristic factor, e is a constant, h1 is the number of dimensional change characteristics in the optimal single dimension change characteristic sector, k j is the j-th dimension change characteristic in the optimal single-dimensional change characteristic sector, q j is the weight corresponding to the j-th dimension change characteristic, It is an ideal single-dimensional variation characteristic.
[0030] In this embodiment, as mentioned above, the closer to the 16th day, the greater the corresponding dimension change characteristic configuration weight, and the weight value range is [0.1, 3], which can be specifically allocated according to actual conditions.
[0031] The beneficial effect of the above technical solution is that the present invention calculates the superior single dimension change characteristic factor based on the superior single dimension change characteristic sector, which not only ensures the calculation accuracy of the superior single dimension change characteristic factor, but also lays the foundation for the calculation of the first comprehensive single dimension change characteristic factor.
[0032] In some embodiments of the present application, when calculating the inferior single dimension variation characteristic factor based on the inferior single dimension variation characteristic sector, it includes: Calculating the sector mean and sector variance corresponding to the inferior single-dimensional variation characteristic sector; Constructing a sector interval according to the sector mean and the sector variance, analyzing the inferior single dimension change characteristic sector, and determining the number of dimension change characteristics falling into the sector interval; The inferior single dimension change characteristic factor is calculated according to the following formula: ; Among them, a is the inferior single dimension change characteristic factor, s1 is the number of dimension change characteristics in the inferior single dimension change characteristic sector, d f is the fth dimension change characteristic of the inferior single dimension change characteristic sector, is the sector mean, For all s2 is the maximum value in , and s2 is the number of dimensional change characteristics falling into the sector interval.
[0033] In this embodiment, the sector interval includes a sector mean and a sector variance.
[0034] The beneficial effect of the above technical solution is: the present invention calculates the inferior single-dimensional change characteristic factor based on the inferior single-dimensional change characteristic sector, and also ensures the calculation accuracy of the inferior single-dimensional change characteristic factor, and then calculates the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster in combination with the superior single-dimensional change characteristic factor.
[0035] S130: Calculating a first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor; In some embodiments of the present application, when calculating the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor, it includes: Setting a first calculation factor for the superior single dimension change characteristic factor, and setting a second calculation factor for the inferior single dimension change characteristic factor; The first comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic cluster is calculated according to the following formula: ; Among them, r is the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster, t1 is the first calculation factor, and t2 is the second calculation factor.
[0036] In this embodiment, the first calculation factor is preferably 0.7, and the second calculation factor is preferably 0.3, which can be adjusted according to actual conditions.
[0037] The beneficial effect of the above technical solution is: the present invention calculates the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster based on the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor, without the need for human participation in the calculation, thereby ensuring the calculation accuracy of the first comprehensive single-dimensional change characteristic factor and providing a basis for the calculation of the multi-dimensional data quality management coefficient of the data to be quality managed.
[0038] S140: performing a second analysis on the single-dimensional change characteristic cluster, and calculating a second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector based on the second analysis result; In some embodiments of the present application, when performing a second analysis on the single-dimensional variation characteristic cluster and calculating a second comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic sector based on the second analysis result, the method includes: Extracting the same dimension change characteristics from the single dimension change characteristic clusters, and constructing the same dimension change characteristic sequence; Count the number of identical sequences of characteristic sequences with the same dimension variation; Calculate the same score and sum value of the dimension change characteristics in each sequence of the same dimension change characteristics, and calculate the same sum value of all the same score and sum values; Extracting different dimensional change characteristics from the single dimensional change characteristic cluster, and calculating different sum values of all different dimensional change characteristics; A second comprehensive unidimensional variation characteristic factor of the unidimensional variation characteristic sector is calculated according to the same sequence quantity, the same sum value and the different sum value.
[0039] The beneficial effect of the above technical solution is: the present invention calculates the second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector according to the same sequence number, the same total value and the different total values, which provides another basis for the calculation of the multi-dimensional data quality management coefficient of the data to be quality managed.
[0040] In some embodiments of the present application, when the second comprehensive unidimensional variation characteristic factor of the unidimensional variation characteristic sector is calculated according to the same sequence quantity, the same sum value and the different sum value, it includes: The second comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic sector is calculated according to the following formula: ; Among them, y is the second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector, u is the number of identical sequences, p1 is the same sum value, and p2 is the different sum value.
[0041] S150: Determine curve construction points according to the first comprehensive single-dimensional change characteristic factor and the second comprehensive single-dimensional change characteristic factor, construct a multi-dimensional change characteristic factor curve, and calculate the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve.
[0042] In this embodiment, the curve construction point is determined by taking the first comprehensive single-dimensional change characteristic factor as the vertical coordinate and the second comprehensive single-dimensional change characteristic factor as the horizontal coordinate.
[0043] In some embodiments of the present application, when calculating the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve, it includes: Determine an initial curve construction point corresponding to an initial sub-data processing branch cycle; Determine the end curve construction point corresponding to the end sub-data processing branch cycle; Determine the median curve construction point corresponding to the median sub-data processing branch cycle; Determining an initial Euclidean distance between the initial curve construction point and the median curve construction point; Determining a terminal Euclidean distance between the terminal curve construction point and the median curve construction point; Extracting all first curve construction points between the initial curve construction point and the median curve construction point, respectively determining first Euclidean distances between the first curve construction points and the median curve construction point, and extracting the maximum first Euclidean distance; Extracting all second curve construction points between the end curve construction point and the median curve construction point, respectively determining second Euclidean distances between the second curve construction points and the end curve construction point, and extracting the maximum second Euclidean distance; The multi-dimensional data quality management coefficient of the data to be quality managed is calculated according to the initial Euclidean distance, the final Euclidean distance, the maximum first Euclidean distance and the maximum second Euclidean distance.
[0044] In this embodiment, the initial sub-data processing branch cycle is the second day mentioned above, and the final sub-data processing branch cycle is the sixteenth day mentioned above.
[0045] In this embodiment, the number of curve construction points is determined. If the number is an odd number, the median curve construction point of the curve construction points is directly determined. If the number is an even number, the two middle median curve construction points are determined, and one is randomly selected as the median curve construction point, which can be determined according to the multi-dimensional change characteristic factor curve.
[0046] In this embodiment, when the multi-dimensional data quality management coefficient is larger, the data quality of the data to be quality managed is higher. Conversely, when the multi-dimensional data quality management coefficient is smaller, the data quality of the data to be quality managed is lower.
[0047] The beneficial effect of the above technical solution is: the present invention calculates the multi-dimensional data quality management coefficient of the data to be quality managed according to the initial Euclidean distance, the final Euclidean distance, the maximum first Euclidean distance and the maximum second Euclidean distance, fully analyzes each curve construction point, ensures the comprehensiveness of the calculation of the multi-dimensional data quality management coefficient, ensures the comprehensiveness and accuracy of the management of the data to be quality managed, does not require manual participation in management, avoids the subjective influence brought by manual labor, takes into account the entire data processing life cycle, and can also perform integrated analysis of all dimensional change characteristics to avoid the singleness and limitations of data management.
[0048] In some embodiments of the present application, when calculating the multi-dimensional data quality management coefficient of the data to be quality managed according to the initial Euclidean distance, the final Euclidean distance, the first maximum Euclidean distance, and the second maximum Euclidean distance, it includes: ; Wherein, c is the multi-dimensional data quality management coefficient of the data to be quality managed, v1 is the initial Euclidean distance, v2 is the final Euclidean distance, v3 is the maximum first Euclidean distance, v4 is the maximum second Euclidean distance, b1 is the first calculation coefficient, b2 is the second calculation coefficient, b3 is the third calculation coefficient, b4 is the fourth calculation coefficient, b2>b4>b3>b1, b2>0, b4>0, b3>0, b1>0, n is The variance of .
[0049] In order to further explain the technical idea of the present invention, the technical solution of the present invention is now described in combination with specific application scenarios.
[0050] Correspondingly, such as Figure 2 As shown, the present application also provides a data quality management system for distributed data, including: A data collection module, used to determine multiple sub-data processing branch cycles of the data to be quality managed based on the data processing life cycle, collect the dimension change characteristics of the data to be quality managed corresponding to each sub-data processing branch cycle, and determine multiple single-dimensional change characteristic clusters according to the dimension change characteristics; a first calculation module, configured to perform a first analysis on the single-dimensional change characteristic cluster, determine a single-dimensional change characteristic sector, and calculate a single-dimensional change characteristic factor of the single-dimensional change characteristic sector, wherein the single-dimensional change characteristic sector includes a superior single-dimensional change characteristic sector and an inferior single-dimensional change characteristic sector, and the single-dimensional change characteristic factor includes a superior single-dimensional change characteristic factor and an inferior single-dimensional change characteristic factor; A second calculation module, configured to calculate a first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor; a third calculation module, configured to perform a second analysis on the single-dimensional change characteristic cluster, and calculate a second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector based on the second analysis result; A data management module is used to determine the curve construction points according to the first comprehensive single-dimensional change characteristic factor and the second comprehensive single-dimensional change characteristic factor, construct a multi-dimensional change characteristic factor curve, and calculate the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve.
[0051] In the description of the above embodiments, specific features, structures, materials or characteristics may be combined in a suitable manner in any one or more embodiments or examples.
[0052] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed by the present invention may be used in combination with each other in any manner, and the fact that these combinations are not fully described in this specification is only for the sake of omitting space and saving resources.
[0053] Those skilled in the art can understand that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A data quality management method for distributed data, characterized in that: include: Determine multiple sub-data processing branch cycles of the data to be quality managed based on the data processing life cycle, collect dimension change characteristics of the data to be quality managed corresponding to each sub-data processing branch cycle, and determine multiple single-dimensional change characteristic clusters according to the dimension change characteristics; Performing a first analysis on the single-dimensional change characteristic cluster to determine a single-dimensional change characteristic sector, and calculating a single-dimensional change characteristic factor of the single-dimensional change characteristic sector, wherein the single-dimensional change characteristic sector includes a superior single-dimensional change characteristic sector and an inferior single-dimensional change characteristic sector, and the single-dimensional change characteristic factor includes a superior single-dimensional change characteristic factor and an inferior single-dimensional change characteristic factor; Calculate a first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor; Performing a second analysis on the single-dimensional change characteristic cluster, and calculating a second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector based on the second analysis result; Determine curve construction points according to the first comprehensive single-dimensional change characteristic factor and the second comprehensive single-dimensional change characteristic factor, construct a multi-dimensional change characteristic factor curve, and calculate the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve.
2. The data quality management method for distributed data according to claim 1, characterized in that: When the single-dimensional change characteristic cluster is analyzed for the first time, the single-dimensional change characteristic sector is determined, and the single-dimensional change characteristic factor of the single-dimensional change characteristic sector is calculated, it includes: Acquire an ideal single-dimensional variation characteristic corresponding to the single-dimensional variation characteristic cluster; Performing a first analysis on the single-dimensional change characteristic cluster according to the ideal single-dimensional change characteristic, when the dimension change characteristic in the single-dimensional change characteristic cluster is greater than the ideal single-dimensional change characteristic, dividing the corresponding dimension change characteristic into the optimal single-dimensional change characteristic sector; When the dimension variation characteristic in the single-dimensional variation characteristic cluster is less than or equal to the ideal single-dimensional variation characteristic, the corresponding dimension variation characteristic is divided into the inferior single-dimensional variation characteristic sector; Calculating the superior single dimension change characteristic factor based on the superior single dimension change characteristic sector; The inferior single dimension variation characteristic factor is calculated based on the inferior single dimension variation characteristic sector.
3. The data quality management method for distributed data according to claim 2, characterized in that: When calculating the superior single dimension change characteristic factor based on the superior single dimension change characteristic sector, it includes: Determine a sub-data processing branch cycle corresponding to each dimensional change characteristic in the superior single dimensional change characteristic sector; According to the sub-data processing branch cycle, a weight is configured for each dimension change characteristic; The superior single dimension change characteristic factor is calculated according to the following formula: ； ； ； ； Among them, g is the optimal single dimension change characteristic factor, e is a constant, h1 is the number of dimensional change characteristics in the optimal single dimension change characteristic sector, k j is the j-th dimension change characteristic in the optimal single-dimensional change characteristic sector, q j is the weight corresponding to the j-th dimension change characteristic, It is an ideal single-dimensional variation characteristic.
4. The data quality management method for distributed data according to claim 3, characterized in that: When calculating the inferior single dimension change characteristic factor based on the inferior single dimension change characteristic sector, it includes: Calculating the sector mean and sector variance corresponding to the inferior single-dimensional variation characteristic sector; Constructing a sector interval according to the sector mean and the sector variance, analyzing the inferior single dimension change characteristic sector, and determining the number of dimension change characteristics falling into the sector interval; The inferior single dimension change characteristic factor is calculated according to the following formula: ； Among them, a is the inferior single dimension change characteristic factor, s1 is the number of dimension change characteristics in the inferior single dimension change characteristic sector, d f is the fth dimension change characteristic of the inferior single dimension change characteristic sector, is the sector mean, For all s2 is the maximum value in , and s2 is the number of dimensional change characteristics falling into the sector interval.
5. The data quality management method for distributed data according to claim 4, characterized in that: When calculating the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor, it includes: Setting a first calculation factor for the superior single dimension change characteristic factor, and setting a second calculation factor for the inferior single dimension change characteristic factor; The first comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic cluster is calculated according to the following formula: ； Among them, r is the first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster, t1 is the first calculation factor, and t2 is the second calculation factor.
6. The data quality management method for distributed data according to claim 1, characterized in that: When performing a second analysis on the single-dimensional change characteristic cluster and calculating a second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector based on the second analysis result, the method includes: Extracting the same dimension change characteristics from the single dimension change characteristic clusters, and constructing the same dimension change characteristic sequence; Count the number of identical sequences of characteristic sequences with the same dimension variation; Calculate the same score and sum value of the dimension change characteristics in each sequence of the same dimension change characteristics, and calculate the same sum value of all the same score and sum values; Extracting different dimensional change characteristics from the single dimensional change characteristic cluster, and calculating different sum values of all different dimensional change characteristics; A second comprehensive unidimensional variation characteristic factor of the unidimensional variation characteristic sector is calculated according to the same sequence quantity, the same sum value and the different sum value.
7. The data quality management method for distributed data according to claim 6, characterized in that: When the second comprehensive unidimensional variation characteristic factor of the unidimensional variation characteristic sector according to the same sequence quantity, the same sum value and the different sum value comprises: The second comprehensive single-dimensional variation characteristic factor of the single-dimensional variation characteristic sector is calculated according to the following formula: ； Among them, y is the second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector, u is the number of identical sequences, p1 is the same sum value, and p2 is the different sum value.
8. The data quality management method for distributed data according to claim 1, characterized in that: When calculating the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve, it includes: Determine an initial curve construction point corresponding to an initial sub-data processing branch cycle; Determine the end curve construction point corresponding to the end sub-data processing branch cycle; Determine the median curve construction point corresponding to the median sub-data processing branch cycle; Determining an initial Euclidean distance between the initial curve construction point and the median curve construction point; Determining a terminal Euclidean distance between the terminal curve construction point and the median curve construction point; Extracting all first curve construction points between the initial curve construction point and the median curve construction point, respectively determining first Euclidean distances between the first curve construction points and the median curve construction point, and extracting the maximum first Euclidean distance; Extracting all second curve construction points between the end curve construction point and the median curve construction point, respectively determining second Euclidean distances between the second curve construction points and the end curve construction point, and extracting the maximum second Euclidean distance; The multi-dimensional data quality management coefficient of the data to be quality managed is calculated according to the initial Euclidean distance, the final Euclidean distance, the maximum first Euclidean distance and the maximum second Euclidean distance.
9. The data quality management method for distributed data according to claim 8, characterized in that: When calculating the multi-dimensional data quality management coefficient of the data to be quality managed according to the initial Euclidean distance, the final Euclidean distance, the first maximum Euclidean distance, and the second maximum Euclidean distance, it includes: ； Wherein, c is the multi-dimensional data quality management coefficient of the data to be quality managed, v1 is the initial Euclidean distance, v2 is the final Euclidean distance, v3 is the maximum first Euclidean distance, v4 is the maximum second Euclidean distance, b1 is the first calculation coefficient, b2 is the second calculation coefficient, b3 is the third calculation coefficient, b4 is the fourth calculation coefficient, b2>b4>b3>b1, b2>0, b4>0, b3>0, b1>0, n is The variance of .
10. A data quality management system for distributed data, applied to a data quality management method for distributed data as claimed in any one of claims 1 to 9, characterized in that: include: A data collection module, used to determine multiple sub-data processing branch cycles of the data to be quality managed based on the data processing life cycle, collect the dimension change characteristics of the data to be quality managed corresponding to each sub-data processing branch cycle, and determine multiple single-dimensional change characteristic clusters according to the dimension change characteristics; a first calculation module, configured to perform a first analysis on the single-dimensional change characteristic cluster, determine a single-dimensional change characteristic sector, and calculate a single-dimensional change characteristic factor of the single-dimensional change characteristic sector, wherein the single-dimensional change characteristic sector includes a superior single-dimensional change characteristic sector and an inferior single-dimensional change characteristic sector, and the single-dimensional change characteristic factor includes a superior single-dimensional change characteristic factor and an inferior single-dimensional change characteristic factor; A second calculation module, configured to calculate a first comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic cluster according to the superior single-dimensional change characteristic factor and the inferior single-dimensional change characteristic factor; a third calculation module, configured to perform a second analysis on the single-dimensional change characteristic cluster, and calculate a second comprehensive single-dimensional change characteristic factor of the single-dimensional change characteristic sector based on the second analysis result; A data management module is used to determine the curve construction points according to the first comprehensive single-dimensional change characteristic factor and the second comprehensive single-dimensional change characteristic factor, construct a multi-dimensional change characteristic factor curve, and calculate the multi-dimensional data quality management coefficient of the data to be quality managed based on the multi-dimensional change characteristic factor curve.