A data governance processing method, apparatus, equipment, and medium

Abstract

The present application relates to the technical field of electronic digital data processing, and in particular to a data governance processing method, device, equipment and medium. The method includes: obtaining a list of associated information of an external data source; obtaining a list of attribute information of an external data source; if a certain external data source is of incremental type, obtaining the quality of data sent by the corresponding external data source, the amount of data sent at a single time and the time interval for sending data according to the associated information of the data sent by the corresponding external data source in the most recent several times; clustering the incremental external data sources to obtain several groups; obtaining the central external data source of each of the several groups, and determining the data governance process matching each central external data source as the data governance process matching the corresponding group. The present invention improves the efficiency of obtaining the data governance process of the external data source.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a processing method, device, equipment and medium for data management. Background Art

[0002] Some industries need to collect data from multiple external data sources. For example, banks need to collect data from multiple external data sources. The time intervals for banks to collect data from different data sources are also different. Some collect data once a day, some collect data once a week, some collect data once a month, and some collect data only when there is a need. In addition to the different time intervals for collecting data, the quality of data collected by banks from different data sources also varies. For example, the quality of data provided by some data sources is poor, while the quality of data provided by some data sources is good. In order to improve the security and reliability of data collected from the outside, it is usually necessary to execute corresponding data governance processes for data collected from external data sources. In the prior art, corresponding data governance processes are usually manually formulated for each external data source, which has the problem of low efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide a data governance processing method, device, equipment and medium to improve the efficiency of the data governance process of obtaining external data sources.

[0004] According to a first aspect of the present invention, a data governance processing method is provided, the processing method comprising the following steps:

[0005] Obtain an external data source's associated information list; the external data source's associated information list includes associated information of several external data sources, each external data source's associated information includes associated information of data recently sent by the corresponding external data source, and each data sent's associated information includes the completeness, accuracy, consistency, data volume, and sending time of the data sent that time.

[0006] Obtain an attribute information list of an external data source; the attribute information list of an external data source includes attribute information of several external data sources, and the attribute information of each external data source includes the type and importance of the corresponding external data source; the type of each external data source is incremental or full.

[0007] If the type of an external data source is incremental, the quality of the data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data are obtained based on the associated information of the data sent by the corresponding external data source for the most recent several times.

[0008] According to the quality of data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data and the importance, the incremental external data sources are clustered to obtain several groups; any group includes several incremental external data sources.

[0009] A central external data source of each of the plurality of groups is obtained, and a data governance process matching each central external data source is determined as a data governance process matching the corresponding group.

[0010] Furthermore, the processing method also includes:

[0011] If the type of an external data source is full-volume type, the quality of the data sent by the corresponding external data source and the amount of data sent at a single time are obtained according to the associated information of the data most recently sent by the corresponding external data source.

[0012] The external data sources of the full type are clustered according to the quality of the data sent by the external data sources of the full type, the amount of data sent at a single time, and the importance, to obtain several clusters; any cluster includes several external data sources of the full type.

[0013] A central external data source of each of the plurality of clusters is obtained, and a data governance process matching each central external data source is determined as a data governance process matching the corresponding cluster.

[0014] Further, obtaining the quality of data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data according to the association information of the data sent by the corresponding external data source for the most recent several times includes:

[0015] The quality of the data sent by the corresponding external data source is obtained according to the completeness, accuracy and consistency of the data sent by the incremental external data source for several times recently.

[0016] The amount of data sent by the corresponding external data source in a single time is obtained according to the amount of data sent by the external data source of the incremental type in the most recent several times.

[0017] The time interval for sending data from the corresponding external data source is obtained according to the sending time of the data sent by the external data source of the incremental type for the latest several times.

[0018] Further, obtaining the quality of data sent by the corresponding external data source according to the completeness, accuracy and consistency of the data sent by the incremental external data source for several times in the last few times includes:

[0019] Obtain a target judgment matrix; the elements of the target judgment matrix include important coefficients of data integrity compared to accuracy, important coefficients of data integrity compared to consistency, and important coefficients of data accuracy compared to consistency.

[0020] Obtaining the eigenvector of the target judgment matrix; the elements of the eigenvector of the target judgment matrix include a first weight corresponding to the completeness of the data, a second weight corresponding to the accuracy of the data, and a third weight corresponding to the consistency of the data.

[0021] If the target judgment matrix passes the consistency check, the first weight is determined as the weight corresponding to the completeness of the data, the second weight is determined as the weight corresponding to the accuracy of the data, and the third weight is determined as the weight corresponding to the consistency of the data.

[0022] The quality of data sent by an external data source of incremental type in the most recent several times is obtained based on the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data and weight corresponding to the consistency of the data.

[0023] The average quality of the data sent by the external data source of the incremental type for the most recent several times is determined as the quality of the data sent by the corresponding external data source.

[0024] Furthermore, clustering the incremental external data sources according to the quality of data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data, and the importance includes:

[0025] Obtain a feature vector of a first external data source; the feature vector of the first external data source includes the quality of data sent by the first external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the first external data source is any type of incremental external data source.

[0026] Obtain a feature vector of a second external data source; the feature vector of the second external data source includes the quality of data sent by the second external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the second external data source is any type of incremental external data source different from the first external data source.

[0027] Obtain the distance between the first external data source and the second external data source; the distance between the first external data source and the second external data source is the sum of the first sub-distance, the second sub-distance, the third sub-distance and the fourth sub-distance, the first sub-distance is the product of the difference in quality of data sent by the first external data source and the second external data source and the weight corresponding to the quality, the second sub-distance is the product of the difference in normalized single data volume sent by the first external data source and the second external data source and the weight corresponding to the data volume, the third sub-distance is the product of the difference in normalized time intervals for sending data between the first external data source and the second external data source and the weight corresponding to the time interval, and the fourth sub-distance is the product of the difference in importance of the first external data source and the second external data source and the weight corresponding to the importance.

[0028] According to a second aspect of the present invention, a data governance processing device is provided, the processing device comprising:

[0029] The first acquisition module is used to obtain a list of associated information of an external data source; the list of associated information of an external data source includes associated information of several external data sources, and the associated information of each external data source includes associated information of data recently sent by the corresponding external data source, and the associated information of each data sent includes the completeness, accuracy, consistency, data volume and sending time of the data sent that time.

[0030] The second acquisition module is used to obtain a list of attribute information of an external data source; the list of attribute information of an external data source includes attribute information of several external data sources, and the attribute information of each external data source includes the type and importance of the corresponding external data source; the type of each external data source is incremental or full.

[0031] The first judgment module is used to obtain the quality of data sent by the corresponding external data source, the amount of data sent at a single time and the time interval for sending data based on the associated information of the data sent by the corresponding external data source for several recent times if the type of the external data source is incremental.

[0032] The first clustering module is used to cluster the incremental external data sources according to the quality of the data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data and the importance, so as to obtain several groups; any group includes several incremental external data sources.

[0033] The first matching module is used to obtain a central external data source of each of the plurality of groups, and determine a data governance process matching each central external data source as a data governance process matching the corresponding group.

[0034] Furthermore, the processing device further includes:

[0035] The second judgment module is used to obtain the quality of data sent by the corresponding external data source and the amount of data sent at a single time according to the associated information of the data most recently sent by the corresponding external data source if the type of the external data source is full volume type.

[0036] The second clustering module is used to cluster the external data sources of the full type according to the quality of the data sent by the external data sources of the full type, the amount of data sent at a single time, and the importance, to obtain several clusters; any cluster includes several external data sources of the full type.

[0037] The second matching module is used to obtain a central external data source for each of the plurality of clusters, and determine a data governance process that matches each central external data source as a data governance process that matches the corresponding cluster.

[0038] Furthermore, the first judgment module includes:

[0039] The first quality acquisition module is used to acquire the quality of data sent by the corresponding external data source according to the completeness, accuracy and consistency of the data sent by the incremental external data source for several times recently.

[0040] The first data volume acquisition module is used to acquire the corresponding data volume sent by the external data source at a single time according to the data volume sent by the external data source of the incremental type for the latest several times.

[0041] The first time interval acquisition module is used to acquire the time interval for sending data from the corresponding external data source according to the sending time of the latest several data sent by the incremental external data source.

[0042] Furthermore, the first quality acquisition module includes:

[0043] The first matrix acquisition module is used to acquire a target judgment matrix; the elements of the target judgment matrix include important coefficients of data integrity compared to accuracy, important coefficients of data integrity compared to consistency, and important coefficients of data accuracy compared to consistency.

[0044] The first vector acquisition module is used to obtain the eigenvector of the target judgment matrix; the elements of the eigenvector of the target judgment matrix include a first weight corresponding to the completeness of the data, a second weight corresponding to the accuracy of the data, and a third weight corresponding to the consistency of the data.

[0045] The first determination module is used to determine the first weight as the weight corresponding to the completeness of the data, the second weight as the weight corresponding to the accuracy of the data, and the third weight as the weight corresponding to the consistency of the data if the target judgment matrix passes the consistency check.

[0046] The second quality acquisition module is used to obtain the quality of data sent by an external data source of incremental type in the latest several times according to the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data and weight corresponding to the consistency of the data.

[0047] The first averaging module is used to determine the average quality of the data sent by the external data source of the incremental type for several recent times as the quality of the data sent by the corresponding external data source.

[0048] Furthermore, the first clustering module includes:

[0049] The second vector acquisition module is used to obtain the characteristic vector of the first external data source; the characteristic vector of the first external data source includes the quality of data sent by the first external data source, the normalized amount of data sent in a single time, the normalized time interval and importance of sending data, and the first external data source is any type of incremental external data source.

[0050] The third vector acquisition module is used to obtain the characteristic vector of the second external data source; the characteristic vector of the second external data source includes the quality of the data sent by the second external data source, the normalized amount of data sent in a single time, the normalized time interval and importance of sending data, and the second external data source is any type of incremental external data source different from the first external data source.

[0051] A distance acquisition module is used to acquire the distance between the first external data source and the second external data source; the distance between the first external data source and the second external data source is the sum of a first sub-distance, a second sub-distance, a third sub-distance and a fourth sub-distance, the first sub-distance is the product of a difference value between the quality of data sent by the first external data source and the second external data source and a weight corresponding to the quality, the second sub-distance is the product of a difference value between the normalized amount of data sent at a single time by the first external data source and the second external data source and a weight corresponding to the amount of data, the third sub-distance is the product of a difference value between the normalized time intervals for sending data by the first external data source and the second external data source and a weight corresponding to the time interval, and the fourth sub-distance is the product of a difference value between the importance of the first external data source and the second external data source and a weight corresponding to the importance.

[0052] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data governance processing method when executing the computer program.

[0053] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned data governance processing method when executed by a processor.

[0054] Compared with the prior art, the present invention has at least the following beneficial effects:

[0055] The present invention can obtain the association information (including completeness, accuracy, consistency, data volume and sending time) of the data sent by each external data source in the most recent several times based on the obtained association information list of the external data source; the type and importance of each external data source can be obtained based on the attribute information list of the external data source; on this basis, for an external data source of incremental type, the quality of the data sent, the amount of data sent at a single time and the time interval for sending data are obtained according to the association information of the data sent at the most recent several times, and the external data sources of incremental type are clustered in combination with their importance to obtain several groups; the quality of the data sent, the amount of data sent at a single time, the time interval for sending data and the importance of the external data sources in the same group are relatively similar, and the external data sources in different groups are relatively similar. The quality of the data sent, the amount of data sent at a single time, the time interval for sending data and the importance of the data vary greatly; the present invention determines the data governance process that matches each central external data source as the data governance process that matches the corresponding group, which can avoid the process of separately matching the data governance process for each external data source in each group. It is only necessary to match the data governance process once for the central external data source of each group, which reduces the number of matching data governance processes and improves the efficiency of obtaining the data governance process for the external data source; and the quality, the amount of data sent at a single time, the time interval for sending data and the importance of the central external data source of each group are relatively similar to those of the external data sources within the group, so the accuracy of determining the data processing process that matches the central external data source as the data processing process for the external data source within the group is also high. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 A first flow chart of a data governance processing method provided in the first embodiment of the present invention;

[0058] Figure 2A flowchart of the steps of obtaining the quality of data sent by a corresponding external data source, the amount of data sent at a single time, and the time interval for sending data provided in the first embodiment of the present invention;

[0059] Figure 3 A flowchart of the steps of obtaining the quality of data sent by a corresponding external data source provided in the first embodiment of the present invention;

[0060] Figure 4 A flowchart of the steps of clustering an incremental external data source provided in the first embodiment of the present invention;

[0061] Figure 5 A flowchart of the steps of a process for obtaining a weight corresponding to data quality, a weight corresponding to data volume, a weight corresponding to a time interval, and a weight corresponding to an importance level provided in the first embodiment of the present invention;

[0062] Figure 6 A second flow chart of the data governance processing method provided in the first embodiment of the present invention;

[0063] Figure 7 A first schematic diagram of a data management processing device provided in Embodiment 2 of the present invention;

[0064] Figure 8 A schematic diagram of a first judgment module provided in Embodiment 2 of the present invention;

[0065] Fig. 9 A schematic diagram of a first quality acquisition module provided in Embodiment 2 of the present invention;

[0066] Fig.10 A schematic diagram of a first clustering module provided in Embodiment 2 of the present invention;

[0067] Fig.11 A schematic diagram of a distance acquisition module provided in Embodiment 2 of the present invention;

[0068] Fig.12 A second schematic diagram of a data governance processing device provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0070] Embodiment 1:

[0071] According to this embodiment, Figure 1 As shown, a data governance processing method is provided, and the processing method includes the following steps:

[0072] S100, obtaining a list of associated information of external data sources; the list of associated information of external data sources includes associated information of several external data sources, and the associated information of each external data source includes associated information of data recently sent by the corresponding external data source, and the associated information of each data sent includes the completeness, accuracy, consistency, data volume and sending time of the data sent this time.

[0073] In this embodiment, the association information list of the external data source is pre-established, and the association information list of the external data source includes a plurality of records, each record corresponding to an external data source.

[0074] In this embodiment, the above several times are empirical values, and the above several times are greater than or equal to 2 times.

[0075] In this embodiment, the completeness of the data is used to characterize the completeness of the data. The higher the value of the completeness, the stronger the completeness; the accuracy of the data is used to characterize the accuracy of the data. The higher the value of the accuracy, the stronger the accuracy; the consistency of the data is used to characterize the consistency of the data. The higher the value of the consistency, the stronger the consistency; those skilled in the art know that any method of obtaining the completeness, accuracy and consistency of data in the prior art falls within the protection scope of the present invention. As a specific embodiment, the data sent by the external data source is table data, the completeness of the data is obtained by the ratio of the number of missing values ​​in the table to the amount of data included in the table, the accuracy of the data is obtained by the proportion of data in the table that conforms to the format and range of the field, and the consistency of the data is obtained by the proportion of the number of records in the table that meet the preset constraints.

[0076] S200, obtaining a list of attribute information of external data sources; the list of attribute information of external data sources includes attribute information of several external data sources, and the attribute information of each external data source includes the type and importance of the corresponding external data source; the type of each external data source is incremental or full.

[0077] In this embodiment, the type of the external data source is incremental or full. The incremental external data source refers to an external data source that only sends data that has changed since the last synchronization, and the full external data source refers to an external data source that sends all data regardless of whether the data has changed. Compared with the full external data source, the interval between each data transmission of the incremental external data source is generally shorter and the time interval is relatively fixed.

[0078] In this embodiment, a larger value of the importance of the external data source indicates that the data sent by the external data source is more important.

[0079] S300, if the type of a certain external data source is incremental, then the quality of the data sent by the corresponding external data source, the amount of data sent at a single time and the time interval for sending data are obtained according to the association information of the data sent by the corresponding external data source for the last several times.

[0080] As a preferred embodiment, Figure 2 As shown, obtaining the quality of data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data according to the association information of the data sent by the corresponding external data source for the most recent several times includes:

[0081] S310, obtaining the quality of the data sent by the corresponding external data source according to the completeness, accuracy and consistency of the data sent by the incremental external data source for several times recently.

[0082] In this embodiment, the greater the value of the data quality, the higher the completeness, accuracy and consistency of the data. Figure 3 As shown, obtaining the quality of data sent by the corresponding external data source according to the completeness, accuracy and consistency of the data sent by the external data source of the incremental type for the most recent several times includes:

[0083] S311, obtaining a target judgment matrix; the elements of the target judgment matrix include important coefficients of data completeness compared to accuracy, important coefficients of data completeness compared to consistency, and important coefficients of data accuracy compared to consistency.

[0084] In this embodiment, the target judgment matrix is ​​a matrix of 3 rows and 3 columns, and the three elements of the first row of the matrix are respectively: the importance coefficient of the completeness of the data compared to the completeness, the importance coefficient of the completeness of the data compared to the accuracy, and the importance coefficient of the completeness of the data compared to the consistency; the three elements of the second row of the matrix are respectively: the importance coefficient of the accuracy of the data compared to the completeness, the importance coefficient of the accuracy of the data compared to the accuracy, and the importance coefficient of the accuracy of the data compared to the consistency; the three elements of the third row of the matrix are respectively: the importance coefficient of the consistency of the data compared to the completeness, the importance coefficient of the consistency of the data compared to the accuracy, and the importance coefficient of the consistency of the data compared to the consistency. Among them, the importance coefficient of data completeness compared to completeness, the importance coefficient of data accuracy compared to accuracy, and the importance coefficient of data consistency compared to consistency are all 1, the product of the importance coefficient of data completeness compared to consistency and the importance coefficient of data consistency compared to completeness is 1, the product of the importance coefficient of data consistency compared to accuracy and the importance coefficient of data accuracy compared to consistency is 1, and the product of the importance coefficient of data completeness compared to accuracy and the importance coefficient of data accuracy compared to completeness is 1.

[0085] As an optional specific implementation, the importance coefficient of data completeness compared to accuracy, the importance coefficient of data completeness compared to consistency, and the importance coefficient of data accuracy compared to consistency are all empirical values.

[0086] S312, obtaining the eigenvector of the target judgment matrix; the elements of the eigenvector of the target judgment matrix include a first weight corresponding to the completeness of the data, a second weight corresponding to the accuracy of the data, and a third weight corresponding to the consistency of the data.

[0087] Those skilled in the art know that the process of obtaining the eigenvector of a matrix is ​​a prior art and will not be described in detail here.

[0088] S313, if the target judgment matrix passes the consistency check, the first weight is determined as the weight corresponding to the completeness of the data, the second weight is determined as the weight corresponding to the accuracy of the data, and the third weight is determined as the weight corresponding to the consistency of the data.

[0089] Those skilled in the art know that the process of determining whether a matrix passes the consistency check is a prior art and will not be described in detail here.

[0090] In this embodiment, if the target judgment matrix cannot pass the consistency check, the target judgment matrix is ​​updated until the updated target judgment matrix can pass the consistency check.

[0091] Based on S311 - S313 , the weight corresponding to the completeness of the data, the weight corresponding to the accuracy of the data, and the weight corresponding to the consistency of the data can be obtained.

[0092] S314, obtaining the quality of the data sent by the external data source of incremental type in the corresponding times recently according to the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data and weight corresponding to the consistency of the data.

[0093] In this embodiment, for the data sent for the ith time in the latest several times by an external data source of incremental type, the quality of the data sent for the ith time is obtained according to the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data, and weight corresponding to the consistency of the data; optionally, the quality of the data is obtained by using a weighted summation formula. The value range of i is 1 to n, and n is the number of times corresponding to the several times.

[0094] S315, determining the average quality of the data sent by the external data source of the incremental type for the most recent several times as the quality of the data sent by the corresponding external data source.

[0095] Based on S311 - S315 , the quality of the data sent by the external data source is obtained based on the completeness, accuracy and consistency of the data sent by the incremental external data source for several times recently.

[0096] As a preferred specific implementation, based on S311-S314, S315 is not executed, but the following steps are executed: according to the quality of the data sent by the external data source of the incremental type for the most recent several times, the quality change curve of the external data source of the incremental type is obtained, the horizontal axis of the quality change curve is the number of times the data is sent, and the vertical axis of the quality change curve is the quality of the data sent; if the fluctuation range of the data quality change curve is less than or equal to the preset fluctuation range, the average quality of the data sent by the external data source of the incremental type for the most recent several times is determined as the quality of the data sent by the corresponding external data source. If the fluctuation range of the data quality change curve is greater than the preset fluctuation range, the difference between the number of times the quality fluctuation occurs in the data quality change curve and the number of times the quality fluctuation occurs last is sequentially added to the preset number sequence; the preset number sequence is initialized as an empty sequence; if the variance of the elements in the number sequence is less than or equal to the preset variance threshold, the mean of the elements in the number sequence is determined as the specified number; the average quality of the data sent by the external data source of the incremental type in the most recent specified number of times is determined as the quality of the data sent by the external data source of the incremental type. If the variance of the elements in the frequency sequence is greater than the preset variance threshold, it is determined whether the elements in the frequency sequence meet the condition that the element at the rear position is greater than or equal to the element at the front position; if it is satisfied, the last element in the frequency sequence is determined as the specified number; the average quality of the data sent by the external data source of the incremental type in the latest specified number of times is determined as the quality of the data sent by the external data source of the incremental type. If it is not satisfied, the average quality of the data sent by the external data source of the incremental type for the latest several times is determined as the quality of the data sent by the corresponding external data source. Among them, if the difference between the quality of the data corresponding to each number in the data quality change curve and the mean of the data quality change curve is less than or equal to the first preset quality difference threshold, it is determined that the fluctuation range of the data quality change curve is less than or equal to the preset fluctuation range; otherwise, it is determined that the fluctuation range of the data quality change curve is greater than the preset fluctuation range. The mean of the data quality change curve is the mean of the quality of the data corresponding to all the times included in the data quality change curve. The number of times of quality fluctuation refers to the number of times the difference between the quality of the data and the quality of the last data is greater than or equal to the second preset quality difference threshold. The second preset quality difference threshold and the first preset quality difference threshold are both greater than 0, and optionally, the first preset quality difference threshold is greater than the second preset quality difference threshold, and the first preset quality difference threshold, the second preset quality difference threshold and the preset variance threshold are all empirical values. Thus, the quality of the data sent by the external data source is relatively accurate.

[0097] S320, obtaining the amount of data sent by the corresponding external data source in a single time according to the amount of data sent by the external data source of the incremental type in the most recent several times.

[0098] As an optional specific implementation, the average amount of data sent by the incremental external data source in the most recent several times is determined as the amount of data sent by the corresponding external data source in a single time.

[0099] S330, obtaining the time interval for sending data from the corresponding external data source according to the sending time of the data sent by the incremental external data source for several times recently.

[0100] As an optional specific implementation method, the average time interval of the latest several data transmissions of an incremental type external data source is determined as the time interval for the corresponding external data source to send data; the time interval between two adjacent transmissions is the difference between the transmission time of the data sent later in the adjacent transmissions and the transmission time of the data sent earlier in the adjacent transmissions.

[0101] S400, clustering the incremental external data sources according to the quality of the data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data and the importance, to obtain several groups; any group includes several incremental external data sources.

[0102] In this embodiment, the differences in the quality of data sent by external data sources in the same group obtained by clustering, the amount of data sent at a time, the time interval for sending data, and the importance of data are relatively small, while the differences in the quality of data sent by external data sources in different groups obtained by clustering, the amount of data sent at a time, the time interval for sending data, and the importance of data are relatively large. When clustering incremental external data sources, a four-dimensional space is established, and each dimension of the four-dimensional space is the quality of the data, the amount of data sent at a time after normalization, the time interval for sending data after normalization, and the importance of data. Each incremental external data source is regarded as an object in the four-dimensional space, and clustering is performed according to the distance between the objects. Optionally, a density clustering method is used for clustering, for example, a DBSCAN clustering method is used for clustering.

[0103] In this embodiment, the distance between any two incremental external data sources is related to the difference value of data quality between the two external data sources, the difference value of the amount of data sent at a time, the difference value of the time interval for sending data, and the difference value of the importance, and the difference value is the absolute value of the difference. As a preferred specific implementation method, Figure 4 As shown, clustering the incremental external data sources according to the quality of data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data, and the importance includes:

[0104] S410, obtaining a feature vector of a first external data source; the feature vector of the first external data source includes the quality of data sent by the first external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the first external data source is any type of incremental external data source.

[0105] In this embodiment, the quality of data sent by any external data source and the importance of the first external data source are both greater than 0 and less than or equal to 1. Those skilled in the art know that any normalization processing method in the prior art falls within the protection scope of the present invention.

[0106] S420, obtaining a feature vector of a second external data source; the feature vector of the second external data source includes the quality of data sent by the second external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the second external data source is any type of incremental external data source different from the first external data source.

[0107] S430, obtaining the distance between the first external data source and the second external data source; the distance between the first external data source and the second external data source is the sum of the first sub-distance, the second sub-distance, the third sub-distance and the fourth sub-distance, the first sub-distance is the product of the difference value of the quality of the data sent by the first external data source and the second external data source and the weight corresponding to the quality, the second sub-distance is the product of the difference value of the normalized single data amount sent by the first external data source and the second external data source and the weight corresponding to the data amount, the third sub-distance is the product of the difference value of the normalized time interval for sending data by the first external data source and the second external data source and the weight corresponding to the time interval, and the fourth sub-distance is the product of the difference value of the importance of the first external data source and the second external data source and the weight corresponding to the importance.

[0108] As an optional specific implementation, the weight corresponding to the data quality, the weight corresponding to the data volume, the weight corresponding to the time interval and the weight corresponding to the importance are empirical values. Figure 5 As shown, the process of obtaining the weight corresponding to the quality of the data, the weight corresponding to the amount of data, the weight corresponding to the time interval and the weight corresponding to the importance includes:

[0109] S431, obtain an intermediate judgment matrix; the elements of the intermediate judgment matrix include the importance coefficient of data quality compared to data quantity, the importance coefficient of data quality compared to time interval, the importance coefficient of data quality compared to importance, the importance coefficient of data quantity compared to time interval, the importance coefficient of data quantity compared to importance and the importance coefficient of time interval compared to importance.

[0110] In this embodiment, the target judgment matrix is ​​a matrix of 4 rows and 4 columns, and the 4 elements of the first row of the matrix are respectively: the importance coefficient of data quality compared to data quality, the importance coefficient of data quality compared to data quantity, the importance coefficient of data quality compared to time interval, and the importance coefficient of data quality compared to importance; the 4 elements of the second row of the matrix are respectively: the importance coefficient of data quantity compared to data quality, the importance coefficient of data quantity compared to data quantity, the importance coefficient of data quantity compared to time interval, and the importance coefficient of data quantity compared to importance; the 4 elements of the third row of the matrix are respectively: the importance coefficient of time interval compared to data quality, the importance coefficient of time interval compared to data quantity, the importance coefficient of time interval compared to time interval, and the importance coefficient of time interval compared to importance; the 4 elements of the fourth row of the matrix are respectively: the importance coefficient of importance compared to data quality, the importance coefficient of importance compared to data quantity, the importance coefficient of importance compared to time interval, and the importance coefficient of importance compared to importance. Among them, the importance coefficient of data quality compared to data quality, the importance coefficient of data quantity compared to data quantity, the importance coefficient of time interval compared to time interval and the importance coefficient of importance degree compared to importance degree are all 1, the product of the importance coefficient of data quality compared to data quantity and the importance coefficient of data quantity compared to data quality is 1, the product of the importance coefficient of data quality compared to time interval and the importance coefficient of time interval compared to data quality is 1, the product of the importance coefficient of data quality compared to importance degree and the importance coefficient of importance degree compared to data quality is 1, the product of the importance coefficient of data quantity compared to time interval and the importance coefficient of time interval compared to data quantity is 1, the product of the importance coefficient of data quantity compared to importance degree and the importance coefficient of importance degree compared to data quantity is 1, and the product of the importance coefficient of time interval compared to importance degree and the importance coefficient of importance degree compared to time interval is 1.

[0111] As an optional specific implementation method, the importance coefficient of quality compared to data volume, the importance coefficient of data quality compared to time interval, the importance coefficient of data quality compared to importance, the importance coefficient of data volume compared to time interval, the importance coefficient of data volume compared to importance and the importance coefficient of time interval compared to importance are all empirical values.

[0112] S432, obtaining the eigenvector of the intermediate judgment matrix; the elements of the eigenvector of the intermediate judgment matrix include a fourth weight corresponding to the quality of the data, a fifth weight corresponding to the amount of data, a sixth weight corresponding to the time interval and a seventh weight corresponding to the importance.

[0113] Those skilled in the art know that the process of obtaining the eigenvector of a matrix is ​​a prior art and will not be described in detail here.

[0114] S433, if the intermediate judgment matrix passes the consistency check, the fourth weight is determined as the weight corresponding to the quality of the data, the fifth weight is determined as the weight corresponding to the amount of data, the sixth weight is determined as the weight corresponding to the time interval, and the seventh weight is determined as the weight corresponding to the importance.

[0115] In this embodiment, if the intermediate judgment matrix cannot pass the consistency check, the intermediate judgment matrix is ​​updated until the updated intermediate judgment matrix can pass the consistency check.

[0116] Based on S431 - S433 , the weight corresponding to the quality of the data, the weight corresponding to the amount of data, the weight corresponding to the time interval, and the weight corresponding to the importance can be obtained.

[0117] Based on the method of S410-S430, the distance between any two incremental external data sources can be obtained; on this basis, the incremental external data sources can be clustered using a clustering method. Those skilled in the art know that any clustering method in the prior art falls within the protection scope of the present invention.

[0118] S500, obtaining a central external data source of each of the plurality of groups, and determining a data governance process matching each central external data source as a data governance process matching the corresponding group.

[0119] In this embodiment, the quality of data from the central external data source of any group is the average quality of data from the corresponding group, the amount of data sent at a single time from the central external data source of any group is the average amount of data sent at a single time by the corresponding group, the time interval for sending data from the central external data source of any group is the average time interval for sending data from the corresponding group, and the importance of the central external data source of any group is the average importance of the corresponding group.

[0120] As an optional specific implementation method, a relationship table of incremental external data sources is pre-established, and the relationship between external data sources with different data qualities, different amounts of data sent at a single time, different time intervals for sending data, and different levels of importance and data governance processes are stored in the relationship table. By matching the data quality, amount of data sent at a single time, time interval for sending data, and importance of each group's central external data source in the relationship table, the data governance process matching each central external data source can be obtained.

[0121] This embodiment can obtain the association information (including completeness, accuracy, consistency, data volume and sending time) of the data sent by each external data source in the most recent several times based on the obtained association information list of the external data source; the type and importance of each external data source can be obtained based on the attribute information list of the external data source; on this basis, for an external data source of incremental type, the quality of the data sent, the amount of data sent in a single time and the time interval for sending data are obtained according to the association information of the data sent in the most recent several times, and the external data sources of incremental type are clustered in combination with their importance to obtain several groups; the quality of the data sent, the amount of data sent in a single time, the time interval for sending data and the importance of the external data sources in the same group are relatively similar, and the external data sources in different groups are similar. The quality of the data sent, the amount of data sent at a single time, the time interval for sending data, and the importance of the data vary greatly. In this embodiment, the data governance process that matches each central external data source is determined as the data governance process that matches the corresponding group, which can avoid the process of matching the data governance process separately for each external data source in each group. It is only necessary to match the data governance process once for the central external data source of each group, which reduces the number of matching data governance processes and improves the efficiency of obtaining the data governance process for the external data source. Moreover, the quality, amount of data sent at a single time, time interval for sending data, and importance of the central external data source of each group are relatively similar to those of the external data sources within the group, so the accuracy of determining the data processing process that matches the central external data source as the data processing process for the external data source within the group is also high.

[0122] As a preferred embodiment, Figure 6 As shown, the processing method also includes:

[0123] S600: If the type of a certain external data source is full-volume type, the quality of the data sent by the corresponding external data source and the amount of data sent at a single time are obtained according to the association information of the data most recently sent by the corresponding external data source.

[0124] S700, clustering the external data sources of the full type according to the quality of the data sent by the external data sources of the full type, the amount of data sent at a single time, and the importance to obtain several clusters; any cluster includes several external data sources of the full type.

[0125] In this embodiment, the differences in the quality of data sent by external data sources in the same cluster, the amount of data sent at a time, and the importance of data are relatively small, while the differences in the quality of data sent by external data sources in different clusters are relatively large. When clustering external data sources of the full type, a three-dimensional space is established, and each dimension of the three-dimensional space is the quality of the data, the normalized amount of data sent at a time, and the importance of data. Each external data source of the full type is regarded as an object in the three-dimensional space, and clustering is performed according to the distance between the objects. Optionally, a density clustering method is used for clustering, for example, a DBSCAN clustering method is used for clustering.

[0126] In this embodiment, the distance between any two full-type external data sources is related to the difference value of the data quality between the two external data sources, the difference value of the normalized single-sent data volume, and the difference value of the importance. Optionally, a weighted summation formula is used to obtain the distance between two full-type external data sources, and the weight corresponding to the quality of the data, the weight corresponding to the single-sent data volume, and the weight corresponding to the importance are empirical values.

[0127] S800, obtaining a central external data source of each of the plurality of clusters, and determining a data governance process matching each central external data source as a data governance process matching the corresponding cluster.

[0128] In this embodiment, the quality of data from the central external data source of any cluster is the average quality of data from the corresponding cluster, the amount of data sent in a single time from the central external data source of any cluster is the average amount of data sent in a single time from the corresponding cluster, the time interval for sending data from the central external data source of any cluster is the average time interval for sending data from the corresponding cluster, and the importance of the central external data source of any cluster is the average importance of the corresponding cluster.

[0129] As an optional specific implementation method, a relationship table of full-scale external data sources is pre-established, and the relationship between external data sources with different data qualities, different amounts of data sent at a single time and different levels of importance and data governance processes is stored in the relationship table. By matching the data quality, amount of data sent at a single time and importance of the central external data source of each cluster in the relationship table, the data governance process matching each central external data source can be obtained.

[0130] Embodiment 2:

[0131] According to this embodiment, Figure 7 As shown, a data governance processing device is provided, and the processing device includes:

[0132] The first acquisition module 100 is used to obtain a list of associated information of an external data source; the list of associated information of an external data source includes associated information of several external data sources, and the associated information of each external data source includes associated information of data recently sent by the corresponding external data source, and the associated information of each data sent includes the completeness, accuracy, consistency, data volume and sending time of the data sent this time.

[0133] In this embodiment, the association information list of the external data source is pre-established, and the association information list of the external data source includes a plurality of records, each record corresponding to an external data source.

[0134] In this embodiment, the above several times are empirical values, and the above several times are greater than or equal to 2 times.

[0135] In this embodiment, the completeness of the data is used to characterize the completeness of the data. The higher the value of the completeness, the stronger the completeness; the accuracy of the data is used to characterize the accuracy of the data. The higher the value of the accuracy, the stronger the accuracy; the consistency of the data is used to characterize the consistency of the data. The higher the value of the consistency, the stronger the consistency; those skilled in the art know that any method of obtaining the completeness, accuracy and consistency of data in the prior art falls within the protection scope of the present invention. As a specific embodiment, the data sent by the external data source is table data, the completeness of the data is obtained by the ratio of the number of missing values ​​in the table to the amount of data included in the table, the accuracy of the data is obtained by the proportion of data in the table that conforms to the format and range of the field, and the consistency of the data is obtained by the proportion of the number of records in the table that meet the preset constraints.

[0136] The second acquisition module S200 is used to obtain a list of attribute information of an external data source; the list of attribute information of an external data source includes attribute information of several external data sources, and the attribute information of each external data source includes the type and importance of the corresponding external data source; the type of each external data source is incremental or full.

[0137] In this embodiment, the type of the external data source is incremental or full. The incremental external data source refers to an external data source that only sends data that has changed since the last synchronization, and the full external data source refers to an external data source that sends all data regardless of whether the data has changed. Compared with the full external data source, the interval between each data transmission of the incremental external data source is generally shorter and the time interval is relatively fixed.

[0138] In this embodiment, a larger value of the importance of the external data source indicates that the data sent by the external data source is more important.

[0139] The first judgment module 300 is used to obtain the quality of data sent by the corresponding external data source, the amount of data sent at a single time and the time interval for sending data based on the correlation information of the data sent by the corresponding external data source for the last several times if the type of the external data source is incremental.

[0140] As a preferred embodiment, Figure 8 As shown, the first judgment module 300 includes:

[0141] The first quality acquisition module 310 is used to acquire the quality of the data sent by the corresponding external data source according to the completeness, accuracy and consistency of the data sent by the incremental external data source for several times recently.

[0142] In this embodiment, the greater the value of the data quality, the higher the completeness, accuracy and consistency of the data. Fig. 9 As shown, the first quality acquisition module 310 includes:

[0143] The first matrix acquisition module 311 is used to acquire a target judgment matrix; the elements of the target judgment matrix include important coefficients of data integrity compared to accuracy, important coefficients of data integrity compared to consistency, and important coefficients of data accuracy compared to consistency.

[0144] In this embodiment, the target judgment matrix is ​​a matrix of 3 rows and 3 columns, and the three elements of the first row of the matrix are respectively: the importance coefficient of the completeness of the data compared to the completeness, the importance coefficient of the completeness of the data compared to the accuracy, and the importance coefficient of the completeness of the data compared to the consistency; the three elements of the second row of the matrix are respectively: the importance coefficient of the accuracy of the data compared to the completeness, the importance coefficient of the accuracy of the data compared to the accuracy, and the importance coefficient of the accuracy of the data compared to the consistency; the three elements of the third row of the matrix are respectively: the importance coefficient of the consistency of the data compared to the completeness, the importance coefficient of the consistency of the data compared to the accuracy, and the importance coefficient of the consistency of the data compared to the consistency. Among them, the importance coefficient of data completeness compared to completeness, the importance coefficient of data accuracy compared to accuracy, and the importance coefficient of data consistency compared to consistency are all 1, the product of the importance coefficient of data completeness compared to consistency and the importance coefficient of data consistency compared to completeness is 1, the product of the importance coefficient of data consistency compared to accuracy and the importance coefficient of data accuracy compared to consistency is 1, and the product of the importance coefficient of data completeness compared to accuracy and the importance coefficient of data accuracy compared to completeness is 1.

[0145] As an optional specific implementation, the importance coefficient of data completeness compared to accuracy, the importance coefficient of data completeness compared to consistency, and the importance coefficient of data accuracy compared to consistency are all empirical values.

[0146] The first vector acquisition module 312 is used to obtain the eigenvector of the target judgment matrix; the elements of the eigenvector of the target judgment matrix include a first weight corresponding to the completeness of the data, a second weight corresponding to the accuracy of the data, and a third weight corresponding to the consistency of the data.

[0147] Those skilled in the art know that the process of obtaining the eigenvector of a matrix is ​​a prior art and will not be described in detail here.

[0148] The first determination module 313 is used to determine the first weight as the weight corresponding to the completeness of the data, the second weight as the weight corresponding to the accuracy of the data, and the third weight as the weight corresponding to the consistency of the data if the target judgment matrix passes the consistency check.

[0149] Those skilled in the art know that the process of determining whether a matrix passes the consistency check is a prior art and will not be described in detail here.

[0150] In this embodiment, if the target judgment matrix cannot pass the consistency check, the target judgment matrix is ​​updated until the updated target judgment matrix can pass the consistency check.

[0151] Based on modules 311 - 313 , the weight corresponding to the completeness of the data, the weight corresponding to the accuracy of the data, and the weight corresponding to the consistency of the data can be obtained.

[0152] The second quality acquisition module 314 is used to obtain the quality of the data sent by the external data source of incremental type in the latest several times according to the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data and weight corresponding to the consistency of the data.

[0153] In this embodiment, for the data sent for the ith time in the latest several times by an external data source of incremental type, the quality of the data sent for the ith time is obtained according to the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data, and weight corresponding to the consistency of the data; optionally, the quality of the data is obtained by using a weighted summation formula. The value range of i is 1 to n, and n is the number of times corresponding to the several times.

[0154] The first averaging module 315 is used to determine the average quality of the data sent by the external data source of the incremental type for several recent times as the quality of the data sent by the corresponding external data source.

[0155] Based on modules 311 - 315 , the quality of the data sent by the external data source is obtained based on the completeness, accuracy and consistency of the data sent by the incremental external data source for several times in the last time.

[0156] As a preferred specific implementation, based on modules 311-314, module 315 is not executed, but another module is executed, which is used to: obtain a quality change curve of an external data source of type incremental type according to the quality of the data sent by the external data source of type incremental type for the last several times, the abscissa of the quality change curve is the number of times the data is sent, and the ordinate of the quality change curve is the quality of the data sent; if the fluctuation range of the data quality change curve is less than or equal to the preset fluctuation range, the average quality of the data sent by the external data source of type incremental type for the last several times is determined as the quality of the data sent by the corresponding external data source. If the fluctuation range of the data quality change curve is greater than the preset fluctuation range, the difference between the number of times the quality fluctuation occurs in the data quality change curve and the last number of times the quality fluctuation occurs is sequentially added to the preset number sequence; the preset number sequence is initialized as an empty sequence; if the variance of the elements in the number sequence is less than or equal to the preset variance threshold, the mean of the elements in the number sequence is determined as the specified number; the average quality of the data sent by the external data source of type incremental type in the last specified number of times is determined as the quality of the data sent by the external data source of type incremental type. If the variance of the elements in the frequency sequence is greater than the preset variance threshold, it is determined whether the elements in the frequency sequence meet the condition that the element at the rear position is greater than or equal to the element at the front position; if it is satisfied, the last element in the frequency sequence is determined as the specified number; the average quality of the data sent by the external data source of the incremental type in the latest specified number of times is determined as the quality of the data sent by the external data source of the incremental type. If it is not satisfied, the average quality of the data sent by the external data source of the incremental type for the latest several times is determined as the quality of the data sent by the corresponding external data source. Among them, if the difference between the quality of the data corresponding to each number in the data quality change curve and the mean of the data quality change curve is less than or equal to the first preset quality difference threshold, it is determined that the fluctuation range of the data quality change curve is less than or equal to the preset fluctuation range; otherwise, it is determined that the fluctuation range of the data quality change curve is greater than the preset fluctuation range. The mean of the data quality change curve is the mean of the quality of the data corresponding to all the times included in the data quality change curve. The number of times of quality fluctuation refers to the number of times the difference between the quality of the data and the quality of the last data is greater than or equal to the second preset quality difference threshold. The second preset quality difference threshold and the first preset quality difference threshold are both greater than 0, and optionally, the first preset quality difference threshold is greater than the second preset quality difference threshold, and the first preset quality difference threshold, the second preset quality difference threshold and the preset variance threshold are all empirical values. Thus, the quality of the data sent by the external data source is relatively accurate.

[0157] The first data volume acquisition module 320 is used to acquire the corresponding data volume sent by the external data source in a single time according to the data volume sent by the external data source of the incremental type for the latest several times.

[0158] As an optional specific implementation, the average amount of data sent by the incremental external data source in the most recent several times is determined as the amount of data sent by the corresponding external data source in a single time.

[0159] The first time interval acquisition module 330 is used to acquire the time interval for sending data from the corresponding external data source according to the sending time of the latest several data sent by the incremental external data source.

[0160] As an optional specific implementation method, the average time interval of the latest several data transmissions of an incremental type external data source is determined as the time interval for the corresponding external data source to send data; the time interval between two adjacent transmissions is the difference between the transmission time of the data sent later in the adjacent transmissions and the transmission time of the data sent earlier in the adjacent transmissions.

[0161] The first clustering module 400 is used to cluster the incremental external data sources according to the quality of the data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data and the importance, and obtain several groups; any group includes several incremental external data sources.

[0162] In this embodiment, the differences in the quality of data sent by external data sources in the same group obtained by clustering, the amount of data sent at a time, the time interval for sending data, and the importance of data are relatively small, while the differences in the quality of data sent by external data sources in different groups obtained by clustering, the amount of data sent at a time, the time interval for sending data, and the importance of data are relatively large. When clustering incremental external data sources, a four-dimensional space is established, and each dimension of the four-dimensional space is the quality of the data, the normalized amount of data sent at a time, the normalized time interval for sending data, and the importance of data. Each incremental external data source is regarded as an object in the four-dimensional space, and clustering is performed according to the distance between the objects. Optionally, a density clustering method is used for clustering, for example, a DBSCAN clustering method is used for clustering.

[0163] In this embodiment, the distance between any two incremental external data sources is related to the difference value of data quality between the two external data sources, the difference value of the amount of data sent at a time, the difference value of the time interval for sending data, and the difference value of the importance, and the difference value is the absolute value of the difference. As a preferred specific implementation method, Fig.10 As shown, the first clustering module 400 includes:

[0164] The second vector acquisition module 410 is used to obtain the characteristic vector of the first external data source; the characteristic vector of the first external data source includes the quality of the data sent by the first external data source, the normalized amount of data sent in a single time, the normalized time interval and importance of sending data, and the first external data source is any type of incremental external data source.

[0165] In this embodiment, the quality of data sent by any external data source and the importance of the first external data source are both greater than 0 and less than or equal to 1. Those skilled in the art know that any normalization processing method in the prior art falls within the protection scope of the present invention.

[0166] The third vector acquisition module 420 is used to obtain the characteristic vector of the second external data source; the characteristic vector of the second external data source includes the quality of the data sent by the second external data source, the normalized amount of data sent in a single time, the normalized time interval and importance of sending data, and the second external data source is any type of incremental external data source different from the first external data source.

[0167] The distance acquisition module 430 is used to acquire the distance between the first external data source and the second external data source; the distance between the first external data source and the second external data source is the sum of the first sub-distance, the second sub-distance, the third sub-distance and the fourth sub-distance, the first sub-distance is the product of the difference in quality of data sent by the first external data source and the second external data source and the weight corresponding to the quality, the second sub-distance is the product of the difference in the normalized amount of data sent in a single time between the first external data source and the second external data source and the weight corresponding to the data amount, the third sub-distance is the product of the difference in the normalized time interval for sending data between the first external data source and the second external data source and the weight corresponding to the time interval, and the fourth sub-distance is the product of the difference in importance between the first external data source and the second external data source and the weight corresponding to the importance.

[0168] As an optional specific implementation, the weight corresponding to the data quality, the weight corresponding to the data volume, the weight corresponding to the time interval and the weight corresponding to the importance are empirical values. Fig.11 As shown, the distance acquisition module 430 includes:

[0169] The second matrix acquisition module 431 is used to obtain an intermediate judgment matrix; the elements of the intermediate judgment matrix include the importance coefficient of data quality compared to data quantity, the importance coefficient of data quality compared to time interval, the importance coefficient of data quality compared to importance, the importance coefficient of data quantity compared to time interval, the importance coefficient of data quantity compared to importance and the importance coefficient of time interval compared to importance.

[0170] In this embodiment, the target judgment matrix is ​​a matrix of 4 rows and 4 columns, and the 4 elements of the first row of the matrix are respectively: the importance coefficient of data quality compared to data quality, the importance coefficient of data quality compared to data quantity, the importance coefficient of data quality compared to time interval, and the importance coefficient of data quality compared to importance; the 4 elements of the second row of the matrix are respectively: the importance coefficient of data quantity compared to data quality, the importance coefficient of data quantity compared to data quantity, the importance coefficient of data quantity compared to time interval, and the importance coefficient of data quantity compared to importance; the 4 elements of the third row of the matrix are respectively: the importance coefficient of time interval compared to data quality, the importance coefficient of time interval compared to data quantity, the importance coefficient of time interval compared to time interval, and the importance coefficient of time interval compared to importance; the 4 elements of the fourth row of the matrix are respectively: the importance coefficient of importance compared to data quality, the importance coefficient of importance compared to data quantity, the importance coefficient of importance compared to time interval, and the importance coefficient of importance compared to importance. Among them, the importance coefficient of data quality compared to data quality, the importance coefficient of data quantity compared to data quantity, the importance coefficient of time interval compared to time interval and the importance coefficient of importance degree compared to importance degree are all 1, the product of the importance coefficient of data quality compared to data quantity and the importance coefficient of data quantity compared to data quality is 1, the product of the importance coefficient of data quality compared to time interval and the importance coefficient of time interval compared to data quality is 1, the product of the importance coefficient of data quality compared to importance degree and the importance coefficient of importance degree compared to data quality is 1, the product of the importance coefficient of data quantity compared to time interval and the importance coefficient of time interval compared to data quantity is 1, the product of the importance coefficient of data quantity compared to importance degree and the importance coefficient of importance degree compared to data quantity is 1, and the product of the importance coefficient of time interval compared to importance degree and the importance coefficient of importance degree compared to time interval is 1.

[0171] As an optional specific implementation method, the importance coefficient of quality compared to data volume, the importance coefficient of data quality compared to time interval, the importance coefficient of data quality compared to importance, the importance coefficient of data volume compared to time interval, the importance coefficient of data volume compared to importance and the importance coefficient of time interval compared to importance are all empirical values.

[0172] The fourth vector acquisition module 432 is used to obtain the eigenvector of the intermediate judgment matrix; the elements of the eigenvector of the intermediate judgment matrix include the fourth weight corresponding to the quality of the data, the fifth weight corresponding to the amount of data, the sixth weight corresponding to the time interval and the seventh weight corresponding to the importance.

[0173] Those skilled in the art know that the process of obtaining the eigenvector of a matrix is ​​a prior art and will not be described in detail here.

[0174] The second determination module 433 is used to determine the fourth weight as the weight corresponding to the quality of the data, the fifth weight as the weight corresponding to the amount of data, the sixth weight as the weight corresponding to the time interval, and the seventh weight as the weight corresponding to the importance if the intermediate judgment matrix passes the consistency check.

[0175] In this embodiment, if the intermediate judgment matrix cannot pass the consistency check, the intermediate judgment matrix is ​​updated until the updated intermediate judgment matrix can pass the consistency check.

[0176] Based on modules 431 - 433 , the weight corresponding to the quality of the data, the weight corresponding to the amount of data, the weight corresponding to the time interval and the weight corresponding to the importance can be obtained.

[0177] Based on modules 410-430, the distance between any two incremental external data sources can be obtained; on this basis, the incremental external data sources can be clustered using a clustering method. Those skilled in the art know that any clustering method in the prior art falls within the protection scope of the present invention.

[0178] The first matching module 500 is used to obtain a central external data source of each of the plurality of groups, and determine a data governance process matching each central external data source as a data governance process matching the corresponding group.

[0179] In this embodiment, the quality of data from the central external data source of any group is the average quality of data from the corresponding group, the amount of data sent at a single time from the central external data source of any group is the average amount of data sent at a single time by the corresponding group, the time interval for sending data from the central external data source of any group is the average time interval for sending data from the corresponding group, and the importance of the central external data source of any group is the average importance of the corresponding group.

[0180] As an optional specific implementation method, a relationship table of incremental external data sources is pre-established, and the relationship between external data sources with different data qualities, different amounts of data sent at a single time, different time intervals for sending data, and different levels of importance and data governance processes are stored in the relationship table. By matching the data quality, amount of data sent at a single time, time interval for sending data, and importance of each group's central external data source in the relationship table, the data governance process matching each central external data source can be obtained.

[0181] This embodiment can obtain the association information (including completeness, accuracy, consistency, data volume and sending time) of the data sent by each external data source in the most recent several times based on the obtained association information list of the external data source; the type and importance of each external data source can be obtained based on the attribute information list of the external data source; on this basis, for an external data source of incremental type, the quality of the data sent, the amount of data sent in a single time and the time interval for sending data are obtained according to the association information of the data sent in the most recent several times, and the external data sources of incremental type are clustered in combination with their importance to obtain several groups; the quality of the data sent, the amount of data sent in a single time, the time interval for sending data and the importance of the external data sources in the same group are relatively similar, and the external data sources in different groups are similar. The quality of the data sent, the amount of data sent at a single time, the time interval for sending data, and the importance of the data vary greatly. In this embodiment, the data governance process that matches each central external data source is determined as the data governance process that matches the corresponding group, which can avoid the process of matching the data governance process separately for each external data source in each group. It is only necessary to match the data governance process once for the central external data source of each group, which reduces the number of matching data governance processes and improves the efficiency of obtaining the data governance process for the external data source. Moreover, the quality, amount of data sent at a single time, time interval for sending data, and importance of the central external data source of each group are relatively similar to those of the external data sources within the group, so the accuracy of determining the data processing process that matches the central external data source as the data processing process for the external data source within the group is also high.

[0182] As a preferred embodiment, Fig.12 As shown, the processing device also includes:

[0183] The second judgment module 600 is used to obtain the quality of data sent by the corresponding external data source and the amount of data sent at a single time according to the associated information of the data most recently sent by the corresponding external data source if the type of the external data source is full volume type.

[0184] The second clustering module 700 is used to cluster the external data sources of the full type according to the quality of the data sent by the external data sources of the full type, the amount of data sent at a single time and the importance, to obtain several clusters; any cluster includes several external data sources of the full type.

[0185] In this embodiment, the differences in the quality of data sent by external data sources in the same cluster, the amount of data sent at a time, and the importance of data are relatively small, while the differences in the quality of data sent by external data sources in different clusters are relatively large. When clustering external data sources of the full type, a three-dimensional space is established, and each dimension of the three-dimensional space is the quality of the data, the amount of data sent at a time, and the importance of data. Each external data source of the full type is regarded as an object in the three-dimensional space, and clustering is performed according to the distance between the objects. Optionally, a density clustering method is used for clustering, for example, a DBSCAN clustering method is used for clustering.

[0186] In this embodiment, the distance between any two full-type external data sources is related to the difference value of the data quality between the two external data sources, the difference value of the normalized single-sent data volume, and the difference value of the importance. Optionally, a weighted summation formula is used to obtain the distance between two full-type external data sources, and the weight corresponding to the quality of the data, the weight corresponding to the single-sent data volume, and the weight corresponding to the importance are empirical values.

[0187] The second matching module 800 is used to obtain a central external data source of each of the plurality of clusters, and determine the data governance process matching each central external data source as the data governance process matching the corresponding cluster.

[0188] In this embodiment, the quality of data from the central external data source of any cluster is the average quality of data from the corresponding cluster, the amount of data sent in a single time from the central external data source of any cluster is the average amount of data sent in a single time from the corresponding cluster, the time interval for sending data from the central external data source of any cluster is the average time interval for sending data from the corresponding cluster, and the importance of the central external data source of any cluster is the average importance of the corresponding cluster.

[0189] As an optional specific implementation method, a relationship table of full-scale external data sources is pre-established, and the relationship between external data sources with different data qualities, different amounts of data sent at a single time and different levels of importance and data governance processes is stored in the relationship table. By matching the data quality, amount of data sent at a single time and importance of the central external data source of each cluster in the relationship table, the data governance process matching each central external data source can be obtained.

[0190] Embodiment three:

[0191] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0192] Obtain an external data source's associated information list; the external data source's associated information list includes associated information of several external data sources, each external data source's associated information includes associated information of data recently sent by the corresponding external data source, and each data sent's associated information includes the completeness, accuracy, consistency, data volume, and sending time of the data sent that time.

[0193] Obtain an attribute information list of an external data source; the attribute information list of an external data source includes attribute information of several external data sources, and the attribute information of each external data source includes the type and importance of the corresponding external data source; the type of each external data source is incremental or full.

[0194] If the type of an external data source is incremental, the quality of the data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data are obtained based on the associated information of the data sent by the corresponding external data source for the most recent several times.

[0195] According to the quality of data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data and the importance, the incremental external data sources are clustered to obtain several groups; any group includes several incremental external data sources.

[0196] A central external data source of each of the plurality of groups is obtained, and a data governance process matching each central external data source is determined as a data governance process matching the corresponding group.

[0197] Embodiment 4:

[0198] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0199] Obtain an external data source's associated information list; the external data source's associated information list includes associated information of several external data sources, each external data source's associated information includes associated information of data recently sent by the corresponding external data source, and each data sent's associated information includes the completeness, accuracy, consistency, data volume, and sending time of the data sent that time.

[0200] Obtain an attribute information list of an external data source; the attribute information list of an external data source includes attribute information of several external data sources, and the attribute information of each external data source includes the type and importance of the corresponding external data source; the type of each external data source is incremental or full.

[0201] If the type of an external data source is incremental, the quality of the data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data are obtained based on the associated information of the data sent by the corresponding external data source for the most recent several times.

[0202] According to the quality of data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data and the importance, the incremental external data sources are clustered to obtain several groups; any group includes several incremental external data sources.

[0203] A central external data source of each of the plurality of groups is obtained, and a data governance process matching each central external data source is determined as a data governance process matching the corresponding group.

[0204] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0205] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A data governance processing method, characterized in that: The processing method comprises the following steps: Obtaining a list of association information of external data sources; the list of association information of external data sources includes association information of several external data sources, and the association information of each external data source includes association information of data recently sent by the corresponding external data source, and the association information of each data sent includes the completeness, accuracy, consistency, data volume and sending time of the data sent at that time; Obtaining a list of attribute information of an external data source; the list of attribute information of an external data source includes attribute information of a plurality of external data sources, and the attribute information of each external data source includes the type and importance of the corresponding external data source; the type of each external data source is incremental or full; If the type of an external data source is incremental, the quality of the data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data are obtained according to the association information of the data sent by the corresponding external data source for the most recent several times; Clustering the incremental external data sources according to the quality of data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data, and the importance of the data to obtain a number of groups; any group includes a number of incremental external data sources; Acquire a central external data source for each of the plurality of groups, and determine a data governance process matching each central external data source as a data governance process matching the corresponding group; Clustering of incremental external data sources according to the quality of data sent by the incremental external data sources, the amount of data sent at a time, the time interval for sending data, and the importance includes: Acquire a feature vector of a first external data source; the feature vector of the first external data source includes the quality of data sent by the first external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the first external data source is any type of incremental external data source; Obtaining a feature vector of a second external data source; the feature vector of the second external data source includes the quality of data sent by the second external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the second external data source is any incremental external data source different from the first external data source; Obtain the distance between the first external data source and the second external data source; the distance between the first external data source and the second external data source is the sum of the first sub-distance, the second sub-distance, the third sub-distance and the fourth sub-distance, the first sub-distance is the product of the difference in quality of data sent by the first external data source and the second external data source and the weight corresponding to the quality, the second sub-distance is the product of the difference in normalized single data volume sent by the first external data source and the second external data source and the weight corresponding to the data volume, the third sub-distance is the product of the difference in normalized time intervals for sending data between the first external data source and the second external data source and the weight corresponding to the time interval, and the fourth sub-distance is the product of the difference in importance of the first external data source and the second external data source and the weight corresponding to the importance.

2. The data governance processing method according to claim 1, characterized in that: The processing method also includes: If the type of an external data source is full, the quality of the data sent by the corresponding external data source and the amount of data sent at a single time are obtained according to the associated information of the data most recently sent by the corresponding external data source; Clustering the external data sources of the full type according to the quality of data sent by the external data sources of the full type, the amount of data sent at a single time, and the importance to obtain a number of clusters; any cluster includes a number of external data sources of the full type; A central external data source of each of the plurality of clusters is obtained, and a data governance process matching each central external data source is determined as a data governance process matching the corresponding cluster.

3. The data governance processing method according to claim 1, characterized in that: Acquiring the quality of data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data according to the association information of the data sent by the corresponding external data source for the most recent several times includes: According to the completeness, accuracy and consistency of the data sent by the incremental external data source for the most recent several times, the quality of the data sent by the corresponding external data source is obtained; According to the amount of data sent by the external data source of the incremental type for the most recent several times, the amount of data sent by the corresponding external data source in a single time is obtained; The time interval for sending data from the corresponding external data source is obtained according to the sending time of the data sent by the external data source of the incremental type for the latest several times.

4. The data governance processing method according to claim 3 is characterized in that: The quality of the data sent by the corresponding external data source is obtained based on the completeness, accuracy and consistency of the data sent by the external data source of the incremental type for the most recent several times, including: Obtaining a target judgment matrix; the elements of the target judgment matrix include an important coefficient of the degree of completeness of the data compared to the degree of accuracy, an important coefficient of the degree of completeness of the data compared to the degree of consistency, and an important coefficient of the degree of accuracy of the data compared to the degree of consistency; Obtaining an eigenvector of a target judgment matrix; the elements of the eigenvector of the target judgment matrix include a first weight corresponding to the completeness of the data, a second weight corresponding to the accuracy of the data, and a third weight corresponding to the consistency of the data; If the target judgment matrix passes the consistency check, the first weight is determined as the weight corresponding to the completeness of the data, the second weight is determined as the weight corresponding to the accuracy of the data, and the third weight is determined as the weight corresponding to the consistency of the data; According to the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data and weight corresponding to the consistency of the data, the quality of the data sent by the external data source of the incremental type at the corresponding times in the latest several times, the accuracy of the data and the consistency of the data, the quality of the data sent by the external data source of the incremental type at the corresponding times in the latest several times; The average quality of the data sent by the external data source of the incremental type for the most recent several times is determined as the quality of the data sent by the corresponding external data source.

5. A data management processing device, characterized in that: The processing device comprises: A first acquisition module is used to acquire a list of association information of an external data source; the list of association information of an external data source includes association information of a plurality of external data sources, and the association information of each external data source includes association information of data recently sent by the corresponding external data source, and the association information of each data sent includes the completeness, accuracy, consistency, data volume and sending time of the data sent at that time; The second acquisition module is used to acquire a list of attribute information of an external data source; the list of attribute information of an external data source includes attribute information of a plurality of external data sources, and the attribute information of each external data source includes a type and importance of the corresponding external data source; the type of each external data source is incremental or full; The first judgment module is used to obtain the quality of data sent by the corresponding external data source, the amount of data sent at a single time, and the time interval for sending data according to the association information of the data sent by the corresponding external data source for the latest several times if the type of the external data source is incremental; A first clustering module is used to cluster the incremental external data sources according to the quality of data sent by the incremental external data sources, the amount of data sent at a single time, the time interval for sending data, and the importance, to obtain a plurality of groups; any group includes a plurality of incremental external data sources; A first matching module, configured to obtain a central external data source for each of the plurality of groups, and determine a data governance process that matches each central external data source as a data governance process that matches the corresponding group; The first clustering module includes: A second vector acquisition module is used to acquire a feature vector of a first external data source; the feature vector of the first external data source includes the quality of data sent by the first external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the first external data source is any type of incremental external data source; a third vector acquisition module, used to acquire a feature vector of a second external data source; the feature vector of the second external data source includes the quality of data sent by the second external data source, the normalized amount of data sent at a single time, the normalized time interval and importance of sending data, and the second external data source is any incremental external data source different from the first external data source; A distance acquisition module is used to acquire the distance between a first external data source and a second external data source; the distance between the first external data source and the second external data source is the sum of a first sub-distance, a second sub-distance, a third sub-distance and a fourth sub-distance, the first sub-distance is the product of a difference value between the quality of data sent by the first external data source and the second external data source and a weight corresponding to the quality, the second sub-distance is the product of a difference value between the normalized amount of data sent at a single time by the first external data source and the second external data source and a weight corresponding to the amount of data, the third sub-distance is the product of a difference value between the normalized time intervals for sending data by the first external data source and the second external data source and a weight corresponding to the time interval, and the fourth sub-distance is the product of a difference value between the importance of the first external data source and the second external data source and a weight corresponding to the importance.

6. The data management processing device according to claim 5, characterized in that: The processing device further comprises: The second judgment module is used to obtain the quality of the data sent by the corresponding external data source and the amount of data sent at a single time according to the association information of the data most recently sent by the corresponding external data source if the type of the external data source is full volume type; A second clustering module is used to cluster the external data sources of the full type according to the quality of the data sent by the external data sources of the full type, the amount of data sent at a single time, and the importance, to obtain a plurality of clusters; The second matching module is used to obtain a central external data source for each of the plurality of clusters, and determine a data governance process that matches each central external data source as a data governance process that matches the corresponding cluster.

7. The data management processing device according to claim 5, characterized in that: The first judgment module includes: A first quality acquisition module is used to acquire the quality of data sent by a corresponding external data source according to the completeness, accuracy and consistency of data sent by the incremental external data source for several times in the last few times; A first data volume acquisition module is used to acquire the corresponding data volume sent by the external data source at a single time according to the data volume sent by the external data source of the incremental type for the latest several times; The first time interval acquisition module is used to acquire the time interval for sending data from the corresponding external data source according to the sending time of the latest several data sent by the incremental external data source.

8. The data management processing device according to claim 7, characterized in that: The first quality acquisition module includes: A first matrix acquisition module is used to acquire a target judgment matrix; the elements of the target judgment matrix include an important coefficient of the degree of completeness of the data compared to the degree of accuracy, an important coefficient of the degree of completeness of the data compared to the degree of consistency, and an important coefficient of the degree of accuracy of the data compared to the degree of consistency; A first vector acquisition module is used to acquire a characteristic vector of a target judgment matrix; the elements of the characteristic vector of the target judgment matrix include a first weight corresponding to the completeness of the data, a second weight corresponding to the accuracy of the data, and a third weight corresponding to the consistency of the data; A first determination module is used to determine the first weight as the weight corresponding to the completeness of the data, the second weight as the weight corresponding to the accuracy of the data, and the third weight as the weight corresponding to the consistency of the data if the target judgment matrix passes the consistency check; A second quality acquisition module is used to acquire the quality of the data sent by the external data source of incremental type at the corresponding time in the latest several times according to the completeness, accuracy, consistency, weight corresponding to the completeness of the data, weight corresponding to the accuracy of the data and weight corresponding to the consistency of the data. The first averaging module is used to determine the average quality of data sent by the external data source of incremental type for several recent times as the quality of data sent by the corresponding external data source.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data governance processing method as described in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data governance processing method as described in any one of claims 1 to 4 is implemented.

Images