Data quality detection method and device, electronic equipment and storage medium

Abstract

The invention relates to a data quality detection method and device, electronic equipment and a storage medium. The method comprises the following steps: dividing a data set into a plurality of sub-data sets; performing dimension reduction processing on the sub-data set through a principal component analysis method; determining an initial similar center point of each sub-data set; performing clustering processing on each sub-data set according to the similarity to obtain a plurality of similarity category clusters; calculating the total similarity of the data set based on the plurality of similarity category clusters; and determining the data quality of the data set according to the total similarity of the data set. By adopting the method provided by the embodiment of the invention, the detection accuracy can be improved, the time cost is reduced, and the operation efficiency of the data quality detection method is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data quality detection method, device, electronic device and storage medium. Background Art

[0002] In the related technologies, data quality detection plays an important role in the development of business needs and the effectiveness of enterprise decision-making. At present, data quality detection is usually implemented based on functional dependency, conditional functional dependency, and association rules. This method performs data quality detection by using the minimum quality rule calculation criteria based on data item grouping and its credibility, mining algorithms, and using quality rules to detect erroneous data. This method is highly complex, consumes a lot of resources, and has poor applicability. Summary of the invention

[0003] The present disclosure provides a data quality detection method, device, electronic device and storage medium. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a data quality detection method is provided, comprising:

[0005] Divide the data set into multiple sub-data sets;

[0006] Performing dimensionality reduction processing on the sub-dataset by using a principal component analysis method;

[0007] Determining an initial similarity center point of each of the sub-data sets;

[0008] Performing clustering processing on each of the sub-datasets according to similarity to obtain a plurality of similarity category clusters;

[0009] Calculating the total similarity of the data set based on the multiple similarity category clusters;

[0010] The data quality of the data set is determined according to the total similarity of the data set.

[0011] In a possible implementation manner, determining the initial similarity center point of each of the sub-data sets includes:

[0012] For each data point x in the sub-dataset, determine the number of sample points within the neighborhood of the data point x;

[0013] Calculating the average distance of the sample points;

[0014] When the number of sample points within the neighborhood of the data point x with the average distance as the radius is greater than a threshold, the data point x is determined as a candidate initial similarity center point;

[0015] Among all the candidate initial similarity center points, select the candidate initial similarity center point c with the highest local similarity i ; Among them, c i represents the candidate initial similarity center point selected for the i-th time;

[0016] Remove the candidate initial similar center point c i The candidate initial similarity center point with the longest distance;

[0017] When p candidate initial similarity center points are removed, the remaining candidate initial similarity center points are determined as initial similarity center points.

[0018] In a possible implementation, the threshold is determined according to an average number of sample points contained in a circle with the average distance as a radius for each sample point.

[0019] In a possible implementation, clustering is performed on each of the sub-data sets according to similarity to obtain a plurality of similarity category clusters, including:

[0020] For the j-th sub-dataset, calculating the cosine similarity between each sample data in the j-th sub-dataset and the initial similarity center point of the j-th sub-dataset;

[0021] Classify the sample data except the j-th sub-dataset into the similarity category cluster with the greatest similarity;

[0022] Iteratively updating the initial similarity center point of the similarity category cluster;

[0023] When a preset iteration condition is met, the updating of the initial similarity center point of the similarity category cluster is stopped to obtain multiple similarity category clusters; wherein the preset iteration condition includes at least one of the following: the number of iterations reaches a set number of iterations, and the difference in square errors between two adjacent iteration processes is less than a preset value.

[0024] In a possible implementation, the calculating the total similarity of the data set based on the multiple similarity category clusters includes:

[0025] For each similarity category cluster, calculate the modified cosine similarity from each sample data in the similarity category cluster to the initial similarity center point of the updated similarity category cluster;

[0026] The modified cosine similarity of each similarity category cluster is summed to obtain the total similarity of the data set.

[0027] In a possible implementation manner, determining the data quality of the data set according to the total similarity of the data set includes:

[0028] Determining whether the total similarity of the data set is greater than or equal to a preset similarity;

[0029] When the total similarity of the data set is greater than or equal to a preset similarity, it is determined that the data quality of the data set is low.

[0030] According to a second aspect of an embodiment of the present disclosure, there is provided a data quality detection device, comprising:

[0031] A subset partitioning module is used to partition a data set into multiple sub-data sets;

[0032] A dimension reduction module, used for performing dimension reduction processing on the sub-dataset by using a principal component analysis method;

[0033] A center point determination module, used to determine an initial similarity center point of each of the sub-data sets;

[0034] A clustering module, used for performing clustering processing on each of the sub-datasets according to similarity to obtain a plurality of similarity category clusters;

[0035] A calculation module, configured to calculate the total similarity of the data set based on the multiple similarity category clusters;

[0036] The quality detection module is used to determine the data quality of the data set according to the total similarity of the data set.

[0037] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0038] processor;

[0039] a memory for storing instructions executable by the processor;

[0040] The processor is configured to execute the instructions to implement the data quality detection method as described in any one of the first aspects.

[0041] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute a data quality detection method as described in any one of the first aspects.

[0042] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data quality detection method as described in any one of the first aspects is implemented.

[0043] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0044] In the disclosed embodiment, a data set is divided into multiple sub-data sets; the sub-data sets are subjected to dimensionality reduction processing by principal component analysis method; the initial similarity center point of each sub-data set is determined; each sub-data set is subjected to clustering processing according to similarity to obtain multiple similarity category clusters; the total similarity of the data set is calculated based on the multiple similarity category clusters; the data quality of the data set is determined according to the total similarity of the data set. In this way, on the one hand, data quality detection can be performed by similarity, which reduces the complexity of the data quality detection method, improves detection accuracy, and reduces time cost; on the other hand, by performing parallel processing by dividing the sub-data sets and reducing the data dimensionality, the operating efficiency of the data quality detection method can also be improved.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0047] Figure 1 It is a flow chart of a data quality detection method shown in an embodiment of the present disclosure.

[0048] Figure 2 It is a flow chart of a data quality detection method shown in an embodiment of the present disclosure.

[0049] Figure 3 It is a structural diagram of a data quality detection method based on data similarity analysis shown in an embodiment of the present disclosure.

[0050] Figure 4 It is a block diagram of a data quality detection device shown in an embodiment of the present disclosure.

[0051] Figure 5 It is a block diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0054] As can be seen from the background technology, in the related technology, the K-means clustering algorithm is a classic machine learning method in the field of data mining, which has the advantages of being simple, efficient and easy to implement. The basic idea of ​​the K-means clustering algorithm is: first select k points as the initial cluster center points, then traverse the global data points, and assign the global data points to the class corresponding to the nearest center point according to the distance between the data points and the center points. After all the global data points are assigned, the next round of iteration is performed to calculate the new round of cluster center points. After that, iterate and update the cluster center continuously until the given iteration condition is reached or the clustering standard function converges.

[0055] Data quality measurement can provide organizations with valuable information and insights through in-depth analysis and mining of data, thereby supporting scientific decision-making, innovative development, and the potential for business growth. With the advent of the big data era, data has become one of the most important assets of an enterprise. By fully leveraging the value of data, organizations can better understand market trends, customer needs, and business performance, and achieve more efficient operations and more competitive strategic advantages. The realization of data value is not only reflected in the optimization of internal business, but also in innovation, the reconstruction of business models, and the intelligence of strategic decision-making. Effective use of data value can create more business opportunities for enterprises, improve market sensitivity, and gain a leading position in the increasingly fierce competition. Therefore, treating data as a strategic resource and tapping its deep value has become one of the indispensable core strategies of contemporary enterprises.

[0056] Data similarity can measure the similarity between two data sets or data points, which is reflected in their characteristics, structure or attributes. In the field of data science and information processing, understanding the similarity between data is crucial for effective analysis, classification and clustering. Commonly used similarity measurement methods include Euclidean distance, Manhattan distance, cosine similarity, etc. Cosine similarity represents the similarity between data points by the cosine value of the angle, and is widely used in fields such as text similarity and recommendation systems. By accurately calculating and comparing the similarity between data, we can more accurately understand the correlation between data and provide a reliable data foundation for decision-making and model building. Therefore, data similarity assessment is a key link in the process of data analysis and mining, and has irreplaceable value for in-depth understanding of the intrinsic structure and pattern of data sets.

[0057] In the related technology, data quality detection methods are usually implemented based on functional dependency, conditional functional dependency, and association rules. The implementation idea is: the minimum quality rule calculation criteria based on data item grouping and its credibility, mining algorithm, and the use of quality rules to detect erroneous data. This form of data quality rules draws on the credibility evaluation mechanism of association rules and the expression ability of conditional functional dependency, and uniformly describes functional dependency, conditional functional dependency, association rules, etc. It has the characteristics of simplicity, objectivity, comprehensiveness, and accurate detection of abnormal data. However, the complexity of this algorithm is too high, and it consumes a lot of resources. It is not suitable for big data scenarios: too high algorithm complexity will consume a lot of computing resources, resulting in a significant increase in the system's computing time when processing large-scale data, thereby reducing the overall performance of the system. This may cause the system to be unable to meet the user's real-time requirements, especially in scenarios that require rapid response. High algorithm complexity will make the system slow and unreliable, making it difficult to use commercially. Moreover, the prior conditions of this algorithm are complex, and it is difficult to meet such high prior conditions in reality; high prior conditions may make overly idealized assumptions about the distribution and characteristics of the data, resulting in the algorithm only being able to operate in specific situations; in actual environments, the distribution and characteristics of the data may change over time or be affected by multiple factors. If the algorithm cannot adapt to these changes, it will cause the algorithm to fail in long-term or diversified applications.

[0058] Based on this, the embodiments of the present disclosure provide a data quality detection method, device, electronic device and storage medium, which can be a data quality analysis method based on data similarity, and can perform data quality assessment through similarity, overcome the problem of complex algorithms in related technologies, improve algorithm accuracy and reduce time costs. On the one hand, in the embodiments of the present disclosure, the accuracy of the quality detection algorithm can be improved by selecting the initial similarity center point of the improved similarity clustering algorithm. On the other hand, the time cost can also be reduced by dividing the sub-data sets for parallel calculation and data dimensionality reduction, improving the algorithm availability and applicability, and making it easier to implement the method.

[0059] The data quality detection method, device, electronic device and storage medium provided by the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0060] Figure 1 FIG. 1 is a flow chart of a data quality detection method shown in an embodiment of the present disclosure. The data quality detection method can be applied to a server, for example, a single server or a server cluster. Figure 1 As shown, the data quality detection method may include the following steps.

[0061] S101, dividing a data set into multiple sub-data sets.

[0062] In the embodiment of the present disclosure, when performing data quality detection, the data set can be divided first, so as to adopt the parallel computing method to improve the algorithm operation efficiency. For example, the user data set can be obtained first, and then the data set can be divided into sub-data sets. For example, the data set M can be divided into multiple sub-data sets (m1, m2, m3, ... m n ), where n represents the number of sub-data sets. It can be understood that n can be determined based on data characteristics and prior experience, for example, experiments or selections can be performed between [2, √M].

[0063] S102, performing dimensionality reduction processing on the sub-dataset by using a principal component analysis method.

[0064] In the embodiment of the present disclosure, after the sub-datasets are divided, the data can be subjected to dimensionality reduction processing. For example, for each sub-dataset, when the sub-dataset is subjected to data dimensionality reduction contact force by the principal component analysis method, the average value of the data under each feature can be calculated first. The calculation formula may be as shown in formula (1). As a specific example, the data set may be, for example, a user's movie type preference feature, such as romantic themes, comedy themes, mystery themes, etc., or other features, which are not limited in this embodiment.

[0065]

[0066] Among them, X i,j It represents the j-th data value under the i-th feature in the sub-dataset, and m represents the number of samples.

[0067] Then, the center position of the data can be further adjusted by subtracting the above calculated average from all the data where the feature is located, that is, After that, a set of eigenvalues ​​and their corresponding eigenvectors can be obtained from the covariance matrix with P features. After sorting the eigenvalues, the P / 2 features with the largest eigenvalues ​​are selected as the principal components of the data set. In this way, data dimensionality reduction can be achieved. It can be understood that parallel processing can be achieved by dividing the sub-datasets and reducing the data dimensionality to improve the operation efficiency of the algorithm.

[0068] S103, determining the initial similarity center point of each sub-data set.

[0069] In the embodiment of the present disclosure, after the sub-datasets are subjected to dimensionality reduction processing by the principal component analysis method, the initial similarity center point of each sub-dataset can be determined. Exemplarily, for each sub-dataset (m1, m2, m3, ... m n ), the initial similarity center point of each sub-data set can be determined respectively. It can be understood that the initial similarity center point of each sub-data set can be multiple, and can be in the form of a set, that is, the initial similarity center point set of each sub-data set. Exemplarily, a point with a large local similarity and a relatively large distance can be selected as the initial center point.

[0070] In a possible implementation, determining the initial similarity center point of each sub-data set includes:

[0071] For each data point x in the sub-dataset, determine the number of sample points in the neighborhood of the data point x;

[0072] Calculate the average distance of sample points;

[0073] When the number of sample points within the neighborhood of the data point x with the average distance as the radius is greater than the threshold, the data point x is determined as a candidate initial similarity center point;

[0074] Among all candidate initial similarity center points, select the candidate initial similarity center point c with the highest local similarity i ; Among them, c i represents the candidate initial similarity center point selected for the i-th time;

[0075] Remove the initial similar center point c i The candidate initial similarity center point with the longest distance;

[0076] When p candidate initial similarity center points are removed, the remaining candidate initial similarity center points are determined as initial similarity center points.

[0077] In a further possible implementation, the threshold may be determined according to the average number of sample points contained in a circle with the average distance of each sample point as the radius.

[0078] In the embodiment of the present disclosure, when determining the initial similarity center point of each sub-data set, for each data point x of each sub-data set, first, the data point x can be set as the center with lp i is the number of sample points within the circular area of ​​the neighborhood. Then calculate the average distance d of all sample data points in the sub-dataset avg , the calculation formula can be entered into formula (2). Then, P min As a threshold for determining whether it is a candidate center point for the initial clustering, the threshold can be determined based on the average number of sample points contained in a circle with the average distance as the radius for each sample point. For example, it can be defined as avg The average number of sample points contained in a circle with a radius of each sample point can be calculated as shown in formula (3). avg The number of sample points within the neighborhood of the radius is greater than the threshold P min , the data point x can be determined as a candidate initial similarity center point. In this way, for each sub-data set, multiple candidate initial similarity center points can be obtained to form a candidate initial similarity center point set (HP).

[0079]

[0080] In formula (2), m represents the number of samples of data points in the sub-dataset, d(x i ,x j ) represents the sample point x i and x j The Euclidean distance between them, n represents the number of sample points in the neighborhood.

[0081] In formula (3), P min represents the threshold, |m| is the number of data points in the sub-dataset, cp i is centered at the data point, d avg The number of data points contained in a circle with radius .

[0082] Then, for each sub-dataset’s candidate initial similarity center point set HP, the point with the highest local similarity can be selected from the candidate initial similarity center points HP as the first initial similarity center point c i , remove the candidate initial data center point c from the candidate initial similarity center point set HP i Then select the initial similar center point c of the candidate i The candidate initial similarity center point with the longest distance is used as the second initial similarity center point, and the candidate initial similarity center point with the longest distance from the second initial similarity center point is removed. The remaining candidate initial similarity center points in the candidate initial similarity center point set HP are selected based on the principle ofi is the initial similarity center point selected for the i-th time, and x is any point in HP. Until the p-th candidate initial similarity center point is removed, k initial similarity center points are selected, and the loop iteration ends. Finally, the initial similarity center point set ID contains k initial similarity center points.

[0083] Thus, unlike the density center summarized in the related art, the initial similarity center point of the embodiment of the present disclosure is determined from a global perspective through the average distance and the minimum threshold, two parameters automatically generated according to the data set. This avoids the need for users to process through prior experience, thereby reducing the difficulty of using the data quality detection method and improving the applicability of the method.

[0084] S104, clustering each sub-data set according to the similarity to obtain a plurality of similarity category clusters.

[0085] In the embodiment of the present disclosure, data clustering can be performed on each sub-data set by calculating similarity, for example, clustering can be performed by calculating cosine similarity to obtain multiple similarity category clusters. It can be understood that the number of sub-data sets and the number of similarity category clusters can be the same or different.

[0086] In a possible implementation, clustering is performed on each of the sub-data sets according to similarity to obtain a plurality of similarity category clusters, including:

[0087] For the j-th sub-dataset, calculate the cosine similarity between each sample data in the j-th sub-dataset and the initial similarity center point of the j-th sub-dataset;

[0088] Classify the sample data except the j-th sub-dataset into the similarity category cluster with the greatest similarity;

[0089] Iteratively update the initial similarity center points of similarity category clusters;

[0090] When a preset iteration condition is met, the updating of the initial similarity center point of the similarity category cluster is stopped to obtain multiple similarity category clusters; wherein the preset iteration condition includes at least one of the following: the number of iterations reaches a set number of iterations, and the difference in square errors between two adjacent iteration processes is less than a preset value.

[0091] In the embodiment of the present disclosure, when clustering is performed on each of the sub-datasets according to the similarity to obtain multiple similarity category clusters, for the j-th sub-dataset, the cosine similarity of each sample data in the j-th sub-dataset to the initial similarity center point of the j-th sub-dataset can be calculated first, and the remaining sample data except the j-th sub-dataset can be assigned to the similarity category cluster with the largest similarity. Then, the initial similarity center point of the similarity category cluster can be iteratively updated. Repeat the above steps until the number of iterations meets the preset conditions, then stop updating the initial similarity center point of the similarity category cluster to obtain multiple similarity category clusters. It can be understood that the preset iteration condition includes at least one of the number of iterations reaching the set number of iterations and the difference in square errors of two adjacent iteration processes being less than a preset value. For example, the set number of iterations is reached or |E1-E2|≤γ is satisfied, E1 and E2 represent the square error values ​​of two iteration processes, and if the difference in square errors of two iterations is less than γ, the error sum of the clustering converges, which means that the loop iteration can end. Finally, k similarity category clusters are saved. It can be understood that the implementation of this process is similar to the process of the Kmeans algorithm. The similarity between data points can be calculated according to formula (4), and then the initial similarity center points of each similarity category cluster are compared, and the maximum similarity is selected to return to the similarity category cluster; in this way, the aforementioned steps are repeated to optimize the function until the function meets the preset iteration conditions and the training is completed.

[0092] S105, calculating the total similarity of the data set based on multiple similarity category clusters.

[0093] In the embodiment of the present disclosure, each sub-data set is clustered according to similarity to obtain multiple similarity category clusters, and then the total similarity of the data set can be calculated based on the multiple similarity category clusters. Exemplarily, the cosine similarity of each similarity category cluster can be calculated to calculate the total similarity of the data set by calculating multiple similarity category clusters.

[0094] In a possible implementation, the total similarity of the data set is calculated based on multiple similarity category clusters, including:

[0095] For each similarity category cluster, calculate the modified cosine similarity from each sample data in the similarity category cluster to the initial similarity center point of the updated similarity category cluster;

[0096] The modified cosine similarity of each similarity category cluster is summed up to obtain the total similarity of the data set.

[0097] In the embodiment of the present disclosure, when the total similarity of a data set is calculated based on multiple similarity category clusters, for each similarity category cluster, the modified cosine similarity of each sample data in the similarity category cluster to the initial similarity center point of the updated similarity category cluster can be calculated first, and its calculation formula can be as shown in formula (4). Then, all the calculated modified cosine similarities sim(S i , T i ) are summed to find the total similarity within the similarity category cluster, and the correction of the sum of k similarity category clusters is used as the total similarity of user data.

[0098]

[0099] Among them, sim(S i , T i ) represents the modified cosine similarity, S ik , T ik Respectively represent the values ​​of data points S and K under the i-th feature, It represents the average value of data points S and K under all features.

[0100] S106, determining the data quality of the data set according to the total similarity of the data set.

[0101] In an embodiment of the present disclosure, after calculating the total similarity of a data set by calculating multiple similarity category clusters, the data quality of the data set can be determined based on the total similarity of the data set to determine the data quality of the user data. Exemplarily, the relationship between the total similarity and the data quality is negatively correlated, that is, the higher the total similarity, the lower the data quality; the lower the total similarity, the higher the data quality. As a specific example, different levels can be set for data quality, and different levels correspond to different similarity ranges, and the data quality is determined according to the similarity range into which the total similarity falls.

[0102] In a possible implementation, determining the data quality of the data set according to the total similarity of the data set includes:

[0103] Determine whether the total similarity of the data set is greater than or equal to a preset similarity;

[0104] When the total similarity of the data set is greater than or equal to the preset similarity, it is determined that the data quality of the data set is low.

[0105] In the embodiment of the present disclosure, the data quality can be determined according to the size of the total similarity of the data set. Exemplarily, the total similarity of the data set can be compared with the preset similarity to determine whether the total similarity of the data set is greater than or equal to the preset similarity, and the specific value of the preset similarity can be set according to actual needs. If the total similarity of the data set is greater than or equal to the preset similarity, it can be considered that the data quality of the user data is low.

[0106] In the disclosed embodiment, a data set is divided into multiple sub-data sets; the sub-data sets are subjected to dimensionality reduction processing by principal component analysis method; the initial similarity center point of each sub-data set is determined; each sub-data set is subjected to clustering processing according to similarity to obtain multiple similarity category clusters; the total similarity of the data set is calculated based on the multiple similarity category clusters; the data quality of the data set is determined according to the total similarity of the data set. In this way, on the one hand, data quality detection can be performed by similarity, which reduces the complexity of the data quality detection method, improves detection accuracy, and reduces time cost; on the other hand, by performing parallel processing by dividing the sub-data sets and reducing the data dimensionality, the operating efficiency of the data quality detection method can also be improved.

[0107] To make the method provided by the embodiment of the present disclosure clearer, the following Figure 2 For explanation. Figure 2 As shown, the data quality detection method provided by the embodiment of the present disclosure includes:

[0108] S1, divide the data set M into multiple sub-data sets (m1, m2, m3, ... m n ), and adopts parallel computing methods to improve the operation efficiency of the algorithm. For the number of sub-data divided, it is necessary to select the number of sub-data divided according to data characteristics and prior experience. Specifically, in theory, as the sub-data set is divided, the operation efficiency of the algorithm will be improved. However, too many sub-data partitions may affect the final overall aggregation, resulting in a decrease in clustering accuracy.

[0109] S2. Use principal component analysis method to reduce the dimension of the data.

[0110] Furthermore, the S2 step includes:

[0111] S201. The principal component analysis method first needs to calculate the average value of the data under each feature, see formula (1).

[0112] S202, further adjust the center position of the data, and subtract the average value from all the data where the feature is located, that is,

[0113] S203, obtaining a set of eigenvalues ​​and their corresponding eigenvectors from the covariance matrix having P features. After sorting the eigenvalues, select P / 2 features with the largest eigenvalues ​​as the principal components of the data set.

[0114] S3. Determine the initial similarity center point set ID for each sub-data set. Specifically, select points with large local similarity and relatively large distance as initial similarity center points.

[0115] Furthermore, step S3 includes:

[0116] S301, set up lp i The neighborhood of data point x is defined as the number of points within a circle centered on data point x. The average distance of all sample points is taken as d avg , and its calculation formula is shown in formula (2).

[0117] S302, further setting P min As the threshold for determining whether it is an initial cluster candidate center point. min The value of is defined as the average number of sample points contained in a circle with d as the radius of each sample point. Its calculation formula is shown in formula (3). avg The number of sample data within the radius is greater than P min , then this point is the initial similarity center point. All candidate initial similarity center points are stored in the candidate initial similarity center point set (HP).

[0118] S303, select the point with the highest local similarity from the candidate initial similarity center point set HP as the first initial data center point c i , then remove c from HP i Then choose to leave c i The farthest point is taken as the second initial similarity center point, according to The remaining candidate initial similarity center points are selected according to the principle of . Until the kth initial similarity center point is selected, the loop iteration ends. Finally, the initial similarity center point set contains k initial similarity center points.

[0119] S4. Cluster the user data according to modified cosine similarity to obtain k similarity category clusters.

[0120] Furthermore, the S4 step includes:

[0121] S401, calculating the cosine similarity of each sample data to the initial similarity center just obtained, and assigning the remaining data samples to the similarity cluster with the largest similarity.

[0122] S402: Update the new center in each similar cluster.

[0123] S403, repeat S401 and S402 until the set number of iterations is reached or |E1-E2|≤γ is satisfied, where E1 and E2 represent the square error values ​​of the two iterative processes. If the difference in the square error of the two iterations is less than γ, the error sum of the clustering converges, which means that the loop iteration can end. Finally, k similarity category clusters are saved.

[0124] S5. Sum the similarities of the k similarity category clusters respectively.

[0125] Furthermore, the step S5 includes:

[0126] S501, calculate the modified cosine similarity between each sample data and the obtained initial similarity center point, see formula (4).

[0127] S502, calculate the modified cosine similarity sim (S i , T i ) to find the sum of similarities within the similarity category cluster.

[0128] S6. Sum the k similarity category clusters as the total similarity of the user data.

[0129] Specifically, the structure diagram of the data quality detection method based on data similarity analysis can be as follows: Figure 3 shown.

[0130] It should be noted that the specific implementation and technical effects of each step in this embodiment are similar to those of the above method embodiment and will not be repeated here.

[0131] Based on the same inventive concept, the embodiment of the present disclosure also provides a data quality detection device, such as Figure 4 As shown, Figure 4 is a block diagram of a data quality detection device shown in an embodiment of the present disclosure. Figure 4 , the data quality detection device 400 may include:

[0132] A subset partitioning module 410 is used to partition a data set into a plurality of sub-data sets;

[0133] A dimension reduction module 420 is used to perform dimension reduction processing on the sub-dataset by using a principal component analysis method;

[0134] A center point determination module 430, used to determine an initial similarity center point of each of the sub-data sets;

[0135] A clustering module 440 is used to perform clustering processing on each of the sub-data sets according to similarity to obtain multiple similarity category clusters;

[0136] A calculation module 450, configured to calculate the total similarity of the data set based on the multiple similarity category clusters;

[0137] The quality detection module 460 is used to determine the data quality of the data set according to the total similarity of the data set.

[0138] In a possible implementation manner, the center point determination module 430 is used to:

[0139] For each data point x in the sub-dataset, determine the number of sample points within the neighborhood of the data point x;

[0140] Calculating the average distance of the sample points;

[0141] When the number of sample points within the neighborhood of the data point x with the average distance as the radius is greater than a threshold, the data point x is determined as a candidate initial similarity center point;

[0142] Among all the candidate initial similarity center points, select the candidate initial similarity center point c with the highest local similarity i ; Among them, c i represents the candidate initial similarity center point selected for the i-th time;

[0143] Remove the candidate initial similar center point c i The candidate initial similarity center point with the longest distance;

[0144] When p candidate initial similarity center points are removed, the remaining candidate initial similarity center points are determined as initial similarity center points.

[0145] In a possible implementation, the threshold is determined according to an average number of sample points contained in a circle with the average distance as a radius for each sample point.

[0146] In a possible implementation, the clustering module 440 is used to:

[0147] For the j-th sub-dataset, calculating the cosine similarity between each sample data in the j-th sub-dataset and the initial similarity center point of the j-th sub-dataset;

[0148] Classify the sample data except the j-th sub-dataset into the similarity category cluster with the greatest similarity;

[0149] Iteratively updating the initial similarity center point of the similarity category cluster;

[0150] When a preset iteration condition is met, the updating of the initial similarity center point of the similarity category cluster is stopped to obtain multiple similarity category clusters; wherein the preset iteration condition includes at least one of the following: the number of iterations reaches a set number of iterations, and the difference in square errors between two adjacent iteration processes is less than a preset value.

[0151] In a possible implementation, the calculation module 450 is used to:

[0152] For each similarity category cluster, calculate the modified cosine similarity from each sample data in the similarity category cluster to the initial similarity center point of the updated similarity category cluster;

[0153] The modified cosine similarity of each similarity category cluster is summed to obtain the total similarity of the data set.

[0154] In a possible implementation, the quality detection module 460 is used to:

[0155] Determining whether the total similarity of the data set is greater than or equal to a preset similarity;

[0156] When the total similarity of the data set is greater than or equal to a preset similarity, it is determined that the data quality of the data set is low.

[0157] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0159] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0160] like Figure 5As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0161] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0162] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a data quality detection method. For example, in some embodiments, the data quality detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the data quality detection method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the data quality detection method in any other appropriate manner (e.g., by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The program code of the computer program product for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of the present disclosure, a storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A storage medium may be a machine-readable signal medium or a machine-readable storage medium. Storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0168] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0170] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data quality detection method, characterized in that: include: Divide the data set into multiple sub-data sets; Performing dimensionality reduction processing on the sub-dataset by using a principal component analysis method; Determining an initial similarity center point of each of the sub-data sets; Performing clustering processing on each of the sub-datasets according to similarity to obtain a plurality of similarity category clusters; Calculating the total similarity of the data set based on the multiple similarity category clusters; The data quality of the data set is determined according to the total similarity of the data set.

2. The data quality detection method according to claim 1, characterized in that: The determining of the initial similarity center point of each of the sub-data sets comprises: For each data point x in the sub-dataset, determine the number of sample points within the neighborhood of the data point x; Calculating the average distance of the sample points; When the number of sample points within the neighborhood of the data point x with the average distance as the radius is greater than a threshold, the data point x is determined as a candidate initial similarity center point; Among all the candidate initial similarity center points, select the candidate initial similarity center point c with the highest local similarity i ; Among them, c i represents the candidate initial similarity center point selected for the i-th time; Remove the candidate initial similar center point c i The candidate initial similarity center point with the longest distance; When p candidate initial similarity center points are removed, the remaining candidate initial similarity center points are determined as initial similarity center points.

3. The data quality detection method according to claim 2, characterized in that: The threshold is determined according to the average number of sample points contained in a circle with the average distance as the radius for each sample point.

4. The data quality detection method according to claim 1, characterized in that: The clustering process is performed on each of the sub-data sets according to the similarity to obtain a plurality of similarity category clusters, including: For the j-th sub-dataset, calculating the cosine similarity between each sample data in the j-th sub-dataset and the initial similarity center point of the j-th sub-dataset; Classify the sample data except the j-th sub-dataset into the similarity category cluster with the greatest similarity; Iteratively updating the initial similarity center point of the similarity category cluster; When a preset iteration condition is met, the updating of the initial similarity center point of the similarity category cluster is stopped to obtain multiple similarity category clusters; wherein the preset iteration condition includes at least one of the following: the number of iterations reaches a set number of iterations, and the difference in square errors between two adjacent iteration processes is less than a preset value.

5. The data quality detection method according to claim 4, characterized in that: The calculating the total similarity of the data set based on the multiple similarity category clusters includes: For each similarity category cluster, calculate the modified cosine similarity from each sample data in the similarity category cluster to the initial similarity center point of the updated similarity category cluster; The modified cosine similarity of each similarity category cluster is summed to obtain the total similarity of the data set.

6. The data quality detection method according to claim 1, characterized in that: Determining the data quality of the data set according to the total similarity of the data set includes: Determining whether the total similarity of the data set is greater than or equal to a preset similarity; When the total similarity of the data set is greater than or equal to a preset similarity, it is determined that the data quality of the data set is low.

7. A data quality detection device, characterized in that: include: A subset partitioning module is used to partition a data set into multiple sub-data sets; A dimension reduction module, used for performing dimension reduction processing on the sub-dataset by using a principal component analysis method; A center point determination module, used to determine an initial similarity center point of each of the sub-data sets; A clustering module, used for performing clustering processing on each of the sub-datasets according to similarity to obtain a plurality of similarity category clusters; A calculation module, configured to calculate the total similarity of the data set based on the multiple similarity category clusters; The quality detection module is used to determine the data quality of the data set according to the total similarity of the data set.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data quality detection method according to any one of claims 1 to 6.

9. A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data quality detection method as claimed in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the data quality detection method according to any one of claims 1 to 6 is implemented.

Images