Mass data processing method based on data warehouse base

Abstract

The invention relates to the technical field of data processing, in particular to a mass data processing method based on a data warehouse base, which comprises the following steps of: acquiring data in the data warehouse base, acquiring local data of each data, acquiring local density of each data according to the local data of each data, and calculating the mass data of each data according to the local density of each data. Obtaining an extreme point in local data of each piece of data, obtaining an outlier degree of each piece of data according to the extreme point, obtaining a feature value of each piece of data according to the local density and the outlier degree of each piece of data, and obtaining a center point according to the feature value of each piece of data; clustering all data according to the central point to obtain a noise cluster; according to the Canopy clustering algorithm, the accuracy of the Canopy clustering algorithm is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a method for processing massive data based on a data warehouse base. Background Art

[0002] As an efficient data storage and management platform, data warehouse (Data Warehouse, referred to as data warehouse) plays an important role in enterprise data integration, decision support and other aspects. However, with the continuous increase in data volume, traditional data warehouse architecture and processing methods are facing performance bottlenecks, and there is an urgent need for innovation and improvement in data collection, storage, processing and other aspects. Massive data processing mainly refers to the efficient storage, query, calculation and analysis of massive data, and ensuring the accuracy, real-time and scalability of data processing. Among them, ensuring the accuracy of data is particularly important. If the data accuracy is low, the accuracy of subsequent calculations and analyses based on the data will be reduced, and even lead to errors in some decisions.

[0003] The patent application document with the publication number CN105095266A currently proposes a clustering optimization method and system based on the Canopy algorithm, including: grouping all the original data based on the Canopy algorithm to obtain N Canopy sets and the center of each Canopy set; using the number of Canopy sets N as the number k of partitions to be constructed by the K-means clustering algorithm; using the center of each Canopy set as the cluster center of the k divided clusters; based on the above-determined number of clusters k and cluster centers, using the K-means clustering algorithm to perform clustering optimization processing on all the original data, and outputting the clustering optimization results.

[0004] As an efficient clustering method, the Canopy clustering algorithm has a wide range of applications in noise removal in massive data processing. It can quickly cluster a large amount of data through a first round of rough clustering and a second round of more refined clustering, thereby separating the noise data. Although the Canopy clustering algorithm can quickly cluster data, thereby separating the noise data by clustering them into a single category, it will randomly select some data as center points for clustering during use. Randomly selecting center points has great uncertainty, which may result in the center points being unable to represent the distribution characteristics of the data well. Especially when there is noise in massive data resulting in uneven data distribution, randomly selected center points may result in unstable or inaccurate clustering results. Furthermore, randomly selected center points are not ideal, and multiple iterations and adjustments may be required to find better clustering results, which will increase computational complexity and time cost, and reduce the efficiency of the algorithm. Summary of the invention

[0005] In order to solve the problem that the Canopy clustering algorithm has great uncertainty when randomly selecting part of the data as the center point for clustering, resulting in the center point being unable to well represent the distribution characteristics of the data and reducing the accuracy of the algorithm, the present invention proposes a method for processing massive data based on a data warehouse base, which includes the following steps: Collect several data in the data warehouse base; obtain local data of each data; obtain local density of each data according to the local data of each data; Obtain the adjacent extreme value points of each data; obtain the degree of outlier of each data according to the adjacent extreme value points of each data; The local density and outlier degree of each data are used to form the characteristic vector of each data, and the modulus of each characteristic vector is obtained as the characteristic value of each data; the center point is obtained according to the characteristic value of each data; all data are clustered according to the center point to obtain noise clusters; the noise clusters are eliminated to obtain denoised data.

[0006] The innovation of the present invention lies in obtaining the eigenvalue of each data according to the local density and outlier degree of the data, and then selecting the center point for clustering according to the eigenvalue of each data. This can adapt to data sets with different distribution characteristics and scales, and the carefully selected center point can converge to a stable state faster during the clustering process, reducing the number of iterations and the amount of calculation, and improving the clustering results.

[0007] Preferably, the obtaining of local data of each data includes: The number of neighborhood data F is preset, and the F data before the collection time corresponding to the i-th data and the F data after the collection time are used as the local data of the i-th data.

[0008] Preferably, obtaining the local density of each data according to the local data of each data includes: ; In the formula, Represents the local density of the i-th data; Represents the maximum value of the local data of the i-th data; Represents the minimum value of the local data of the i-th data; Represents the number of local data of the i-th data; Represents the distance between the i-th data and the j-th local data of the i-th data.

[0009] The greater the local density of the data, the more likely the data is the center point.

[0010] Preferably, obtaining adjacent extreme value points of each data includes: Use the peak detection algorithm to obtain the extreme value points in the local data of the i-th data, and record the two extreme value points adjacent to the i-th data in the local data of the i-th data as adjacent extreme value points of the i-th data.

[0011] Preferably, obtaining the degree of outlier of each data according to the adjacent extreme value points of each data includes: ; In the formula, Represents the degree of outlier of the i-th data; Represents the value of the i-th data; Represents the value of the first adjacent extreme point of the i-th data; Represents the value of the second adjacent extreme point of the i-th data; || represents the absolute value sign.

[0012] The greater the degree of outliers in the data, the more likely it is that the data is the center point.

[0013] Preferably, obtaining the center point according to the characteristic value of each data includes: A characteristic value threshold T2 is preset. If the characteristic value of any data is greater than the characteristic value threshold T2, the data is taken as the center point to obtain a plurality of center points.

[0014] The obtained center point can better reflect the distribution of the data.

[0015] Preferably, clustering all data according to the center point to obtain a noise cluster includes: All collected data are clustered using the Canopy clustering algorithm based on several center points, and the cluster with the least number of data in the clustering result is marked as a noise cluster.

[0016] The resulting noise clusters are more accurate.

[0017] Preferably, the collecting of several data in the data warehouse base includes: In the data warehouse base, by setting up the ETL process and collecting the voltage data of the factory equipment in the order of collection time, the voltage data of the factory equipment in the data warehouse base is obtained; a two-dimensional space is constructed with the timestamp as the X-axis and the voltage data as the Y-axis; the voltage data of the factory equipment is input into the two-dimensional space to obtain a number of data.

[0018] The present invention has the following beneficial effects: the purpose of the present invention is to obtain the characteristic value of each data according to the local density and outlier degree of the data, and then select the center point for clustering according to the characteristic value of each data, so that it can adapt to data sets with different distribution characteristics and scales, and the carefully selected center point can converge to a stable state faster during the clustering process, reduce the number of iterations and the amount of calculation, and improve the clustering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 It is a flowchart of the steps of a method for processing massive data based on a data warehouse base according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0021] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0022] See also Figure 1 , which shows a flowchart of a method for processing massive data based on a data warehouse base provided by an embodiment of the present invention, the method comprising the following steps: S001. Collect data from the data warehouse base.

[0023] In an embodiment of the present invention, in a data warehouse base, an ETL process is set up, and the voltage data of the factory equipment is collected in the order of collection time to obtain the voltage data of the factory equipment in the data warehouse base; a two-dimensional space is constructed with the timestamp as the X-axis and the voltage data as the Y-axis; the voltage data of the factory equipment is input into the two-dimensional space to obtain a number of data.

[0024] S002. Obtain local data of each data, and obtain local density of each data according to the local data of each data.

[0025] It should be noted that the Canopy clustering algorithm, as an efficient clustering method, has a wide range of applications in noise removal in massive data processing. It can quickly cluster a large amount of data through a rough first round of clustering and a more refined second round of clustering, thereby separating the noise data. Although the Canopy clustering algorithm can quickly cluster data, thereby separating the noise data by clustering them into a single category, it will randomly select part of the data as the center point for clustering during use. Randomly selecting the center point has great uncertainty, which may cause the center point to fail to represent the distribution characteristics of the data well, especially in the case of massive data. When the data is unevenly distributed due to noise, randomly selected center points may lead to unstable or inaccurate clustering results. In addition, the randomly selected center points are not ideal, and multiple iterations and adjustments may be required to find a better clustering result, which will increase the computational complexity and time cost and reduce the efficiency of the algorithm. Therefore, this paper proposes a massive data processing method based on a data warehouse base. According to the distribution characteristics and trends of the data, the eigenvalues ​​of each data are obtained, and then part of the data is selected as the center point according to the size of the eigenvalue of the data, so as to reduce the number of algorithm iterations, improve efficiency, ensure that the clustering results are closer to the real structure of the data, and improve the accuracy of clustering.

[0026] It should be further explained that local density reflects the density of data and its local data. In the clustering process, data with higher local density may be more suitable as center points because they can better represent the characteristics of the area where they are located. Therefore, the local density of each data should be considered in the process of clustering data. When the mean distance from any data to all its local data is smaller and the difference between the maximum and minimum values ​​in the local data of the data is smaller, it means that the local density of the data is larger.

[0027] In the embodiment of the present invention, the number of neighborhood data F is preset, and the F data before the collection time corresponding to the i-th data and the F data after the collection time are used as the local data of the i-th data; in the embodiment of the present invention, the number of neighborhood data F is preset to be 50, and in other embodiments, the implementer may preset the value of the number of neighborhood data F according to the specific implementation method; Get the distance between the i-th data and each local data of the i-th data, and take the average of the distances from the i-th data to all local data of the i-th data as the local discreteness of the i-th data; get the difference between the maximum value and the minimum value in the local data of the i-th data, record it as the first difference, and take the inverse of the product of the first difference and the local discreteness of the i-th data as the local density of the i-th data.

[0028] The specific method for obtaining the local density value of each data is: ; In the formula, Represents the local density of the i-th data; Represents the maximum value of the local data of the i-th data; Represents the minimum value of the local data of the i-th data; Represents the number of local data of the i-th data; Represents the distance between the i-th data and the j-th local data of the i-th data; when the mean distance from the i-th data to all its local data is smaller and the difference between the maximum and minimum values ​​in the local data of the i-th data is smaller, it means that the local density of the i-th data is larger.

[0029] S003. Obtain extreme value points in the local data of each data, obtain the degree of outlier of each data according to the extreme value points, and obtain the characteristic value of each data according to the local density and the degree of outlier of each data.

[0030] It should be noted that the degree of outliers reflects the degree of difference between data and its local data. Data with a higher degree of outliers may belong to noise or outliers, while data with a lower degree of outliers are more likely to belong to normal data distribution. In the clustering process, selecting data with a lower degree of outliers as the center point is more conducive to clustering normal data into one category. Since the degree of outliers is a measure of whether a data deviates from other data, when processing conventional data, the degree of outliers of the data can be described based on the degree of difference between the data and its adjacent data; however, when processing massive data in the data warehouse base, the difference between the data and the adjacent data can often only reflect the fluctuation of the local data, and cannot effectively distinguish outliers. If the adjacent data is replaced by the adjacent extreme points, it will help to distinguish outliers in massive data, because the extreme points are the local maximum or minimum points in the data distribution. Compared with the adjacent data, they expand the calculation range of the outlier degree of the data to a certain extent, which can better reflect the degree of difference between the data and its local data.

[0031] In the embodiment of the present invention, a peak detection algorithm is used to obtain an extreme value point in the local data of the i-th data, and two extreme value points adjacent to the i-th data in the local data of the i-th data are recorded as adjacent extreme value points of the i-th data; Get the degree of outlier for each data: ; In the formula, Represents the degree of outlier of the i-th data; Represents the value of the i-th data; Represents the value of the first adjacent extreme point of the i-th data; Represents the value of the second adjacent extreme point of the i-th data; Represents the absolute value of the difference between two adjacent extreme points of the i-th data; Represents the absolute value of the difference between the i-th data and its first adjacent extreme point; The absolute value of the difference between the i-th data point and the second adjacent extreme point; It represents the prominence of the sum of the absolute values ​​of the difference between the i-th data and its two adjacent extreme points relative to the absolute value of the difference between the two adjacent extreme points. The greater the prominence, the greater the degree of outlier of the i-th data.

[0032] It should be noted that when selecting the center point of the clustering process, the local density and outlier degree of the data are considered as the characteristic values ​​of the data, which includes both the density information around the data and the degree of difference between the data and its local data, which helps to stabilize the center point and eliminate noise.

[0033] In the embodiment of the present invention, the local density and outlier degree of each data are used to form a feature vector of each data, and the modulus of each feature vector is obtained as the feature value of each data.

[0034] S004. Obtain a center point according to the characteristic value of each data; cluster all data according to the center point to obtain a noise cluster; remove the noise cluster to obtain denoised data.

[0035] It should be noted that the eigenvalue of each data has been obtained, and data with larger eigenvalues ​​are more likely to be clustered as center points. Therefore, according to the eigenvalue of each data, the center point is obtained and all the data are clustered.

[0036] In an embodiment of the present invention, a characteristic value threshold T2 is preset. If the characteristic value of any data is greater than the characteristic value threshold, the data is used as the center point to obtain a number of center points. All the collected data are clustered using the Canopy clustering algorithm based on the several center points. The cluster with the least number of data contained in the clustering result is marked as a noise cluster, which is eliminated to obtain denoised data, thereby realizing denoising of massive data in the data warehouse base, and then analyzing the denoised data, such as anomaly detection, visualization, correlation analysis, etc., and making some decisions based on the analysis results.

[0037] In the embodiment of the present invention, the preset characteristic value threshold T2=0.68. In other embodiments, implementers may preset the value of the characteristic value threshold T2 according to the specific implementation method.

[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing massive data based on a data warehouse base, characterized in that: include: Collect several data in the data warehouse base; obtain local data of each data; According to the local data of each data, the local density of each data is obtained; Obtain the adjacent extreme value points of each data; obtain the degree of outlier of each data according to the adjacent extreme value points of each data; The local density and outlier degree of each data are used to form the feature vector of each data, and the modulus of each feature vector is obtained as the feature value of each data; the center point is obtained according to the feature value of each data; all data are clustered according to the center point to obtain the noise cluster; The noise clusters are removed to obtain denoised data.

2. A method for processing massive data based on a data warehouse base according to claim 1, characterized in that: The obtaining of local data of each data includes: The number of neighborhood data F is preset, and the F data before the collection time corresponding to the i-th data and the F data after the collection time are used as the local data of the i-th data.

3. The method for processing massive data based on a data warehouse base according to claim 1, characterized in that: The step of obtaining the local density of each data according to the local data of each data includes: ； In the formula, Represents the local density of the i-th data; Represents the maximum value of the local data of the i-th data; Represents the minimum value of the local data of the i-th data; Represents the number of local data of the i-th data; Represents the distance between the i-th data and the j-th local data of the i-th data.

4. The method for processing massive data based on a data warehouse base according to claim 1, characterized in that: The step of obtaining adjacent extreme value points of each data includes: Use the peak detection algorithm to obtain the extreme value points in the local data of the i-th data, and record the two extreme value points adjacent to the i-th data in the local data of the i-th data as adjacent extreme value points of the i-th data.

5. The method for processing massive data based on a data warehouse base according to claim 1, characterized in that: The step of obtaining the degree of outlier of each data according to the adjacent extreme value points of each data includes: ； In the formula, Represents the degree of outlier of the i-th data; Represents the value of the i-th data; Represents the value of the first adjacent extreme point of the i-th data; Represents the value of the second adjacent extreme point of the i-th data; || represents the absolute value sign.

6. The method for processing massive data based on a data warehouse base according to claim 1, characterized in that: The step of obtaining the center point according to the characteristic value of each data includes: A characteristic value threshold T2 is preset. If the characteristic value of any data is greater than the characteristic value threshold T2, the data is taken as the center point to obtain a plurality of center points.

7. The method for processing massive data based on a data warehouse base according to claim 1, characterized in that: The noise clusters are obtained by clustering all the data according to the center point, including: All collected data are clustered using the Canopy clustering algorithm based on several center points, and the cluster with the least number of data in the clustering result is marked as a noise cluster.

8. The method for processing massive data based on a data warehouse base according to claim 1, characterized in that: The data collected in the data warehouse base includes: In the data warehouse base, by setting up the ETL process and collecting the voltage data of the factory equipment in the order of collection time, the voltage data of the factory equipment in the data warehouse base is obtained; a two-dimensional space is constructed with the timestamp as the X-axis and the voltage data as the Y-axis; the voltage data of the factory equipment is input into the two-dimensional space to obtain a number of data.

Images