Multi-element big data optimization storage method

Abstract

The invention relates to the field of data processing, in particular to a multivariate big data optimization storage method, which comprises the following steps of: obtaining a plurality of multivariate data points; obtaining a comparison point, and calculating a correlation description value of each dimension of the comparison point and other multivariate data points; performing clustering processing on all the multivariate data points according to the correlation description values; acquiring a first principal component direction, and acquiring a reference data point according to the first principal component direction; obtaining a unit vector and a modulus length of a description vector of each multivariate data point according to the reference data point; and taking the unit vectors as coding objects, performing Huffman coding on the unit vectors of all the multivariate data points in the category, and storing codes and die lengths corresponding to the unit vectors of the multivariate data points. The data compression amount is improved by selecting proper coding features.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically, to a multivariate big data optimization storage method. Background Art

[0002] With the development of science and technology, a large amount of data is generated every day, such as production line data. In order to facilitate production line management, it is generally necessary to collect multiple production line data in real time. In the long run, a large amount of multiple production line data will be collected, and these collected multiple production line data need to be saved for a period of time, so that when problems occur in subsequent products, the causes can be quickly analyzed. It takes a lot of storage costs to store these data. In order to save storage costs, the collected data needs to be compressed.

[0003] As a lossless compression algorithm, Huffman coding algorithm is often used for data compression processing. Huffman coding algorithm encodes data according to the frequency of occurrence, wherein the higher the frequency of occurrence, the shorter the code length set for it, and the lower the frequency of occurrence, the longer the code length set for it. In reality, the multivariate production line data at different times are not the same, so the frequency of occurrence of each multivariate production line data is low, and the low frequency of data occurrence will result in a longer code length, and the longer code length will result in poor data compression effect. Therefore, how to achieve a better compression effect when the data frequency of occurrence is low becomes the research focus of this invention.

[0004] The patent document with the authorization announcement number CN104283568B shows a data compression coding method based on a partial Huffman tree. The patent document mainly improves the compression efficiency by constructing a coding table in layers. The coding method in the patent document cannot solve the problem of poor data compression effect caused by low frequency of occurrence, so the patent document cannot solve the problem in the present invention well. Summary of the invention

[0005] In order to solve the problem of how to achieve a better compression effect when the data frequency is low, the present invention proposes a multivariate big data optimization storage method, which includes the following steps: Obtain several multivariate data points; The multivariate data point closest to the geometric center of all multivariate data points is taken as the comparison point, and the correlation between each dimension of the comparison point and other dimensions of each multivariate data point is calculated. The mean of the correlation between each dimension of the comparison point and all other dimensions of each multivariate data point is taken as the correlation description value between each dimension of the comparison point and other multivariate data points; all multivariate data points are clustered according to the correlation description value; The PCA algorithm is used to process the multivariate data points of each category, and the principal component direction with the largest eigenvalue is obtained and recorded as the first principal component direction. The outermost multivariate data point on the straight line located by the geometric center of all multivariate data points of each category and the first principal component direction is obtained and recorded as the reference data point; the vector formed by each multivariate data point in the category and the reference data point is obtained as the description vector of each multivariate data point, and the unit vector and modulus of the description vector of each multivariate data point are obtained; The unit vector is taken as the encoding object, Huffman encoding is performed on the unit vectors of all multivariate data points in the category, and the encoding and modulus length corresponding to the unit vector of the multivariate data point are stored.

[0006] The present invention takes into account that multivariate data points have certain distribution directions, and thus the number of multivariate data points with the same distribution direction is relatively large, so that the distribution direction of multivariate data points is used as the encoding object to effectively improve the compression amount; further, the distribution direction of the multivariate data points is reflected by the unit vector of the multivariate data points and the reference data points, which not only has a high description accuracy, but also introduces less irrelevant data, effectively improving the compression amount; further, in order to make the multivariate data points have stronger distribution direction characteristics, the multivariate data points are clustered according to the distribution relationship of the multivariate data points, so that the multivariate data points with the same distribution direction are divided into one category, providing a data basis for improving the compression amount; further, the distribution relationship of the multivariate data points can be more accurately reflected through the correlation description value, providing a basis for accurate clustering. Further, when constructing the correlation description value, considering that the correlation between data of different dimensions can effectively reflect the distribution relationship of the multivariate data, the correlation description value is constructed by using the correlation relationship between data of different dimensions.

[0007] Preferably, the calculation of the correlation between each dimension of the comparison point and other dimensions of other multivariate data points includes: Take any dimension as the target dimension, record the vector formed by the data of the comparison data point in the target dimension and the data of other multivariate data points in the target dimension as the analysis vector of the target dimension, and take the absolute value of the Pearson correlation coefficient between the analysis vector of the target dimension and the analysis vector of other dimensions as the correlation between the comparison point and the target dimension of other multivariate data points and other dimensions; Get the correlation between each dimension of the comparison point and other multivariate data points.

[0008] The present invention reflects the correlation of data of different dimensions through the absolute value of the Pearson correlation coefficient, and its implementation efficiency is higher.

[0009] Preferably, clustering all multivariate data points according to the correlation description values ​​includes: The vector composed of the correlation description values ​​of all dimensions between the comparison point and other multivariate data points is used as the classification basis of other multivariate data points, and the other multivariate data points are clustered to obtain several categories, and the category closest to the comparison data point is taken as the category to which the comparison data point belongs.

[0010] The present invention uses the correlation description value as a classification basis, which can effectively segment data with the same classification feature together, providing a data basis for subsequent effective data compression.

[0011] Preferably, obtaining the outermost multivariate data point on the straight line located by the geometric center of all multivariate data points of each category and the first principal component direction and recording it as the reference data point includes: Get the geometric center of all multivariate data points in each category, fit a straight line using the geometric center and the direction of the first principal component, and get any outermost multivariate data point on the fitted straight line as the reference data point.

[0012] Preferably, the step of obtaining a vector formed by each multivariate data point and a reference data point in a category as a description vector of each multivariate data point includes: The vector obtained by subtracting each multivariate data point within the category from the benchmark data point is used as the description vector of each multivariate data.

[0013] Preferably, the step of obtaining the unit vector and the modulus of the description vector of each multivariate data point includes: Get the modulus of the description vector of the multivariate data point, and divide the description vector of the multivariate data point by the modulus to obtain a unit vector.

[0014] The present invention takes into account that when multivariate data points are distributed in one direction, the degree to which the description vector deviates from the distribution direction is small, and thus the unit vector of the description vector of the multivariate data points has less diversity, and less diversity will result in a shorter encoding length of the Huffman tree; further, considering that when multivariate data points are distributed in one direction, the degree to which the unit vector of the description vector deviates from the distribution direction is small, and more multivariate data points are distributed on the unit vector of the description vector, and thus when the unit vector of the description vector is used as the encoding object, the compression amount is greater.

[0015] Preferably, the step of taking the unit vector as the encoding object and performing Huffman encoding on the unit vectors of all multiple data points in the category includes: The unit vectors of the multivariate data points in the category are statistically analyzed to obtain the frequency of occurrence of the unit vectors. According to the frequency of occurrence of the unit vectors, a Huffman coding algorithm is used to construct a Huffman tree, and the encoding of the unit vector is obtained according to the Huffman tree.

[0016] Preferably, storing the codes and moduli corresponding to the unit vectors of the multivariate data points includes: The codes and moduli corresponding to the unit vectors of the Huffman tree, the benchmark data points, and the multivariate data points are stored.

[0017] The present invention stores the codes and module lengths corresponding to the unit vectors of the Huffman tree, the reference data points, and the multivariate data points, which can save storage space and provide a data basis for data decompression.

[0018] The present invention has the following beneficial effects: The present invention takes into account that multivariate data points have certain distribution directions, and thus the number of multivariate data points with the same distribution direction is relatively large, so that the compression amount can be effectively improved by using the distribution direction of the multivariate data points as the encoding object; Furthermore, the distribution direction of the multivariate data points is reflected by the unit vectors of the multivariate data points and the reference data points, which not only has a higher description accuracy, but also introduces less irrelevant data, effectively improving the compression amount; Furthermore, in order to make the multivariate data points have stronger distribution direction characteristics, the multivariate data points are clustered according to their distribution relationship, so that the multivariate data points with the same distribution direction are divided into one category, providing a data basis for improving the compression amount; Furthermore, the correlation description value can more accurately reflect the distribution relationship of multivariate data points and provide a basis for accurate clustering.

[0019] Furthermore, when constructing the correlation description value, it is considered that the correlation between data of different dimensions can effectively reflect the distribution relationship of multivariate data, and thus the correlation description value is constructed by using the correlation relationship between data of different dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 This is a flowchart of the steps of a multivariate big data optimization storage method according to an embodiment of the present invention; Figure 2 A schematic diagram of a first distribution of multivariate data points provided by an embodiment of the present invention; Figure 3 A schematic diagram of a second distribution of multivariate data points provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] See also Figure 1 , which shows a flowchart of a multivariate big data optimization storage method provided by an embodiment of the present invention, the method comprising the following steps: S1: Obtain several multivariate data points.

[0024] Specifically, a plurality of multivariate data points are obtained, for example, the multivariate data points are data points composed of a plurality of types of production line data at a moment.

[0025] S2: The multivariate data point closest to the geometric center of all multivariate data points is taken as the comparison point, and the correlation between each dimension of the comparison point and other dimensions of each multivariate data point is calculated. The mean of the correlation between each dimension of the comparison point and all other dimensions of each multivariate data point is taken as the correlation description value between the comparison point and each dimension of each multivariate data point; all multivariate data points are clustered according to the correlation description value.

[0026] It should be noted that generally, there is a certain correlation between the dimensions of multivariate data points. For example, there is a correlation between the equipment operation status of the production line and the production capacity of the production line. The equipment does not operate well or even fails, which causes the production line to be unable to produce, and then the production capacity is reduced. Due to the correlation between the dimensions of the multivariate data points, there is a certain distribution pattern between the multivariate data points. For example, the multivariate data points are distributed along a certain direction. If the multivariate data points are distributed along a certain direction, there are more data points distributed in this direction. This feature meets the compression requirements of Huffman coding. In other words, if the data points are distributed along a certain direction, the frequency of occurrence of the data points in a certain direction is large, and the high frequency of occurrence will lead to a reduction in the coding length. Therefore, the data can be encoded according to this feature.

[0027] It should be further explained that a large number of multivariate data points are not distributed in only one direction, such as Figure 2 As shown in the first distribution diagram of the multi-data points, the data points can be classified according to the distribution characteristics of the data, and the data points along one direction can be segmented together, providing a basis for subsequently improving the data compression amount.

[0028] S20: The multivariate data point closest to the geometric center of all multivariate data points is taken as the comparison point, and the correlation between the comparison point and each dimension of other multivariate data points and other dimensions is calculated, and the average of the correlations between each dimension of the comparison point and other multivariate data points and all other dimensions is taken as the correlation description value between the comparison point and each dimension of other multivariate data points.

[0029] It should be noted that the distribution characteristics between multivariate data points can be reflected by the dimensional correlation of the multivariate data points, and thus the data can be clustered according to the dimensional correlation of the multivariate data points. In this embodiment, the dimensional correlation of the multivariate data points is reflected by the correlation description value.

[0030] Preferably, as an example, the multivariate data point closest to the geometric center of all multivariate data points is used as the comparison point, the correlation between each dimension of the comparison point and other dimensions of each other multivariate data point is calculated, and the average of the correlations between each dimension of the comparison point and all other dimensions of each other multivariate data point is used as the correlation description value between the comparison point and each dimension of each other multivariate data point, including: The multivariate data point closest to the geometric center of all multivariate data points is used as the comparison point.

[0031] Take any dimension as the target dimension, record the vector formed by the data of the comparison data point in the target dimension and the data of other multivariate data points in the target dimension as the analysis vector of the target dimension, and take the absolute value of the Pearson correlation coefficient of the analysis vector of the target dimension and the analysis vector of other dimensions as the correlation between the comparison point and the target dimension and other dimensions of other multivariate data points. In the same way, obtain the correlation between each dimension of the comparison point and other multivariate data points and other dimensions.

[0032] The mean value of the correlation between the comparison point and each dimension of other multivariate data points and all other dimensions is taken as the description value of the correlation between the comparison point and each dimension of other multivariate data points.

[0033] It can be understood that the correlation description value of each dimension between the comparison point and other multivariate data points reflects the correlation characteristics of each dimension with other dimensions calculated based on the comparison point and other multivariate data points. This value can reflect the relative distribution relationship between data points.

[0034] S21: Cluster all multivariate data points according to the correlation description value.

[0035] It should be noted that if the correlation characteristics of the comparison point and other multivariate data points A in each dimension are similar to the correlation characteristics of the comparison point and other multivariate data points B in all dimensions, it is highly likely that other multivariate data points A and other multivariate data points B have the same distribution characteristics. Therefore, the multivariate data points can be clustered based on this.

[0036] Preferably, as an example, clustering is performed on all multivariate data points according to the correlation description value, including: The vector composed of the correlation description values ​​of all dimensions between the comparison point and other multivariate data points is used as the classification basis of other multivariate data points, and the DBSCAN clustering algorithm is used to cluster other multivariate data points to obtain several categories.

[0037] In particular, the category closest to the comparison data point is taken as the category to which the comparison data point belongs.

[0038] It can be understood that the vector composed of the correlation description values ​​of all dimensions of each other multivariate data point and the comparison point can reflect the relative position relationship between each other multivariate data point and the comparison point. The more similar the relative position relationship is, the more likely it is that the other multivariate data points have the same distribution characteristics, and thus they are clustered into one category.

[0039] S3: Use the PCA algorithm to process the multivariate data points of each category, obtain the principal component direction with the largest eigenvalue and record it as the first principal component direction, obtain the outermost multivariate data point on the straight line located by the geometric center of all multivariate data points of each category and the first principal component direction and record it as the reference data point; obtain the vector formed by each multivariate data point in the category and the reference data point as the description vector of each multivariate data point, and obtain the unit vector and modulus of the description vector of each multivariate data point.

[0040] It should be noted that if Figure 3 As shown in the second distribution diagram of the multivariate data points, when the multivariate data points are distributed along one direction, there are more data points distributed in this direction, the frequency of occurrence of data points in this distribution direction is higher, and the encoding of the distribution direction is shorter. Therefore, the distribution direction of the data points can be used as the encoding object to increase the compression amount.

[0041] Preferably, as an example, the PCA algorithm is used to process the multivariate data points of each category, and the principal component direction with the largest eigenvalue is obtained and recorded as the first principal component direction. The outermost multivariate data points on the straight line located by the geometric center of all multivariate data points of each category and the first principal component direction are obtained and recorded as reference data points; the vectors formed by each multivariate data point in the category and the reference data point are obtained as the description vectors of each multivariate data point, and the unit vector and the modulus of the description vector of each multivariate data point are obtained, including: The PCA algorithm is used to process the multivariate data points of each category, and the principal component direction with the largest eigenvalue is recorded as the first principal component direction.

[0042] Get the geometric center of all multivariate data points in each category, fit a straight line using the geometric center and the direction of the first principal component, and get any outermost multivariate data point on the fitted straight line as the reference data point.

[0043] The vector obtained by subtracting each multivariate data point within the category from the benchmark data point is used as the description vector of each multivariate data.

[0044] Get the modulus of the description vector of the multivariate data point, and divide the description vector of the multivariate data point by the modulus to obtain a unit vector.

[0045] It can be understood that the unit vector of each multivariate data point reflects the distribution direction of each multivariate data point. If the multivariate data points are distributed along a certain direction, there are many multivariate data points with the same unit vector. The number of multivariate data points with the same unit vector in a category is large, and its encoding length is short. Therefore, the unit vector of the multivariate data point is used as the encoding object, which can effectively save storage.

[0046] It should be noted that using the PCA algorithm to process the multivariate data points of each category to obtain the principal component direction with the largest eigenvalue is an existing technology and will not be described in detail here.

[0047] S4: Take the unit vector as the encoding object and perform Huffman encoding on the unit vectors of all multivariate data points in the category; store the encoding and modulus length corresponding to the unit vector of the multivariate data point.

[0048] S40: Take the unit vector as the encoding object and perform Huffman encoding on the unit vectors of all multiple data points in the category.

[0049] Preferably, as an example, a unit vector is used as a coding object, and Huffman coding is performed on the unit vectors of all multiple data points in the category, including: The unit vectors of the multivariate data points in the category are statistically analyzed to obtain the frequency of occurrence of the unit vectors. According to the frequency of occurrence of the unit vectors, a Huffman tree is constructed using the Huffman coding algorithm, and the unit vectors are encoded according to the Huffman tree.

[0050] S41: Store the codes and moduli corresponding to the unit vectors of the multivariate data points.

[0051] Preferably, as an example, the codes and moduli corresponding to the unit vectors of the multivariate data points are stored, including: The codes and moduli corresponding to the unit vectors of the Huffman tree, the benchmark data points, and the multivariate data points are stored.

[0052] S5: Decompress the data.

[0053] Preferably, as an example, decompressing the data includes: According to the code corresponding to the unit vector of the multivariate data point, combined with the corresponding relationship between the code and the unit vector in the Huffman tree, the unit vector of the multivariate data point is obtained. The unit vector of the multivariate data point is multiplied by the modulus to obtain the description vector of each multivariate data point. The description vector of each multivariate data point is added to the reference data point to obtain each multivariate data point.

[0054] It can be seen that the compressed data can be effectively decoded losslessly through the stored data. Therefore, this compression method not only does not cause data loss, but also effectively increases the data compression amount.

[0055] At this point, this embodiment is completed.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multivariate big data optimization storage method, characterized in that: include: Obtain several multivariate data points; The multivariate data point closest to the geometric center of all multivariate data points is taken as the comparison point, and the correlation between each dimension of the comparison point and other dimensions of each multivariate data point is calculated. The mean of the correlation between each dimension of the comparison point and all other dimensions of each multivariate data point is taken as the correlation description value between each dimension of the comparison point and other multivariate data points; all multivariate data points are clustered according to the correlation description value; The PCA algorithm is used to process the multivariate data points of each category, and the principal component direction with the largest eigenvalue is obtained and recorded as the first principal component direction. The outermost multivariate data point on the straight line located by the geometric center of all multivariate data points of each category and the first principal component direction is obtained and recorded as the reference data point; Obtaining a vector formed by each multivariate data point and a reference data point in the category as a description vector of each multivariate data point, and obtaining a unit vector and a modulus length of the description vector of each multivariate data point; The unit vector is taken as the encoding object, Huffman encoding is performed on the unit vectors of all multivariate data points in the category, and the encoding and modulus length corresponding to the unit vector of the multivariate data point are stored.

2. The method for optimizing storage of multivariate big data according to claim 1, characterized in that: The calculation of the correlation between each dimension of the comparison point and other dimensions of each multivariate data point includes: Take any dimension as the target dimension, record the vector formed by the data of the comparison data point in the target dimension and the data of other multivariate data points in the target dimension as the analysis vector of the target dimension, and take the absolute value of the Pearson correlation coefficient between the analysis vector of the target dimension and the analysis vector of other dimensions as the correlation between the comparison point and the target dimension of other multivariate data points and other dimensions; Get the correlation between each dimension of the comparison point and other multivariate data points.

3. The method for optimizing storage of multivariate big data according to claim 1, characterized in that: The clustering process of all multivariate data points according to the correlation description value includes: The vector composed of the correlation description values ​​of all dimensions between the comparison point and other multivariate data points is used as the classification basis of other multivariate data points, and the other multivariate data points are clustered to obtain several categories, and the category closest to the comparison data point is taken as the category to which the comparison data point belongs.

4. The method for optimizing storage of multivariate big data according to claim 1, characterized in that: The outermost multivariate data points are obtained on the straight line located by the geometric center of all multivariate data points of each category and the first principal component direction and recorded as reference data points, including: Get the geometric center of all multivariate data points in each category, fit a straight line using the geometric center and the direction of the first principal component, and get any outermost multivariate data point on the fitted straight line as the reference data point.

5. The method for optimizing storage of multivariate big data according to claim 1, characterized in that: The step of obtaining a vector formed by each multivariate data point and a reference data point in a category as a description vector of each multivariate data point includes: The vector obtained by subtracting each multivariate data point from the benchmark data point within the category is used as the description vector of each multivariate data.

6. The method for optimizing storage of multivariate big data according to claim 1, characterized in that: The step of obtaining the unit vector and the modulus of the description vector of each multivariate data point includes: Get the modulus of the description vector of the multivariate data point, and divide the description vector of the multivariate data point by the modulus to obtain a unit vector.

7. The method for optimizing storage of multivariate big data according to claim 1, characterized in that: The method uses the unit vector as the encoding object and performs Huffman encoding on the unit vectors of all multiple data points in the category, including: The unit vectors of the multivariate data points in the category are statistically analyzed to obtain the frequency of occurrence of the unit vectors. According to the frequency of occurrence of the unit vectors, a Huffman coding algorithm is used to construct a Huffman tree, and the encoding of the unit vector is obtained according to the Huffman tree.

8. The method for optimizing storage of multivariate big data according to claim 7, characterized in that: The storing of the codes and modulus lengths corresponding to the unit vectors of the multivariate data points includes: The codes and moduli corresponding to the unit vectors of the Huffman tree, the benchmark data points, and the multivariate data points are stored.

Images