A data preprocessing method and system suitable for big data analysis

Abstract

The present invention discloses a data preprocessing method and system suitable for big data analysis, and relates to the field of data preprocessing technology. The method includes collecting raw data through big data; using a dynamic correlation supplementation method to calculate supplementary values, supplementing the missing characteristic values ​​of the data, and supplementing the missing characteristic values ​​is more consistent with the overall trend of the data, thereby improving the accuracy of the supplementary values; obtaining reliability constants from different sources through a reliability constant calculation method, and extracting different proportions of data from different sources; merging, unifying and normalizing the data. The present invention calculates supplementary values ​​through a dynamic correlation supplementation method, supplements the missing characteristic values ​​of the data, and determines the relevant weights of other characteristic values ​​to the missing characteristic values ​​based on the correlation between other characteristic values ​​of the data and the missing characteristic values. The correlation changes dynamically, and the supplementation of the missing characteristic values ​​is more consistent with the overall trend of the data, thereby improving the accuracy of the supplementary values.

Description

Technical Field

[0001] The present invention relates to the technical field of data preprocessing, and in particular to a data preprocessing method and system suitable for big data analysis. Background Art

[0002] Data preprocessing is performed to improve data quality, enhance model performance, reduce algorithm computation and resource consumption, make data more suitable for subsequent analysis and modeling tasks, and obtain more accurate, reliable and efficient results. Data cleaning is used to remove noise, erroneous data and process missing values ​​to improve data accuracy and completeness. Standardization, normalization and other operations are used to eliminate data inconsistencies to ensure data consistency. Feature selection and extraction are used to eliminate irrelevant features and extract key features to reduce data dimensions. Data normalization and other operations are performed to make data more regular, thereby improving data quality. Ultimately, the goals of improving model accuracy, accelerating model convergence, enhancing model generalization capabilities, and reducing training time and costs are achieved, laying a solid foundation for the successful implementation of tasks such as data analysis and machine learning.

[0003] When preprocessing data, it is necessary to delete duplicate parts of the data and supplement missing parts. For example, the method for filling missing values ​​in meteorological data based on the SVD algorithm, patent publication number CN106295175A, includes the following steps: (1) receiving data; (2) preprocessing raw data; (3) selecting a training set and a test set; (4) training parameters; (5) filling missing values; and (6) outputting the filled meteorological station data. The present invention extracts each meteorological attribute separately, composes a data file, and performs SVD algorithm training on each file to obtain the station meteorological data after filling missing values, thereby improving robustness and accuracy of missing value filling.

[0004] The role of supplementary data is to fill in the missing parts of the data, make the data more complete, improve the availability of the data, and enable data analysis and model training to utilize more comprehensive information, thereby improving the accuracy of the analysis results and the performance and generalization ability of the model. When the above method fills the original data, the data with different characteristics are predicted and filled separately, which cannot reflect the relationship between the various characteristics of the data. The accuracy of the filled data is poor. Commonly used filling methods also use mean, median, etc. for filling, which may introduce bias, because this method does not consider the true distribution of the data and the relationship between the characteristics, which may cause the data to lose its original characteristics and rules, resulting in inaccurate subsequent analysis results. Summary of the Invention

[0005] The purpose of the present invention is to provide a data preprocessing method and system suitable for big data analysis to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a data preprocessing method suitable for big data analysis, the method comprising:

[0007] Collect raw data through big data and record the data source;

[0008] After the original data is deduplicated, the dynamic correlation supplementation method is used to calculate the supplementary value to supplement the missing characteristic values ​​of the data;

[0009] The reliability constants of different sources are obtained through the reliability constant calculation method. According to the reliability constants of data from different sources, different proportions of data from different sources are extracted.

[0010] Merge and unify data from different sources and formats, convert data formats, unify encoding, and other operations to keep the data consistent in structure and semantics;

[0011] The integrated data is normalized to make the data have the same scale and distribution.

[0012] Preferably, the dynamic correlation supplementation method includes:

[0013] S1: Calculate the correlation coefficient of each eigenvalue to the missing eigenvalue based on the remaining eigenvalues ​​of the data, according to the formula:

[0014]

[0015] is the number of samples in the acquired data, where is the correlation coefficient between the jth feature and the kth feature, is the eigenvalue of the jth feature of the i-th sample, is the mean of the jth feature in n samples, is the eigenvalue of the kth feature of the i-th sample, is the mean of the kth feature in n samples;

[0016] S2: Determine the correlation weights between other features and missing features based on the correlation coefficient, according to the formula:

[0017]

[0018] in To express the relative weight of the kth feature for filling the missing value of the jth feature, is the number of features for each sample;

[0019] S3: Obtain the fitted linear relationship based on historical data and calculate the relevant supplementary value according to the formula:

[0020]

[0021]

[0022] in is the dependent eigenvalue, is the intercept term, represents the eigenvalue of the kth feature, is the regression coefficient of feature k, is the error term, is the relevant supplementary value of the jth feature in sample i, is the eigenvalue of the kth feature of the i-th sample;

[0023] S4: Use The value of is used to supplement the j-th feature in sample i.

[0024] Preferably, the dynamic correlation supplementation method further comprises, for the autocorrelation part of the time series:

[0025] G1: Select several consecutive complete samples and calculate the autocorrelation coefficient according to the formula:

[0026]

[0027] in Lag The autocorrelation coefficient of the period, To select the number of consecutive data periods, For continuous The average value of the period, is the data value of the tth period in the selected data period, is the data value of the t-δth period in the selected data period;

[0028] G2: Calculate the autocorrelation weight based on the autocorrelation coefficient:

[0029]

[0030] in Lag The autocorrelation weight of the period, is the autocorrelation coefficient of the lag period c, is the maximum lag period set;

[0031] G3: Calculate the final supplementary value based on the autocorrelation weight and the relevant supplementary value:

[0032]

[0033] in is the autocorrelation supplementary value of the j-th feature in sample i, is the final supplementary value of the j-th feature in sample i, For the The eigenvalue of the jth feature of the sample, To set the weight yourself, is the relevant supplementary value of the jth feature in sample i;

[0034] G4: Use The value of is used to supplement the j-th feature in sample i.

[0035] Preferably, the method for obtaining a fitted linear relationship based on historical data includes:

[0036] L1: Collect historical data and set missing feature values ​​as dependent features based on the historical data;

[0037] L2: Setting is the intercept term, represents the eigenvalue of the kth feature, is the regression coefficient of feature k, is the error term, and the formula is determined ;

[0038] L3: Estimate the regression coefficient by the least squares method and determine 、 and The linear relationship of the fit is determined.

[0039] Preferably, the calculation method of the reliability constant includes:

[0040] Get the total number of samples, the number of features in each sample, and the number of missing features in each sample, and remove samples with a missing ratio of more than 0.5 from several samples, and then calculate according to the formula:

[0041]

[0042]

[0043] in is the reliability constant of the data source, is the data defect constant, reflecting the degree of data missing, The initial credibility of the data source is between 0 and 1, which is set by technical personnel. The value indicates the degree of recognition of the authenticity of the data source. is the number of missing features in sample i, is the total number of samples in the acquired data, is the number of features for each sample;

[0044] When the reliability constant of the data source ≤0.3 or ≥0.7, the initial credibility of the data source is updated through the credibility update method. The value of is updated to , improve the accuracy of the next calculation.

[0045] Preferably, the calculation method of the reliability constant includes:

[0046] Get the total number of samples, the number of features in each sample, and the number of missing features in each sample, and calculate according to the formula:

[0047]

[0048]

[0049] in is the reliability constant of the data source, is the data defect constant, reflecting the degree of data missing, The credibility of the data source is between 0 and 1, and its value indicates the degree of recognition of the authenticity of the data source. is the number of samples missing feature j, is the total number of samples in the acquired data, is the number of features for each sample;

[0050] When the reliability constant of the data source ≤0.3 or When ≥0.7, the credibility of the data source is updated by the credibility update method The value of is updated to improve the accuracy of the next calculation.

[0051] Preferably, the credibility updating method includes:

[0052] When 0.2< When ≤0.3, ; When 0.1< When ≤0.2, ;when When ≤0.1, , and prompt technical personnel to redefine the data source;

[0053] When 0.7≤ When ≤0.8, ; When 0.8< When ≤0.9, ;when >0.9, ;

[0054] in To ensure the credibility of the updated data source, is the credibility of the data source before the update, and ,in is the initial credibility, set by technical personnel, It is the update speed constant, set by the technicians. The larger the value, the greater the update amplitude.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The supplementary value is calculated through the dynamic correlation supplementation method to supplement the missing eigenvalues ​​of the data. According to the correlation between other eigenvalues ​​of the data and the missing eigenvalues, the relevant weights of other eigenvalues ​​to the missing eigenvalues ​​are determined. According to the different acquired data, the correlation changes dynamically. The supplementation of missing eigenvalues ​​is more in line with the overall trend of the data, thereby improving the accuracy of the supplementary value.

[0057] At the same time, according to the changing trend of the data in different time series, the autocorrelation coefficient of the missing feature is determined, and the autocorrelation weight of the eigenvalue of the previous period of the missing feature to the eigenvalue of the missing feature is calculated. The changing trend of the missing feature is fully considered, and the final supplementary value is determined by combining the influence of other eigenvalues ​​on the missing feature, which further improves the accuracy of the supplementary value.

[0058] In addition, according to the different data sources, the reliability constant of each data source can be calculated separately to determine the credibility of the data source. The data can be screened using the reliability constant of the data source as the screening ratio to reduce the number of samples and improve the calculation speed. Moreover, the credibility of the data source can be iterated through the credibility update method, and the recognition degree of the data source can be continuously optimized in long-term use, thereby further improving the accuracy of the screened samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of the data preprocessing method of the present invention;

[0060] Figure 2 Schematic diagram of the flow of the dynamic correlation supplementation method of the present invention;

[0061] Figure 3 Schematic diagram of the process of calculating the reliability constant (sample calculation) of the present invention;

[0062] Figure 4 Schematic diagram of the process of calculating the reliability constant (calculation by characteristics) of the present invention;

[0063] Figure 5 Schematic diagram of the process of the credibility updating method of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] In this application, for ease of understanding, the method steps used do not need to be executed in the order of the steps in this embodiment during actual operation. In other embodiments, these steps may be performed simultaneously or in a different order.

[0066] Example 1

[0067] The role of supplementary data is to fill in the missing parts of the data, make the data more complete, improve the availability of the data, and enable data analysis and model training to utilize more comprehensive information, thereby improving the accuracy of the analysis results and the performance and generalization ability of the model. When filling in data, when the features to be filled have a strong correlation with the remaining features, the missing features need to be calculated based on the remaining features to improve the accuracy of the supplement.

[0068] like Figure 1 As shown, the present invention provides a technical solution: a data preprocessing method suitable for big data analysis, comprising:

[0069] Collect raw data through big data and record the data source;

[0070] It should be noted that raw data can be collected through various service platforms, social platforms, network platforms and third-party providers, and the data source needs to be manually confirmed by technical personnel to have a certain degree of credibility.

[0071] After the original data is deduplicated, the dynamic correlation supplementation method is used to calculate the supplementary value to supplement the missing characteristic values ​​of the data;

[0072] Merge and unify data from different sources and formats, convert data formats, unify encoding, and other operations to keep the data consistent in structure and semantics;

[0073] The integrated data is normalized to make the data have the same scale and distribution.

[0074] It should be noted that technical means such as merging and unifying the format of data are existing technologies. For example, in Python, the strptime() and strftime() functions of the datetime module can be used to convert date and time data into a unified format; string processing functions in programming languages ​​​​are used, such as Python's upper() or lower() methods to convert all texts into uppercase or lowercase forms. They will not be elaborated here. The normalization processing of the integrated data is also an existing technology, and other processing methods such as standardization processing can be performed according to actual needs, and are not limited to the choices in this embodiment.

[0075] Moreover, when extracting data from different sources in different proportions, the reliability constant of the data source can be used as the proportion, and random samples can be extracted from the sample according to the proportion. Then, the samples extracted from each data source can be merged together to form new data. It should be noted that a fixed total number of samples can also be set, and the proportion of the reliability constants of different data sources can be calculated. A certain number of samples can be extracted from the corresponding data sources according to the proportion to make the total number of samples constant. The extraction method should not be limited to the extraction method listed in this specific embodiment.

[0076] like Figure 2 As shown, when the dynamic correlation supplement method does not include the autocorrelation part of the time series, the dynamic correlation supplement method includes:

[0077] S1: Calculate the correlation coefficient of each eigenvalue to the missing eigenvalue based on the remaining eigenvalues ​​of the data, according to the formula:

[0078]

[0079] is the number of samples in the acquired data, where is the correlation coefficient between the jth feature and the kth feature, is the eigenvalue of the jth feature of the i-th sample, is the mean of the jth feature in n samples, is the eigenvalue of the kth feature of the i-th sample, is the mean of the kth feature in n samples;

[0080] S2: Determine the correlation weights between other features and missing features based on the correlation coefficient, according to the formula:

[0081]

[0082] middle To express the relative weight of the kth feature for filling the missing value of the jth feature, is the number of features for each sample;

[0083] S3: Obtain the fitted linear relationship based on historical data and calculate the relevant supplementary value according to the formula:

[0084]

[0085]

[0086] in is the dependent eigenvalue, is the intercept term, represents the eigenvalue of the kth feature, is the regression coefficient of feature k, is the error term, is the relevant supplementary value of the jth feature in sample i, is the eigenvalue of the kth feature of the i-th sample;

[0087] S4: Use The value of is used to supplement the j-th feature in sample i.

[0088] It should be noted that for ease of understanding, simplified simulation data is set as shown in Table 1 below:

[0089] Table 1

[0090] Trading day data type Opening price (yuan) Closing price (yuan) Trading volume (lots) Price-to-book ratio 1 35.2 35.5 42000 2.9 2 35.8 35.6 40000 2.8 3 36.1 Missing 43000 2.9 4 35.9 36.2 45000 3.0 5 36.3 36.0 40000 2.9

[0091] Based on the above data (the original data collected), first calculate (Correlation coefficient between opening price and closing price), put the above data into the calculation formula:

[0092]

[0093] get The value is approximately 0.66 (due to the missing closing price of the third day, the data of the third day is removed when calculating the mean of the closing price and the correlation coefficient between the opening price and the closing price).

[0094] Similarly, we can calculate (the correlation coefficient between volume and closing price) is about 0.50, The correlation coefficient between the price-to-book ratio and the closing price is about 0.75.

[0095]

[0096] It can be calculated is about 0.34, is about 0.27, About 0.39.

[0097] For the convenience of calculation, we assume that the linear relationship obtained from historical data is closing price = opening price × 2.5 + trading volume × 0.00004 + price-to-book ratio × 0.7, that is, is 0, is 2.5, is 0.00004, is 0.7, is 0, substitute into the formula:

[0098]

[0099] Available (The relevant supplementary value of the closing price on the third day) is approximately 36.1. The supplementary value is calculated using the dynamic correlation supplementation method to supplement the missing eigenvalues ​​of the data. Based on the correlation between other eigenvalues ​​of the data and the missing eigenvalues, the relevant weights of other eigenvalues ​​to the missing eigenvalues ​​are determined. Depending on the acquired data, the correlation changes dynamically, and the supplementation of missing eigenvalues ​​is more in line with the overall trend of the data, thereby improving the accuracy of the supplementary value.

[0100] In this embodiment, the method for obtaining a fitted linear relationship based on historical data includes:

[0101] L1: Collect historical data and set missing feature values ​​as dependent features based on the historical data;

[0102] L2: Setting is the intercept term, represents the eigenvalue of the kth feature, is the regression coefficient of feature k, is the error term, and the formula is determined ;

[0103] L3: Estimate the regression coefficient by the least squares method and determine 、 and The linear relationship of the fit is determined.

[0104] It should be noted that the method of determining the regression coefficient by the least square method is an existing technology and will not be described in detail here. In addition, other methods can also be used to fit the linear relationship and are not limited to the least square method used in this embodiment.

[0105] Example 2:

[0106] When the missing features of the data have an autocorrelated part in time, the accuracy of supplementing the missing feature values ​​only by the remaining feature values ​​is poor. In Example 1, the closing prices of each trading day have a certain influence on each other, and the closing price is calculated only by the opening price, trading volume and price-to-book ratio of the third trading day, without fully considering the mutual influence between the closing prices of multiple trading days before and after. In this example, the time correlation of the missing features themselves is calculated, and the supplemented value is more accurate.

[0107] like Figure 2 As shown, the difference from the first embodiment is that the dynamic correlation supplement method further includes:

[0108] G1: Select several consecutive complete samples and calculate the autocorrelation coefficient according to the formula:

[0109]

[0110] in Lag The autocorrelation coefficient of the period, To select the number of consecutive data periods, For continuous The average value of the period, is the data value of the tth period in the selected data period, is the data value of the t-δth period in the selected data period;

[0111] G2: Calculate the autocorrelation weight based on the autocorrelation coefficient:

[0112]

[0113] in Lag The autocorrelation weight of the period, is the autocorrelation coefficient of lag c, is the maximum lag period set;

[0114] G3: Calculate the final supplementary value based on the autocorrelation weight and the relevant supplementary value:

[0115]

[0116] in is the autocorrelation supplementary value of the j-th feature in sample i, is the final supplementary value of the j-th feature in sample i, For the The eigenvalue of the jth feature of the sample, To set the weight yourself, is the relevant supplementary value of the jth feature in sample i;

[0117] G4: Use The value of is used to supplement the j-th feature in sample i.

[0118] It should be noted that for the convenience of calculation and considering only one lag period, the closing prices of the fourth and fifth trading days are selected as the continuous data periods for calculation.

[0119] According to the formula:

[0120]

[0121] It can be calculated is 0.4, and then according to the formula:

[0122]

[0123] Available (The autocorrelation supplement value of the closing price on the third day) is 35.6 (for the convenience of calculation, only one lag is considered, so The value is 1. In actual use, the more lag periods considered, the more accurate the calculation). Set The value is 0.5, and we can get The final supplementary value of the closing price on the third day is 35.85. Based on the changing trends of the data in different time series, the autocorrelation coefficient of the missing feature is determined, and the autocorrelation weight of the eigenvalue of the previous period of the missing feature to the eigenvalue of the missing feature is calculated. The changing trend of the missing feature is fully considered, and the final supplementary value is determined by combining the influence of other eigenvalues ​​on the missing feature, which further improves the accuracy of the supplementary value.

[0124] Example 3:

[0125] Merging data from different sources will generate a large amount of computational effort, and the authenticity of data from different sources is different. Therefore, it is necessary to obtain the reliability constants of different sources through the reliability constant calculation method. According to the reliability constants of data from different sources, different proportions of data from different sources are extracted to reduce the computational burden, increase the preprocessing speed, and ensure that the number of samples from data sources with higher accuracy is larger, thereby improving the accuracy of the data.

[0126] like Figure 3 As shown in Figure 2, the reliability constant calculation method includes:

[0127] Get the total number of samples, the number of features in each sample, and the number of missing features in each sample, and remove samples with a missing ratio of more than 0.5 from several samples, and then calculate according to the formula:

[0128] ,

[0129] ,

[0130] in is the reliability constant of the data source, is the data defect constant, reflecting the degree of data missing, The credibility of the data source is between 0 and 1, which is set by technical personnel. The value indicates the degree of recognition of the authenticity of the data source. is the number of missing features in sample i, is the total number of samples in the acquired data, is the number of features for each sample.

[0131] It should be noted that for the convenience of calculation, simulated data is used as shown in Table 2 below. The data in the table is the time (in days) that investors pay attention to different stocks obtained from a stock exchange forum:

[0132] Table 2

[0133] stock investors Number One Number Two Number Three Number Four Number Five 100 15 25 45 18 22 200 5 Missing 5 Missing Missing 300 0 Missing 15 8 9

[0134] According to the above data (the original data collected), in the sample of investor No. 2, the missing ratio of data reaches more than 0.5, so the No. 2 sample is eliminated, and then the calculation is made. (Data defect constant), put the above data into the formula:

[0135]

[0136] Calculation can be obtained is 0.83, Between 0 and 1, the larger the value, the more complete the data. The value is 0.9, indicating a high degree of recognition of the authenticity of the data source. Then according to the formula:

[0137]

[0138] Calculate (reliability constant) is 0.747.

[0139] Example 4:

[0140] When calculating the reliability constant of the three samples in Example 3, the influence of serious defects in a single sample can be eliminated. However, when the missing features of the data are concentrated, the reliability constant calculated by this calculation method is too high and cannot accurately reflect the reliability of the data source. The difference from Example 3 is that this embodiment provides another reliability constant calculation method, which is calculated by characteristics, is more sensitive to the missing features, and is more accurate.

[0141] like Figure 4As shown in Figure 2, the reliability constant calculation method includes:

[0142] Get the total number of samples, the number of features in each sample, and the number of missing features in each sample, and calculate according to the formula:

[0143] ,

[0144]

[0145] in is the reliability constant of the data source, is the data defect constant, reflecting the degree of data missing, The credibility of the data source is between 0 and 1, which is set by technical personnel. The value indicates the degree of recognition of the authenticity of the data source. is the number of samples missing feature j, is the total number of samples in the acquired data, is the number of features for each sample.

[0146] It should be noted that, for ease of understanding, the data in Example 3 are used for calculation, according to the formula:

[0147]

[0148] Available is 0.73, set The value is 0.9, indicating a high degree of recognition of the authenticity of the data source. Then according to the formula:

[0149]

[0150] Calculate (reliability constant) is 0.657.

[0151] It should be noted that Example 3 and Example 4 use different calculation methods, and the calculation results of the reliability constant of the data source are different. Implementation of Example 3 by dividing the samples for calculation can eliminate the influence of serious defects in a single sample. Implementation of Example 4 by dividing the features for calculation is more sensitive to the lack of features and more accurate in calculation. It can be selected according to actual needs, and different calculation methods can be selected for different data sources.

[0152] Embodiment 5:

[0153] The reliability of the data source is not static. Affected by various factors, the quasi-reliability of the data source may gradually increase or decrease. In Examples 3 and 4, the reliability of the data source cannot be updated based on the collected data, making it difficult to ensure the long-term accuracy of the data. Based on Examples 3 and 4, this embodiment provides a credibility update method to continuously update the credibility, ensure the reliability of the data source, and make the integrated data more accurate.

[0154] like Figure 5 As shown, in the third and fourth embodiments, when the reliability constant of the data source ≤0.3 or When ≥0.7, the credibility of the data source is updated by the credibility update method The value of is updated to improve the accuracy of the next calculation. The credibility update method includes:

[0155] When 0.2< When ≤0.3, ; When 0.1< When ≤0.2, ;when When ≤0.1, , and prompt technical personnel to redefine the data source;

[0156] When 0.7≤ When ≤0.8, ; When 0.8< When ≤0.9, ;when >0.9, ;

[0157] in To ensure the credibility of the updated data source, is the credibility of the data source before the update, and ,in is the initial credibility, set by technical personnel, It is the update speed constant, set by the technicians. The larger the value, the greater the update amplitude.

[0158] It should be noted that 、 、 Both Different names in different calculation cycles, setting is 0.03, in Example 3, it is calculated (Reliability constant) is 0.747, 0.7≤ ≤0.8, and the reliability before the update is 0.9, The calculated result is 0.93, but assuming Set to 0.94, The maximum cannot exceed 0.94, so the updated is 0.93. The next time the reliability constant of this data source is calculated, Calculated according to 0.93, in addition, assuming Set to 0.92, The maximum cannot exceed 0.92, although the updated The calculated result is 0.93. The next time the reliability constant of this data source is calculated, It can be calculated according to 0.92. In general, when the calculated reliability constant is larger, the credibility of the data source can be improved. When the calculated reliability constant is smaller, the credibility of the data source can be reduced. When the credibility of the data source is less than the threshold, the technicians can be prompted to abandon the data source and look for a new data source to ensure the availability of the acquired data.

[0159] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is limited by the accompanying embodiments and their equivalents.

Claims

1. A data preprocessing method suitable for big data analysis, characterized by: include: Collect relevant raw data through big data and record the data source; After the original data is deduplicated, the dynamic correlation supplementation method is used to calculate the supplementary value to supplement the missing characteristic values ​​of the data; The reliability constants of different sources are obtained through the reliability constant calculation method. According to the reliability constants of data from different sources, different proportions of data from different sources are extracted. Merge and unify data from different sources and formats, convert data formats, and perform unified encoding operations to keep the data consistent in structure and semantics; Normalize the integrated data to make them have the same scale and distribution; The dynamic correlation supplementation method includes: S1: Calculate the correlation coefficient of each eigenvalue to the missing eigenvalue based on the remaining eigenvalues ​​of the data, according to the formula: ， is the number of samples in the acquired data, where is the correlation coefficient between the jth feature and the kth feature, is the eigenvalue of the jth feature of the i-th sample, is the mean of the jth feature in n samples, is the eigenvalue of the kth feature of the i-th sample, is the mean of the kth feature in n samples; S2: Determine the correlation weights between other features and missing features based on the correlation coefficient, according to the formula: ， in To express the relative weight of the kth feature for filling the missing value of the jth feature, is the number of features for each sample; S3: Obtain the fitted linear relationship based on historical data and calculate the relevant supplementary value according to the formula: ， ， in is the dependent eigenvalue, is the intercept term, represents the eigenvalue of the kth feature, is the regression coefficient of feature k, is the error term, is the relevant supplementary value of the jth feature in sample i, is the eigenvalue of the kth feature of the i-th sample; S4: Use The value of is used to supplement the j-th feature in sample i.

2. The data preprocessing method suitable for big data analysis according to claim 1, characterized in that: The dynamic correlation supplementation method also includes the following for the autocorrelation part of the time series: G1: Select several consecutive complete samples and calculate the autocorrelation coefficient according to the formula: ， in Lag The autocorrelation coefficient of the period, To select the number of consecutive data periods, For continuous The average value of the period, is the data value of the tth period in the selected data period, is the data value of the t-δth period in the selected data period; G2: Calculate the autocorrelation weight based on the autocorrelation coefficient: ， in Lag The autocorrelation weight of the period, is the autocorrelation coefficient of the lag period c, is the maximum lag period set; G3: Calculate the final supplementary value based on the autocorrelation weight and the relevant supplementary value: ， in is the autocorrelation supplementary value of the j-th feature in sample i, is the final supplementary value of the j-th feature in sample i, For the The eigenvalue of the jth feature of the sample, To set the weight yourself, is the relevant supplementary value of the jth feature in sample i; G4: Use The value of is used to supplement the j-th feature in sample i.

3. The data preprocessing method suitable for big data analysis according to claim 1, characterized in that: The method for obtaining a fitted linear relationship based on historical data includes: L1: Collect historical data and set missing feature values ​​as dependent features based on the historical data; L2: Setting is the intercept term, represents the eigenvalue of the kth feature, is the regression coefficient of feature k, is the error term, and the formula is determined ; L3: Estimate the regression coefficient by the least squares method and determine 、 and , thereby determining the fitted linear relationship.

4. The data preprocessing method suitable for big data analysis according to claim 1, characterized in that: The reliability constant calculation method includes: Get the total number of samples, the number of features in each sample, and the number of missing features in each sample, and remove samples with a missing ratio of more than 0.5 from several samples, and then calculate according to the formula: ， ， in is the reliability constant of the data source, is the data defect constant, reflecting the degree of data missing, The initial credibility of the data source is between 0 and 1, which is set by technical personnel. The value indicates the degree of recognition of the authenticity of the data source. is the number of missing features in sample i, is the total number of samples in the acquired data, is the number of features for each sample; When the reliability constant of the data source ≤0.3 or ≥0.7, the initial credibility of the data source is updated through the credibility update method. The value of is updated to , improve the accuracy of the next calculation.

5. The data preprocessing method suitable for big data analysis according to claim 1, characterized in that: The reliability constant calculation method includes: Get the total number of samples, the number of features in each sample, and the number of missing features in each sample, and calculate according to the formula: ， ， in is the reliability constant of the data source, is the data defect constant, reflecting the degree of data missing, The credibility of the data source is between 0 and 1, and its value indicates the degree of recognition of the authenticity of the data source. is the number of samples missing feature j, is the total number of samples in the acquired data, is the number of features for each sample; When the reliability constant of the data source ≤0.3 or When ≥0.7, the credibility of the data source is updated by the credibility update method The value of is updated to improve the accuracy of the next calculation.

6. A data preprocessing method suitable for big data analysis according to claim 4 or 5, characterized in that: The credibility updating method includes: When 0.2< When ≤0.3, ; When 0.1< When ≤0.2, ;when When ≤0.1, , and prompt technical personnel to redefine the data source; When 0.7≤ When ≤0.8, ; When 0.8< When ≤0.9, ;when >0.9, ; in To ensure the credibility of the updated data source, is the credibility of the data source before the update, and ,in is the initial credibility, set by technical personnel, It is the update speed constant, set by the technicians. The larger the value, the greater the update amplitude.

7. A data preprocessing system suitable for big data analysis, using the data preprocessing method suitable for big data analysis according to any one of claims 1 to 6, characterized in that: include: Data collection module: used to collect relevant raw data through big data and record the source of the data; Data processing module: used to remove duplicates from original data, calculate supplementary values ​​using a dynamic correlation supplementation method, supplement missing eigenvalues ​​of the data, and determine the relevant weights of other eigenvalues ​​to the missing eigenvalues ​​based on their correlation. The correlation changes dynamically depending on the acquired data, making supplementation of missing eigenvalues ​​more consistent with the overall trend of the data and improving the accuracy of the supplementary values. Data screening module: used to obtain reliability constants from different sources through the reliability constant calculation method. According to the reliability constants of data from different sources, different proportions of data from different sources are extracted to reduce the calculation burden and improve the preprocessing speed. Data integration module: merges and unifies data from different sources and formats, converts data formats, and performs unified encoding operations to ensure consistency in data structure and semantics; Data conversion module: Normalizes the integrated data to make the data have similar scale and distribution. When the integrated data is used for model training, it can improve the convergence speed of the machine learning algorithm and the accuracy of the model.

Images