Adaptive collection method for big data

Abstract

Provided in the present invention is a method, i.e., an adaptive collection method for big data. The method comprises: using a Bisquare algorithm to construct a unary linear regression model, so as to sense a change in data, and dynamically adjusting a collection time interval on the basis of a gradient, which specifically involves: performing linear fitting on a finite number of nearest discrete data points by means of a least squares method, obtaining an initial slope of a fitting model and an estimated value of an intercept, using a residual minimization method to obtain an updated value of a weight, using residual minimization to derive the initial slope and make same equal to zero, and performing a weighted least squares method on a newly obtained weight wi, so as to obtain ai+2 and bi+2; and by means of repeatedly performing the above process, when the relative difference between slopes and between intercepts for a fitting polynomial before iteration and a fitting polynomial after iteration is less than a tolerance value, the fitting model reaching an optimal state, and determining, on the basis of an obtained optimal fitting slope, a time interval for the next collection. The degree of distortion is relatively low, and the precision of data collection can be improved.

Description

A big data adaptive collection method Technical Field

[0001] The present invention relates to the field of big data governance and fusion technology, and in particular to a big data adaptive collection method. Background Art

[0002] Big data governance refers to the data management process aimed at maximizing data value and minimizing production risks. The role of data governance is to ensure data normalization and reduce data redundancy; the core of data governance is planning, monitoring, and implementation. Data governance planning refers to the definition of standardized management data models. Users can create standard models that meet business requirements and configure the model's attributes and rules to design quality inspection rules and data cleansing principles. Data governance monitoring involves previewing data information and tracking table and field relationships, thereby gaining a comprehensive understanding of the data's entire lifecycle during the data governance process. Data governance implementation refers to the detailed data governance services provided after configuring the rules and standards for data cleansing, data concentration, and quality auditing.

[0003] Big data governance is a tedious project that requires continuous implementation and constant exploration by researchers. The framework of big data governance is shown in Figure 1, which includes work tasks in five architectural aspects: organization, standards, processes, technology, and evaluation.

[0004] Against the backdrop of the rapid development of information technology, the emergence and widespread application of big data have posed new challenges to its management and integration. Big data governance aims to effectively manage, integrate, and analyze big data, ensuring high quality, consistency, and security. Big data integration involves integrating and correlating multidimensional data from multiple sources, formats, and domains to generate more comprehensive, accurate, and valuable information.

[0005] Scholars both domestically and internationally have conducted extensive research on big data governance and fusion. Within the field of big data governance, some studies have focused on data quality assessment and cleaning, such as missing value processing and noise data filtering. Furthermore, scholars have investigated data security and privacy protection methods, including data encryption, access control, and privacy masking. In the field of big data fusion, researchers have proposed data matching and fusion algorithms to address heterogeneity and inconsistencies. Furthermore, researchers have proposed algorithms for mining implicit relationships and patterns from multi-source data.

[0006] However, while there is some research on big data governance and integration, several issues remain, such as a lack of standards and specifications, practical validation, and application examples. Therefore, this study proposes a set of adaptive algorithms for big data governance, addressing the limitations of existing algorithms in terms of standardization, data adaptability, large-scale data processing, and practicality, from the perspectives of data collection and data security. Technical issues

[0007] The technical problem to be solved by the present invention is to provide a big data adaptive collection method, which can automatically adjust the collection time interval according to the data characteristics and changes, has low distortion, and can improve the accuracy of data collection. Technical Solutions

[0008] The present invention provides a method for adaptively collecting big data, characterized by using a Bisquare algorithm to construct a univariate linear regression model, utilizing the univariate linear regression model to sense data changes, and dynamically adjusting the collection time interval according to the inclination of the univariate linear regression model. The dynamic adjustment includes the following steps:

[0009] S1, through the least squares method, the nearest finite number of discrete data points m2(x 2， y2),

[0010] m1(x1, y1)…m n (x n ,y n ) to perform a linear fit and then obtain the estimated values ​​of the initial slope and intercept of the fitted model and :

[0011]

[0012] Where, is the average value of the y-axis coordinates of the discrete data points, is the average x-axis coordinate of the discrete data points, i is the serial number of the discrete data point, and n is the length of y;

[0013] S2. Use the residual minimization method to obtain the updated value of the weight. The residual minimization calculation formula is:

[0014]

[0015] Where, is the residual value of the weight of the data point, is the y-axis coordinate value of different discrete data points, is the weight of the corresponding data point, is the best fitting value for the corresponding data point, is the derivative value of the corresponding data point, is the x-axis coordinate value of the discrete data point;

[0016] S3. According to the obtained and , and then use the residual to minimize the Take the derivative and make is equal to zero, we can get:

[0017]

[0018] S4. The newly obtained weight Perform weighted least squares to obtain the slope of the i+2th discrete data point With the intercept :

[0019]

[0020] S5, iterating by repeating step S4 until the relative difference between the slope and intercept of the fitting polynomial before and after the iteration is less than the tolerance value, at which point the fitting model reaches an optimal state, and in this state, outputting the optimal fitting slope;

[0021] S6. Determine the time interval for the next acquisition based on the range of the obtained best fitting slope.

[0022] Furthermore, the S6 is specifically: if the absolute value of the slope The smaller the value, the larger the sampling time interval is selected; on the contrary, if the absolute value of the slope The larger the value, the smaller the collection time interval.

[0023] Furthermore, after determining the time interval for the next collection, the big data adaptive collection method will differentiate the upper and lower bounds of the optimal fit model from the standard data source during the next data collection, and exclude points with a difference exceeding 0 as abnormal data points.

[0024] Furthermore, when storing data, security protection is also provided by a big data security model. The data confidentiality mechanism adopted by the big data security model includes offline training of users, confirmation and storage of models, and aggregation of models. The offline training of users uses differential privacy technology to conduct deep learning on local data, update cryptographic weights and local errors in real time, and upload them to the InterPlanetary File System IPFS node, and compress their hash values ​​into a protocol. The confirmation and storage of the model and the aggregation of the model are both for users to download confirmed transactions and download corresponding weights through various addresses to complete model aggregation, and update to the blockchain after confirmation.

[0025] Furthermore, the offline training utilizes simulation experiments to collaboratively train privacy-preserving deep learning based on blockchain platforms. The two most basic modeling methods, multi-layer perceptrons and convolutional neural networks, are used to model independent and identically distributed data and non-independent and identically distributed data, respectively. The algorithm's performance is evaluated from three perspectives: data compression rate, training accuracy, and training efficiency. Beneficial effects

[0026] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: In terms of data collection, the Bisquare algorithm is used to construct a univariate linear regression model, and the univariate linear regression model is used to accurately perceive data changes. By reducing the impact of noise data and dynamically adjusting the collection time interval according to the inclination of the univariate linear regression model, it has low distortion and can improve the accuracy of data collection. In terms of data storage, the big data privacy protection algorithm based on blockchain technology can achieve a data compression rate of 0.0018. On the one hand, it has good data storage effect and a high compression rate in IPFS, which reduces storage costs; on the other hand, it maintains the high availability of the model while ensuring data privacy.

[0027] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] FIG1 is a schematic diagram of the framework of big data governance of the present invention;

[0030] FIG2 is a flow chart of a process for determining a collection time interval according to an embodiment of the present invention;

[0031] FIG3 is a schematic diagram of the noise filtering principle according to an embodiment of the present invention;

[0032] FIG4 is a schematic diagram of a privacy-preserving deep learning framework based on efficient blockchain storage according to an embodiment of the present invention;

[0033] FIG5 is a schematic diagram showing a comparison result between a fitting model according to an embodiment of the present invention and an optimal signal source fitting model;

[0034] FIG6 is a schematic diagram showing the comparison between the least squares method and the Bisquare fitting model according to an embodiment of the present invention;

[0035] FIG7 is a schematic diagram illustrating the storage efficiency of an efficient deep learning privacy protection algorithm based on blockchain according to an embodiment of the present invention;

[0036] FIG8 is a schematic diagram showing a comparison of model accuracy under different systems according to an embodiment of the present invention. Modes for Carrying Out the Invention

[0037] The embodiment of the present application provides a big data adaptive collection method. In terms of data collection, it can automatically adjust the collection time interval according to data characteristics and changes, has low distortion, and can improve the accuracy of data collection.

[0038] The technical solution in the embodiments of this application has the following general principles: For data collection, the Bisquare algorithm is used to construct a univariate linear regression model. This univariate linear regression model is then used to accurately perceive data changes and dynamically adjust the collection interval based on the inclination of the univariate linear regression model, thereby reducing distortion and improving data collection accuracy. For data storage, a big data privacy protection algorithm based on blockchain technology can achieve a data compression ratio of 0.0018, significantly reducing the storage cost of the model.

[0039] Example

[0040] This embodiment provides a method for adaptively collecting big data. The method uses the Bisquare algorithm to construct a univariate linear regression model, uses the univariate linear regression model to sense data changes, and dynamically adjusts the collection time interval based on the inclination of the univariate linear regression model. The dynamic adjustment includes the following steps:

[0041] S1, through the least squares method, the nearest finite number of discrete data points ,

[0042] Perform a linear fit and then obtain estimates of the initial slope and intercept of the fitted model and :

[0043]

[0044] Where, is the average value of the y-axis coordinates of the discrete data points, is the average x-axis coordinate of the discrete data points, is the serial number of the discrete data point, and n is the length of y;

[0045] S2. Use the residual minimization method to obtain the updated value of the weight. The residual minimization calculation formula is:

[0046]

[0047] Where, is the residual value of the weight of the data point, is the y-axis coordinate value of different discrete data points, is the weight of the corresponding data point, is the best fitting value for the corresponding data point, is the derivative value of the corresponding data point, is the x-axis coordinate value of the discrete data point;

[0048] S3. According to the obtained and , and then use the residual to minimize the Take the derivative and make is equal to zero, we can get:

[0049]

[0050] S4. The newly obtained weight Perform weighted least squares to obtain the slope of the i+2th discrete data point With the intercept :

[0051]

[0052] S5, as shown in FIG2, by repeating the step S4 to achieve iteration until the relative difference between the slope and intercept of the fitting polynomial before and after the iteration is less than the tolerance value, the fitting model reaches the optimal state, in which case the optimal fitting slope is output;

[0053] S6. Determine the time interval for the next acquisition based on the range of the best fitting slope. The smaller the value, the larger the sampling time interval is selected; on the contrary, if the absolute value of the slope The larger the value is, the smaller the collection time interval will be. For example, if the maximum time interval is set to , the minimum time interval is , in and Between, then:

[0054] when When the sampling time interval is the maximum value ;

[0055] when When the collection time interval is set to an appropriate value ;

[0056] when When the sampling time interval is the minimum .

[0057] After determining the time interval for the next collection, the big data adaptive collection method will differentiate the upper and lower bounds of the optimal fit model from the standard data source during the next data collection, and exclude points with a difference exceeding 0 as abnormal data points.

[0058] In the process of data collection and processing, the most important part is to filter outlier noise in the data to improve the fit of the model. The data collection strategy based on the Bisquare algorithm has a great advantage in noise filtering. As shown in Figure 3, the noise filtering principle is based on the existing Bisquare matching model. The data acquisition system can set upper and lower limits at both ends of the Bisquare matching model and set the upper and lower limit thresholds as needed. .

[0059] When storing data, it is also protected by a big data security model. The data confidentiality mechanism adopted by the big data security model includes offline training of users, confirmation and storage of models, and aggregation of models. The offline training of users uses differential privacy technology to conduct deep learning on local data, update cryptographic weights and local errors in real time, and upload them to the Interstellar File System IPFS node, and compress their hash values ​​into a protocol. The confirmation and storage of the model and the aggregation of the model are both users downloading confirmed transactions, and downloading corresponding weights through various addresses to complete model aggregation, and then update to the blockchain after confirmation.

[0060] The offline training uses simulation experiments to collaboratively train privacy-preserving deep learning based on blockchain platforms. It adopts the two most basic modeling methods, multi-level perceptron and convolutional neural network, to model independent and identically distributed data and non-independent and identically distributed data respectively, and evaluates the performance of the algorithm from three perspectives: data compression rate, training accuracy, and training efficiency.

[0061] The blockchain is a public ledger. After a transaction is confirmed and updated, a new block is generated. The data within this block cannot be altered or corrupted. In this ledger, each data block is divided into two parts: the first part includes the address, timestamp, random number, Merkle tree root hash, and so on of the previous block. The block body records the gradient data uploaded by the user and is recorded in the form of a hash chain. The Merkle root hash is then recorded in the block header.

[0062] In classic deep learning network architectures, the model often contains multiple hidden layers, each of which contains a certain number of neurons and is assigned corresponding weights. Previous research has generally used gradient descent algorithms to improve the loss function. The most commonly used method is stochastic gradient descent. This algorithm uses training data and current weights to calculate the gradient. Furthermore, if the gradient is known, the current weight value can also be calculated. The weight update method is as follows:

[0063]

[0064] In the above formula (8), the learning rate is introduced, and then the process is updated through continuous iteration until the predetermined cost function or the predetermined number of iterations is reached to achieve convergence. In order to protect data privacy, the most commonly used algorithm is a typical distributed algorithm, which applies differential privacy technology in gradient calculation and processes the calculated local gradient values ​​in a secure manner. In addition, the algorithm also constructs a semi-trusted central server to achieve aggregate calculation of the global model. In the system, each training node will train the local data. After the calculation is completed, the system will upload the local gradient to the central server, and then the central server will be responsible for updating the global model. The update method is:

[0065]

[0066] Among them, by vector The analysis shows that when the selection rate is between 1% and 2%, the algorithm's computational efficiency is greatly improved, and the effectiveness of the algorithm is guaranteed. The local gradient protection method in this strategy is achieved by adding Laplace noise to local learning.

[0067] Based on the above, the specific process of the data privacy protection algorithm based on blockchain and deep learning algorithms constructed by the present invention is shown in Figure 4. The data confidentiality mechanism includes three main parts: offline training of users, confirmation and storage of models, and aggregation of models. During the offline training phase, differential privacy technology is used to perform deep learning on local data, updating cryptographic weights and local errors in real time. The data is uploaded to the Inter Planetary File System (IPFS) node and its hash value is compressed into a protocol. During the confirmation and model aggregation phases, users can download confirmed transactions and download the corresponding weights from each address to complete the model aggregation. After confirmation, the blockchain is updated.

[0068] In order to verify the application effect of the adaptive algorithm of the present invention in big data management and integration, the following experimental design is made:

[0069] 1. Big data adaptive collection effect based on Bisquare algorithm.

[0070] Experimental Design: To validate the performance of the data acquisition model based on the Bisquare algorithm, LabVIEW technology was used to design an optimal observation signal source with five amplitudes conforming to a normal distribution, enabling it to more accurately simulate the characteristics of the observed signal. This source was then used as the optimal data source for data collection. Furthermore, during data collection and testing, most noise is random, with irregular amplitudes. Therefore, Gaussian white noise and uniform white noise were added to the original optimal signal source to simulate the effects of different types of noise in actual data collection and transmission. The Bisquare adaptive algorithm was used to simulate sampling of these two signal sources and compare them with the optimal signal source. The goodness of fit of the Bisquare model was then tested. Furthermore, the superiority of the Bisquare adaptive method in data collection was verified by comparing the data collection performance of the Bisquare adaptive method with that of the least squares adjustment (LS).

[0071] (1) Fitting effect of Bisquare model

[0072] First, the fitting effect was studied and compared with the fitting effect of the best signal source model. The results are shown in Figure 5. As can be seen from Figure 5, the Bisquare algorithm can effectively reduce the impact of noise on model fitting. The data trend of the Bisquare fitting is very consistent with the best signal source and can accurately reflect the changes in the data.

[0073] (2) Comparison of adaptive acquisition effects between the least squares method and the Bisquare algorithm

[0074] The study tested the designed standard data source using the least squares method and the Bisquare algorithm. Figure 6 shows the comparative results of the experimental collection performance. It can be seen that the distortion of the least squares adaptive collection is quite high. This is because the least squares fitting model cannot effectively distinguish noisy data. In addition, for data containing a large amount of noise, the model will produce significant distortion. Based on the Bisquare adaptive sampling algorithm, a weighted algorithm is used to minimize the impact of abnormal noise. The resulting fitting model is more similar to the optimal data source.

[0075] Furthermore, the degree of data acquisition distortion for the two aforementioned methods can be expressed as the area between the data curves obtained using the two methods and the optimal data source curve. Calculations show that, within the same timeframe, the least squares method and the Bisquare method achieved adaptive sampling values ​​of 1.74 and 1.21, respectively. Furthermore, in a one-minute data collection test, the Bisquare algorithm achieved lower distortion for each data point than the least squares method. These data clearly demonstrate the superior data acquisition performance of the Bisquare-based adaptive acquisition algorithm.

[0076] 2. The effect of efficient deep learning privacy protection algorithm based on blockchain

[0077] Experimental Design: A simulation experiment will be used to collaboratively train privacy-preserving deep learning on a blockchain platform. The algorithm's performance will be evaluated from three perspectives: data compression rate, training accuracy, and training efficiency. The experiment will employ two fundamental modeling approaches: multi-layer perceptrons and convolutional neural networks. The experiment will model independent and identically distributed (IID) and non-IID data, respectively. The experiment will examine modeling accuracy under different data distributions, modeling effectiveness under different privacy protection levels, and storage efficiency on the blockchain. The experimental environment is shown in Table 2.

[0078]

[0079] Furthermore, when conducting experiments with IID data, the data was randomly shuffled and evenly distributed to 100 clients for local training. Each client received 600 examples for training. In experiments with non-IID data, the data was sorted by numerical label, divided into 200 data segments of size 300, and then distributed to individual clients for processing.

[0080] (1) Storage efficiency

[0081] The study used the block compression coefficient as an indicator to evaluate the experimental results. By performing one iteration on 100 clients, the spatial compression ratio of the MNIST data was simulated and the two storage methods were compared. The results of the comparison are shown in Figure 7.

[0082] Figure 7 shows that when storing data in IPFS, the increase in data size is minimal, and the data block compression ratio is 0.0018, which greatly reduces the storage overhead of the model. Furthermore, on the IPFS network, users do not need to pay any fees; access users only need to wait for confirmation, without any additional burden.

[0083] (2) Comparative analysis of model performance

[0084] In the experiment, the study will use two different methods to conduct comparative experiments: Convolutional Neural Network (CNN) and Multilayer Perceptron (MLP), and evaluate the proposed model from the two aspects of accuracy and efficiency. The efficiency of the algorithm is compared with the classic federated averaging algorithm, and the efficiency coefficient of the algorithm is obtained, which is calculated as follows:

[0085]

[0086] in, is the time cost of this algorithm, is the time cost of the traditional federated learning algorithm. The specific experimental results are shown in Table 3.

[0087]

[0088] Table 3 shows that, under certain privacy thresholds, both models achieved very high prediction accuracy in the test set. Conversely, under the same privacy threshold, the convolutional neural network achieved higher clustering accuracy, but the clustering process took longer, resulting in poorer results. The MLP method achieved better clustering than the CNN, but its results differed significantly from those of the CNN method. These results demonstrate that the proposed method is also feasible within other deep learning algorithms and improves model usability even with increasing network complexity.

[0089] Secondly, we compare the proposed clustering algorithm with a traditional distributed clustering algorithm based on a semi-trusted central server. We propose a clustering algorithm based on trust, called CTR-ACC, which uses a trusted third party to cluster the clustering results and then publishes the clustering results to each node during the clustering process. The comparison results of the two schemes are shown in Figure 8.

[0090] Figure 8 shows that the proposed algorithm can obtain equally accurate measurement results compared to the ctr-acc system. Furthermore, when the privacy budget threshold is greater than or equal to 4, the proposed algorithm also achieves a higher recognition rate. In summary, the proposed algorithm can both protect data privacy and maintain high model availability.

[0091] In summary, the experiments show that: (1) the data collection model based on the Bisquare algorithm can accurately reflect the changes in data, and by reducing the influence of noise data, a fitting model that is closer to the optimal data source is obtained. In comparison, the adaptive collection distortion of the least squares method is higher; (2) the efficient deep learning privacy protection algorithm based on blockchain technology has a better data storage effect and a higher compression rate in IPFS, which reduces storage costs; on the other hand, compared with traditional solutions, the algorithm proposed in the study maintains the high availability of the model while ensuring data privacy.

[0092] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A big data adaptive collection method, characterized by: A univariate linear regression model is constructed using the Bisquare algorithm, and the univariate linear regression model is used to sense data changes, and the time interval of collection is dynamically adjusted according to the inclination of the univariate linear regression model. The dynamic adjustment includes the following steps: S1. Use the least squares method to transform the nearest finite number of discrete data points ， Perform a linear fit and then obtain estimates of the initial slope and intercept of the fitted model and ： In the formula, For discrete data points Axis coordinate mean, For discrete data points Axis coordinate mean, is the sequence number of the discrete data point, for Length; S2. Use the residual minimization method to obtain the updated value of the weight. The residual minimization calculation formula is: In the formula, is the residual value of the weight of the data point, The serial number is of discrete data points Axis coordinate values, The serial number is The weight of the data point, The serial number is The best fit value for the data points, The serial number is The derivative value of the data point is For discrete data points Axis coordinate values; S3. According to what has been obtained and , and then use the residual to minimize the To derive and is equal to zero, we can get: S4. The newly obtained weight Using weighted least squares method, we get The slope of discrete data points With intercept ： S5, iterating by repeating step S4 until the relative difference between the slope and intercept of the fitting polynomial twice before and after the iteration is less than the tolerance value, at which time the fitting model reaches the best state, and in this state, the best fitting slope is output; S6. Determine the time interval for the next collection according to the range of the obtained best fitting slope.

2. A big data adaptive collection method according to claim 1, characterized in that: The specific S6 is: if the absolute value of the slope The smaller the slope, the larger the sampling time interval is selected. On the contrary, if the absolute value of the slope The larger the value, the smaller the collection time interval will be.

3. The big data adaptive collection method according to claim 1, characterized in that: After determining the time interval for the next collection, the big data adaptive collection method will, during the next data collection, differentiate the upper and lower bounds of the optimal fit model from the standard data source, and exclude points where the difference exceeds 0 as abnormal data points.

4. The method for adaptively collecting big data according to claim 1, characterized in that: When storing data, security protection is also provided through a big data security model. The data confidentiality mechanism adopted by the big data security model includes offline training of users, confirmation and storage of models, and aggregation of models. The offline training of users utilizes differential privacy technology to conduct deep learning on local data, update cryptographic weights and local errors in real time, and upload them to the Interstellar File System IPFS node, and compress their hash values ​​into a protocol. The confirmation and storage of the model and the aggregation of the model are both for users to download confirmed transactions and download corresponding weights through various addresses to complete model aggregation and update to the blockchain after confirmation.

5. A big data adaptive collection method according to claim 4, characterized in that: The offline training uses simulation experiments to conduct collaborative training on privacy-preserving deep learning based on blockchain platform, adopts the two most basic modeling methods of multi-level perceptron and convolutional neural network, models independent and identically distributed data and non-independent and identically distributed data respectively, and evaluates the performance of the algorithm from three perspectives: data compression rate, training accuracy, and training efficiency.

Images