A probabilistic and trusted blockchain method for massive time series data

Abstract

Aiming at the problem that the blockchain on-chain operation is time-consuming and the high-speed time series data on-chain delay leads to the inability to upload batch data to the chain, the present invention proposes a reliable on-chain method for time series data with probability sampling. By dividing the time series data into blocks of equal size and sampling and uploading them within the blocks, the serial problem is parallelized, and only the sampled data is uploaded to the chain. The density of the sampled data ensures that the off-chain data cannot be tampered with.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and specifically relates to a probabilistic credible data uploading method for massive time-series data Background Art

[0002] Currently, the data received by information systems such as sensor networks and the Internet of Things are generally massive, high-speed real-time data. Regarding the problem of uploading high-speed real-time data to the blockchain, due to the operation delay of the blockchain itself and the problem of distributed mutual verification, there is no effective solution so far, and only the traditional data encryption method can be used to achieve centralized data security

[0003] Sampling is an indispensable method in research and data collection, which can obtain meaningful insights from larger data and make a subset of wise decisions. Different research fields have adopted different sampling techniques, and each technique has its unique advantages and limitations. Common sampling techniques include random sampling, stratified sampling, systematic sampling, cluster sampling, and convenience sampling, etc Summary of the Invention

[0004] In view of the deficiencies of the prior art, the present invention proposes a credible probabilistic high-speed time-series data uploading technology based on sampling techniques. Aiming at the problem that the blockchain data uploading operation is time-consuming and the high-speed time-series data uploading delay causes batch data unable to be uploaded, a credible time-series data uploading technology based on probabilistic sampling is proposed. By dividing the time-series data into equal-sized blocks, sampling and uploading within the blocks, the serial problem is parallelized, and only the sampled data is uploaded to the blockchain. The density of the sampled data ensures the immutability of the off-chain data

[0005] To achieve the above object, the present invention provides the following technical solutions

[0006] A probabilistic credible data uploading method for massive time-series data, comprising the following steps

[0007] Step S1: Cut the time-series data in batches for data segmentation, obtain time-series data in m batch quantities, store the time-series data in blocks, and set the size of each data block at M bytes

[0008] Step S2: Obtain the data uploading time tc of the time-series data, the data memory storage time tm, and the data acquisition time tg

[0009] If tc + tm < tg, then upload all the time-series data to the blockchain

[0010] If tc + tm > tg, then perform sampling on the time-series data during acquisition, and upload the sampled time-series data, expressed as

[0011] (k - 1)tg < tc + tm < ktg

[0012] Where k is the number of times data is collected, k≥2;

[0013] Step S3: During non-production time, the uploaded time series data is authenticated by matching hash values.

[0014] Furthermore, the number of acquisitions in step S2 is determined according to the parallel computing capability of the system, specifically:

[0015] In the process of collecting time series data, the data on-chain calculation is performed in parallel. Assuming that the number of parallel calculations is I, if the time of k-time data collection is greater than the time of I-time data on-chain, that is, the time series data acquired within the collection time is less than the processed data of the parallel execution of data on-chain calculation, then all the time series data is on-chain;

[0016] If the time for collecting data k times is less than the time for uploading data I times, that is, the time series data obtained during the collection time is greater than the processed data of the parallel execution data uploading calculation, data sampling is performed at this time, that is, a number of time series data are sampled from the collected time series data for uploading to the chain so that the time series data obtained during the collection time is less than the processed data of the parallel execution data uploading calculation, and the sampling interval frequency is t=k / l and is rounded up; that is, in the data block of the time series data, every t data are uploaded to the chain, which can effectively ensure the balance between data collection and uploading.

[0017] Furthermore, the sampling method in step S2 includes random sampling and / or stratified sampling and / or systematic sampling and / or cluster sampling and / or convenience sampling.

[0018] Further, the sampling interval frequency in step S2 is t=k / 1 and the rounding process is performed to dynamically adjust the sampling interval frequency to an integer without rounding;

[0019] The dynamic adjustment process is to fine-tune the uplink power and / or acquisition power of the processor to fine-tune k and I so that the sampling interval frequency is an integer.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention proposes a trusted chain-up technology for time series data based on sampling. By dividing the time series data into blocks of equal size, sampling and chain-up within the blocks, the serial problem is parallelized, and only the sampled data is chained. The density of the sampled data ensures that the off-chain data cannot be tampered with. Furthermore, by dynamically fine-tuning the chain-up power and / or the acquisition power, the sampling interval frequency is made an integer, and the chain-up efficiency is further improved without affecting the data stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1It is a flowchart of the steps of a method for securely probabilistically high-speed time-series data uploading based on sampling technology. Detailed implementation manners

[0023] To make the technical means, creative features, achieved objectives and effects of the present invention easy to understand, in the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be construed as a limitation to the present invention. In addition, the terms "No. 1", "No. 2", "No. 3" are only for descriptive purposes and cannot be construed as indicating or implying relative importance. The present invention will be further described below in conjunction with specific implementation manners.

[0024] Refer to Figure 1 , a probabilistic trust uploading method for massive time-series data. In one example, it includes the following steps:

[0025] Step S1: High-speed real-time data has a strict time partial order property, that is, the data dn at time n must be collected before the data dn+1 at time n+1. Based on the above understanding, the time-series data is cut in batches for data segmentation to obtain time-series data in m batch quantities, and the time-series data is stored in blocks, and the size of each data block is set at M bytes; among them, for time-series data, since only a certain amount of data has real value, data tampering generally occurs in batches, and M bytes is a parameter that makes the time-series data meaningful.

[0026] Step S2: Obtain the data uploading time tc of the time-series data, the data memory storage time tm, and the data collection time tg;

[0027] If tc + tm < tg, then upload all the time-series data;

[0028] If tc + tm > tg, then sample the time-series data for collection and upload the sampled time-series data, which is expressed as:

[0029] (k - 1)tg < tc + tm < ktg;

[0030] where k is the number of times of data collection, and k ≥ 2;

[0031] Step S3: During non-production time, the time series data on the chain is authenticated by hash value matching. Therefore, for business data, we can authenticate the data by hash value matching of the full amount of data on the chain during non-production time. The non-production time can be set to 12:00 a.m., and authentication can be performed on a daily, weekly, or ten-day basis.

[0032] Furthermore, for high-performance servers, due to the multi-core and multi-CPU architecture, data on-chain calculations can be performed in parallel. The number of collections in step S2 is determined according to the parallel computing capability of the system, specifically:

[0033] In the process of collecting time series data, the data on-chain calculation is performed in parallel. Assuming that the number of parallel calculations is I, if the time of k-time data collection is greater than the time of I-time data on-chain, that is, the time series data acquired within the collection time is less than the processed data of the parallel execution of data on-chain calculation, then all the time series data is on-chain;

[0034] If the time for collecting data k times is less than the time for uploading data I times, that is, the time series data obtained during the collection time is greater than the processed data of the parallel execution data uploading calculation, data sampling is performed at this time, that is, a number of time series data are sampled from the collected time series data for uploading to the chain so that the time series data obtained during the collection time is less than the processed data of the parallel execution data uploading calculation, and the sampling interval frequency is t=k / l and is rounded up; that is, in the data block of the time series data, every t data are uploaded to the chain, which can effectively ensure the balance between data collection and uploading.

[0035] Furthermore, the sampling method in step S2 includes random sampling and / or stratified sampling and / or systematic sampling and / or cluster sampling and / or convenience sampling.

[0036] Further, the sampling interval frequency in step S2 is t=k / 1 and the rounding process is performed to dynamically adjust the sampling interval frequency to an integer without rounding;

[0037] The dynamic adjustment process is to fine-tune the uplink power and / or acquisition power of the processor to fine-tune k and I so that the sampling interval frequency is an integer.

[0038] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A probabilistic trusted chaining method for massive time series data, characterized in that: It includes the following steps: Step S1: Cut the time-series data in batches for data segmentation, obtain the time-series data in m batch quantities, store the time-series data in blocks, and set the size of each data block at M bytes; Step S2: Obtain the data uploading time tc, the data memory storage time tm, and the data acquisition time tg of the time-series data; If tc + tm < tg, then upload all the time-series data; If tc + tm > tg, then sample the time-series data for acquisition and upload the sampled time-series data, which is expressed as: (k - 1)tg < tc + tm < ktg; where k is the number of data acquisition times, and k ≥ 2; Step S3: Authenticate the uploaded time-series data in the non-production time by means of hash value matching.

2. According to claim 1, a probabilistic trusted chaining method for massive time series data is characterized in that: The number of acquisitions in Step S2 is determined according to the concurrent computing power of the system. Specifically: During the process of acquiring time-series data, perform data uploading calculation in parallel. Assume the number of parallel calculations is I. If the time for k times of data acquisition is greater than the time for I times of data uploading, that is, the time-series data obtained within the acquisition time is less than the processed data for parallel data uploading calculation, then upload all the time-series data; If the time for k times of data acquisition is less than the time for I times of data uploading, that is, the time-series data obtained within the acquisition time is greater than the processed data for parallel data uploading calculation. At this time, perform data sampling, that is, sample several time-series data from the acquired time-series data for uploading so that the time-series data obtained within the acquisition time is less than the processed data for parallel data uploading calculation. The sampling interval frequency is t = k / l and rounded up; that is, upload once every t data in the data block of the time-series data, then it can effectively ensure the balance between data acquisition and uploading.

3. According to claim 1, a probabilistic trusted chaining method for massive time series data is characterized in that: The sampling methods in Step S2 include random sampling and / or stratified sampling and / or systematic sampling and / or cluster sampling and / or convenience sampling.

4. According to claim 2, a probabilistic trusted chaining method for massive time series data is characterized in that: Dynamically adjust the sampling interval frequency of t = k / l and rounded up in Step S2 to an integer without rounding; The dynamic adjustment process is to finely adjust the uploading power and / or acquisition power of the microprocessor to finely adjust k and I so that the sampling interval frequency is an integer.

Images