A big data distributed storage method and system for the Internet of Things

Abstract

The present invention provides a big data distributed storage method and system for the Internet of Things, which relates to the field of data processing, wherein the system comprises: a data processing module, which is used to receive the Internet of Things data collected by multiple Internet of Things terminals and perform storage preprocessing; the data processing module is used to perform storage preprocessing on the Internet of Things data and determine whether to cache the Internet of Things data; the data storage module is used to store the preprocessed Internet of Things data; the data cache module is used to allocate multiple optimal data cache nodes to cache the preprocessed Internet of Things data after the data preprocessing node determines to cache the Internet of Things data; the computing power scheduling node is also used to allocate the data preprocessing node for the data query request initiated by the user terminal, and the data preprocessing node is used to retrieve the target Internet of Things data corresponding to the data query request from the data storage module or the data cache module, which has the advantage of improving the security of the big data storage of the Internet of Things.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a big data distributed storage method and system for the Internet of Things. Background Art

[0002] With the rapid development of IoT technology, more and more IoT devices are connected to the Internet, forming a huge network. With the development of IoT, the rapidly growing IoT devices collect a large amount of data, driving the rapid development of smart medical care, smart manufacturing and smart transportation.

[0003] Existing data storage methods are usually centralized, requiring large data centers or servers to carry out data storage and processing tasks. This method has problems such as poor data security, low reliability, and high data maintenance costs.

[0004] Therefore, it is necessary to provide a big data distributed storage method and system for the Internet of Things to improve the security of big data storage in the Internet of Things. Summary of the invention

[0005] The present invention provides a big data distributed storage system for the Internet of Things, comprising: a data processing module, for receiving the Internet of Things data collected by a plurality of Internet of Things terminals and performing storage preprocessing; the data processing module comprises a computing power scheduling node and a plurality of data preprocessing nodes, the computing power scheduling node is used to allocate a data preprocessing node for the Internet of Things data, the data preprocessing node is used to perform storage preprocessing on the Internet of Things data, and to determine whether to cache the Internet of Things data; a data storage module, comprising a storage scheduling node and a plurality of data storage nodes, wherein the storage scheduling node is used to allocate a plurality of optimal data storage nodes for the preprocessed Internet of Things data, the plurality of optimal data storage nodes are used to store the preprocessed Internet of Things data; a data cache module, for a cache scheduling node and a plurality of data cache nodes, wherein the cache scheduling node is used to allocate a plurality of optimal data cache nodes for the preprocessed Internet of Things data after the data preprocessing node determines to cache the Internet of Things data, the plurality of optimal data cache nodes are used to cache the preprocessed Internet of Things data; the computing power scheduling node is also used to allocate a data preprocessing node for a data query request initiated by a user terminal, the data preprocessing node is used to retrieve target Internet of Things data corresponding to the data query request from the data storage module or the data cache module.

[0006] Furthermore, the data preprocessing node performs storage preprocessing on the IoT data, including: determining whether the number of historical data uploads of the IoT terminal is greater than a preset number threshold; if the number of historical data uploads of the IoT terminal is greater than the preset number threshold, determining the first shard size corresponding to the IoT terminal based on access information of the historical IoT data of the IoT terminal; if the number of historical data uploads of the IoT terminal is less than or equal to the preset number threshold, determining the first shard size corresponding to the IoT terminal based on access information of similar IoT terminals; and segmenting the IoT data based on the first shard size.

[0007] Furthermore, the data preprocessing node determines the first shard size corresponding to the IoT terminal based on the access information of the historical IoT data of the IoT terminal, including: calculating a first access count mean and a first access count dispersion parameter based on the access count of each historical IoT data of the IoT terminal; and determining the first shard size corresponding to the IoT terminal based on the first access count mean and the first access count dispersion parameter.

[0008] Furthermore, the data preprocessing node determines the similar Internet of Things terminals, including: obtaining multiple keywords of the Internet of Things data; obtaining multiple keywords of each of the Internet of Things terminals; for each of the Internet of Things terminals, determining the similarity of the Internet of Things terminals based on the multiple keywords of the Internet of Things data and the multiple keywords of the Internet of Things terminals, and determining whether the Internet of Things terminals are similar Internet of Things terminals based on the similarity of the Internet of Things terminals.

[0009] Further, the data preprocessing node determines the first shard size corresponding to the Internet of Things terminal based on the access information of the similar Internet of Things terminal, including: for each of the similar Internet of Things terminals, calculating the first shard size corresponding to the similar Internet of Things terminal; calculating the shard size dispersion parameter based on the first shard size corresponding to each of the similar Internet of Things terminals; when the shard size dispersion parameter is less than a preset shard size dispersion parameter threshold, calculating the mean of the first shard sizes corresponding to the similar Internet of Things terminals as the first shard size corresponding to the Internet of Things terminal; when the shard size dispersion parameter is greater than or equal to the preset shard size dispersion parameter threshold, determining the first shard size corresponding to the Internet of Things terminal based on the mean of the first shard sizes corresponding to the similar Internet of Things terminals and the shard size dispersion parameter according to the adjustment rule.

[0010] Furthermore, the data preprocessing node determines whether to cache the IoT data, including: determining whether the number of historical data uploads of the IoT terminal is greater than a preset number threshold; if the number of historical data uploads of the IoT terminal is greater than the preset number threshold, based on the access information of the historical IoT data of the IoT terminal, determining whether to cache the IoT data; if the number of historical data uploads of the IoT terminal is less than or equal to the preset number threshold, based on the access information of similar IoT terminals, determining whether to cache the IoT data.

[0011] Furthermore, the data preprocessing node determines whether to cache the IoT data based on the access information of the historical IoT data of the IoT terminal, including: for each copy of the historical IoT data, determining a first access time interval mean and a first access time interval dispersion parameter based on the time interval between two adjacent accesses to the historical IoT data; determining a second access time interval mean and a second access time interval dispersion parameter based on the first access time interval mean and the first access time interval dispersion parameter corresponding to each copy of the historical IoT data; and determining whether to cache the IoT data based on the second access time interval mean and the second access time interval dispersion parameter.

[0012] Furthermore, the data preprocessing node is also used to: based on the access information of the historical IoT data of the IoT terminal, determine to cache the IoT data, and determine the second data slice size and cache time based on the second access time interval mean and the second access time interval discreteness parameter; split the IoT data based on the second data slice size; the optimal data cache node is also used to store the IoT data split based on the second data slice size based on the cache time.

[0013] Furthermore, the cache scheduling node allocates multiple optimal data cache nodes to the preprocessed IoT data, including: for each of the data cache nodes, based on the cache time of the IoT data stored in the data cache node, predicting the load of the data cache node at multiple future time points, and calculating the load mean and load discrete parameters of the data cache node; based on the load mean and load discrete parameters of each of the data cache nodes, allocating multiple optimal data cache nodes to the preprocessed IoT data.

[0014] The present invention provides a big data distributed storage method for the Internet of Things, comprising: receiving Internet of Things data collected by multiple Internet of Things terminals and performing storage preprocessing; a computing power scheduling node assigning a data preprocessing node to the Internet of Things data; the data preprocessing node performing storage preprocessing on the Internet of Things data and judging whether to cache the Internet of Things data; a storage scheduling node assigning a plurality of optimal data storage nodes to the preprocessed Internet of Things data; the plurality of optimal data storage nodes storing the preprocessed Internet of Things data; a cache scheduling node assigning a plurality of optimal data cache nodes to the preprocessed Internet of Things data after the data preprocessing node determines to cache the Internet of Things data; the plurality of optimal data cache nodes cache the preprocessed Internet of Things data; the computing power scheduling node assigning a data preprocessing node to a data query request initiated by a user terminal; the data preprocessing node calling the target Internet of Things data corresponding to the data query request from the data storage module or the data cache module.

[0015] Compared with the prior art, the present invention provides a big data distributed storage method and system for the Internet of Things, which has at least the following beneficial effects:

[0016] 1. By collecting IoT data from IoT terminals and performing storage preprocessing, a more intelligent data sharding strategy is used to ensure uniform distribution and efficient access of data among multiple data storage nodes. Multiple data preprocessing nodes are scheduled to optimize the concurrent access mechanism to ensure system stability and performance in high-concurrency scenarios. Automatic load balancing technology is used to evenly distribute data to each node to avoid single-point bottlenecks.

[0017] 2. Improve data access speed: Predictive prefetching can ensure that data is already in the cache when it is accessed, thereby reducing data access latency. Reduce system load: By loading data into the cache in advance, the access frequency of the data storage module can be reduced and data interaction efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0019] Figure 1 It is a module schematic diagram of a big data distributed storage system for the Internet of Things according to some embodiments of this specification;

[0020] Figure 2 It is a flow chart of a big data distributed storage method for the Internet of Things according to some embodiments of this specification. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0022] Figure 1 is a module schematic diagram of a big data distributed storage system for the Internet of Things according to some embodiments of this specification, such as Figure 1 As shown, a big data distributed storage system for the Internet of Things may include a data processing module, a data storage module and a data cache module.

[0023] The data processing module can be used to receive IoT data collected by multiple IoT terminals and perform storage preprocessing.

[0024] IoT terminals can collect various IoT data such as sound, light, heat, electricity, mechanics, chemistry, biology, and location through various devices and technologies such as information sensors, radio frequency identification technology, global positioning system, infrared sensors, laser scanners, etc.

[0025] The data processing module includes a computing power scheduling node and multiple data preprocessing nodes. The computing power scheduling node is used to allocate data preprocessing nodes to IoT data. The data preprocessing nodes are used to store and preprocess IoT data and determine whether to cache IoT data.

[0026] In some embodiments, the data preprocessing node performs storage preprocessing on the IoT data, including:

[0027] Determine whether the number of historical data uploads of the IoT terminal is greater than a preset number threshold;

[0028] If the number of times the IoT terminal uploads historical data is greater than a preset number threshold, determining a first shard size corresponding to the IoT terminal based on access information of the IoT terminal's historical IoT data;

[0029] If the number of historical data uploads of the IoT terminal is less than or equal to a preset number threshold, based on access information of similar IoT terminals, determine a first shard size corresponding to the IoT terminal;

[0030] The IoT data is segmented and processed based on the first shard size.

[0031] In some embodiments, the data preprocessing node determines the first shard size corresponding to the IoT terminal based on the access information of the historical IoT data of the IoT terminal, including:

[0032] Based on the number of accesses to each piece of historical IoT data of the IoT terminal, a first access number mean and a first access number dispersion parameter are calculated;

[0033] Based on the first access number mean and the first access number dispersion parameter, a first shard size corresponding to the Internet of Things terminal is determined.

[0034] Specifically, the IoT data uploaded by the IoT terminal is retrieved once by the user terminal, which indicates that the IoT data is accessed once.

[0035] The first access times mean and the first access times dispersion parameter can be calculated based on the following formula:

[0036]

[0037] Among them, V (i,mean) is the average number of first visits corresponding to the i-th IoT terminal, V (i,j) is the number of accesses to the jth IoT data uploaded by the i-th IoT terminal, J is the total number of IoT data uploaded by the i-th IoT terminal, and W (i,1) is the first access count dispersion parameter corresponding to the i-th IoT terminal.

[0038] The size of the first shard corresponding to the IoT terminal can be calculated according to the following formula:

[0039]

[0040] Among them, S (i,1) is the first shard size corresponding to the i-th IoT terminal, a1 and a2 are preset weights, a1 and a2 are both greater than 0, and a1+a2=1, V0 is the preset first access count average, W (0,1) is the preset first access count discreteness parameter, and S0 is the preset shard size.

[0041] In some embodiments, the data pre-processing node determines similar IoT terminals, including:

[0042] Acquire multiple keywords of IoT data, wherein the keywords can be used to characterize the data content recorded in the IoT data, for example, the keywords can be "temperature", "humidity", "location", etc., and the multiple keywords of IoT data can be extracted by a keyword extraction model, and the keyword extraction model can be a convolutional neural network (CNN) model;

[0043] Acquire multiple keywords of each IoT terminal. Specifically, the multiple keywords may be extracted from historical IoT data of the IoT terminal.

[0044] For each IoT terminal, the similarity of the IoT terminals is determined based on multiple keywords of the IoT data and multiple keywords of the IoT terminals, and based on the similarity of the IoT terminals, it is determined whether the IoT terminals are similar IoT terminals.

[0045] Specifically, the similarity of IoT terminals can be determined based on multiple keywords of IoT data and multiple keywords of IoT terminals through a similarity determination model, and IoT terminals with similarities greater than a preset similarity threshold are regarded as similar IoT terminals, wherein the similarity determination model can be a convolutional neural network (CNN) model.

[0046] In some embodiments, the data preprocessing node determines the first shard size corresponding to the IoT terminal based on the access information of similar IoT terminals, including:

[0047] For each similar IoT terminal, calculate the first shard size corresponding to the similar IoT terminal;

[0048] Calculate a fragment size dispersion parameter based on the first fragment size corresponding to each similar IoT terminal;

[0049] When the fragment size dispersion parameter is less than a preset fragment size dispersion parameter threshold, calculating the average of the first fragment sizes corresponding to similar IoT terminals as the first fragment size corresponding to the IoT terminal;

[0050] When the shard size dispersion parameter is greater than or equal to a preset shard size dispersion parameter threshold, the first shard size corresponding to the IoT terminal is determined according to the adjustment rule based on the mean of the first shard sizes corresponding to similar IoT terminals and the shard size dispersion parameter.

[0051] Specifically, the first shard size corresponding to the similar Internet of Things terminal may be calculated according to the mean value of the first access times of the similar Internet of Things terminals and the first access times dispersion parameter.

[0052] The shard size dispersion parameter can be calculated based on the following formula:

[0053]

[0054] Among them, W1 is the fragment size discreteness parameter, S k is the first shard size corresponding to the kth similar IoT terminal, and K is the total number of similar IoT terminals.

[0055] The size of the first shard corresponding to the IoT terminal can be calculated according to the following adjustment rules:

[0056]

[0057] Among them, W0 is the preset fragment size discreteness parameter, S mean is the average size of the first fragment corresponding to similar IoT terminals.

[0058] In some embodiments, the data preprocessing node determines whether to cache the IoT data, including:

[0059] Determine whether the number of historical data uploads of the IoT terminal is greater than a preset number threshold;

[0060] If the number of times the IoT terminal uploads historical data is greater than a preset number threshold, based on the access information of the IoT terminal's historical IoT data, determining whether to cache the IoT data;

[0061] If the number of historical data uploads of the IoT terminal is less than or equal to the preset number threshold, it is determined whether to cache the IoT data based on the access information of similar IoT terminals.

[0062] In some embodiments, the data preprocessing node determines whether to cache the IoT data based on the access information of the historical IoT data of the IoT terminal, including:

[0063] For each piece of historical IoT data, based on the time intervals between two adjacent accesses to the historical IoT data, determine a first access time interval mean and a first access time interval dispersion parameter;

[0064] Determine a second access time interval mean and a second access time interval dispersion parameter based on a first access time interval mean and a first access time interval dispersion parameter corresponding to each piece of historical IoT data;

[0065] Based on the second access time interval mean and the second access time interval dispersion parameter, it is determined whether to cache the Internet of Things data.

[0066] Specifically, the first access time interval mean and the first access time interval dispersion parameter may be calculated based on the following formula:

[0067]

[0068] Among them, △T ((i,j),mean) is the mean of the first access time interval corresponding to the jth IoT data of the i-th IoT terminal, △T ((i,j),n,n+1)is the time interval between the nth access and the n+1th access of the jth IoT data uploaded by the ith IoT terminal, N is the total number of accesses to the jth IoT data uploaded by the ith IoT terminal, and W (i,2) is the first access time interval discreteness parameter corresponding to the j-th piece of IoT data of the i-th IoT terminal.

[0069] The second access time interval mean value can be obtained by averaging the first access time interval mean value corresponding to each piece of historical Internet of Things data, and the second access time interval dispersion parameter can be obtained by averaging the first access time interval dispersion parameter corresponding to each piece of historical Internet of Things data.

[0070] When the second access time interval mean is less than the preset access time interval mean threshold and the second access time interval dispersion parameter is less than the preset access time interval dispersion parameter, it is determined to cache the Internet of Things data.

[0071] In some embodiments, the data pre-processing node is further configured to:

[0072] After determining to cache the IoT data based on the access information of the historical IoT data of the IoT terminal, determine the second data slice size and cache time based on the second access time interval mean and the second access time interval dispersion parameter;

[0073] Slice the IoT data based on the second data slice size;

[0074] The optimal data cache node is also used to store the IoT data after being segmented and processed based on the second data slice size based on the cache time.

[0075] Specifically, the second data slice size and cache time may be calculated according to the following formula:

[0076]

[0077] Among them, S (i,2) is the second shard size corresponding to the i-th IoT terminal, a3 and a4 are preset weights, a3 and a4 are both greater than 0, and a3+a4=1, △T0 is the preset second access time interval mean, W (0,2) The second access time interval discreteness parameter is preset.

[0078]

[0079] Among them, T i is the cache time, and T0 is the preset cache time.

[0080] In some embodiments, the data preprocessing node determines whether to cache the IoT data based on the access information of similar IoT terminals, including:

[0081] The proportion of similar IoT terminals that need to cache data among similar IoT terminals can be determined, and when the proportion is greater than a preset proportion threshold, it is determined that the IoT data is cached;

[0082] The second data slice size corresponding to each similar Internet of Things terminal is used as the second data slice size of the current Internet of Things terminal, and the cache time corresponding to each similar Internet of Things terminal is used as the cache time of the current Internet of Things terminal.

[0083] The data storage module may include a storage scheduling node and multiple data storage nodes, wherein the storage scheduling node is used to allocate multiple optimal data storage nodes to the preprocessed Internet of Things data, and the multiple optimal data storage nodes are used to store the preprocessed Internet of Things data.

[0084] Specifically, the storage scheduling node can evenly distribute data to each data storage node through an automatic load balancing algorithm to avoid single-point bottlenecks.

[0085] The data cache module can be used for a cache scheduling node and multiple data cache nodes, wherein the cache scheduling node is used to allocate multiple optimal data cache nodes to the preprocessed IoT data after the data preprocessing node determines to cache the IoT data, and the multiple optimal data cache nodes are used to cache the preprocessed IoT data.

[0086] In some embodiments, the cache scheduling node allocates multiple optimal data cache nodes for the pre-processed IoT data, including:

[0087] For each data cache node, based on the cache time of IoT data stored in the data cache node, the load of the data cache node at multiple future time points is predicted, and the load mean and load discrete parameters of the data cache node are calculated;

[0088] Based on the load mean and load discrete parameters of each data cache node, multiple optimal data cache nodes are allocated for the preprocessed IoT data.

[0089] Specifically, the load prediction model can be used to predict the load of the data cache node at multiple future time points based on the cache time of the IoT data stored in the data cache node, wherein the load prediction model can be a long short-term memory network (LSTM, Long Short-Term Memory) model.

[0090] The load mean and load dispersion parameters of the data cache node can be calculated according to the following formula:

[0091]

[0092] Among them, L (g,mean) is the load average of the g-th data cache node, L (g,t) is the load of the g-th data cache node at the t-th future time point, T is the total number of future time points, and W g is the load discrete parameter of the g-th data cache node.

[0093] The matching score corresponding to the data cache node can be calculated according to the load mean and load discrete parameters of the data cache node, and multiple optimal data cache nodes can be allocated to the preprocessed IoT data according to the matching scores from large to small.

[0094] For example, the matching score corresponding to the data cache node can be calculated according to the following formula:

[0095]

[0096] Among them, M g is the matching score of the g-th data cache node, a5 and a6 are preset weights, a a and a6 are both greater than 0, and a5+a6=1, R1 and R2 are preset weights, R1 and R2 are both greater than 0, and L0 is the preset load average.

[0097] The computing power scheduling node is also used to allocate data preprocessing nodes for data query requests initiated by user terminals. The data preprocessing nodes are used to retrieve target IoT data corresponding to the data query requests from the data storage module or the data cache module.

[0098] Specifically, for target IoT data that is not cached, the target IoT data can be read from multiple data storage nodes for storing the target IoT data. For cached target IoT data, the target IoT data can be read from multiple optimal data cache nodes for caching the target IoT data.

[0099] Figure 2 is a flow chart of a method for distributed storage of big data for the Internet of Things according to some embodiments of this specification, such as Figure 2 As shown, a big data distributed storage method for the Internet of Things may include the following process.

[0100] Receive IoT data collected by multiple IoT terminals and perform storage preprocessing;

[0101] The computing power scheduling node allocates data preprocessing nodes for IoT data;

[0102] The data preprocessing node performs storage preprocessing on IoT data and determines whether to cache the IoT data;

[0103] The storage scheduling node allocates multiple optimal data storage nodes for the pre-processed IoT data;

[0104] Multiple optimal data storage nodes store pre-processed IoT data;

[0105] After the data preprocessing node determines to cache the IoT data, the cache scheduling node allocates multiple optimal data cache nodes for the preprocessed IoT data;

[0106] Multiple optimal data caching nodes cache pre-processed IoT data;

[0107] The computing power scheduling node allocates data preprocessing nodes for data query requests initiated by user terminals;

[0108] The data preprocessing node retrieves the target IoT data corresponding to the data query request from the data storage module or the data cache module.

[0109] A big data distributed storage method for the Internet of Things can be executed by a big data distributed storage system for the Internet of Things. For more descriptions of a big data distributed storage method for the Internet of Things, please refer to the relevant description of a big data distributed storage system for the Internet of Things, which will not be repeated here.

[0110] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A big data distributed storage system for the Internet of Things, characterized in that: include: A data processing module is used to receive IoT data collected by multiple IoT terminals and perform storage preprocessing; The data processing module includes a computing power scheduling node and a plurality of data preprocessing nodes, wherein the computing power scheduling node is used to allocate a data preprocessing node to the IoT data, and the data preprocessing node is used to perform storage preprocessing on the IoT data and determine whether to cache the IoT data; A data storage module, comprising a storage scheduling node and a plurality of data storage nodes, wherein the storage scheduling node is used to allocate a plurality of optimal data storage nodes for the preprocessed IoT data, and the plurality of optimal data storage nodes are used to store the preprocessed IoT data; A data cache module, used for caching a scheduling node and a plurality of data cache nodes, wherein the cache scheduling node is used to allocate a plurality of optimal data cache nodes for the preprocessed IoT data after the data preprocessing node determines to cache the IoT data, and the plurality of optimal data cache nodes are used to cache the preprocessed IoT data; The computing power scheduling node is also used to allocate a data preprocessing node for a data query request initiated by a user terminal, and the data preprocessing node is used to retrieve the target Internet of Things data corresponding to the data query request from the data storage module or the data cache module; The data preprocessing node performs storage preprocessing on the IoT data, including: Determine whether the number of historical data uploads of the IoT terminal is greater than a preset number threshold; If the number of times the IoT terminal uploads historical data is greater than a preset number threshold, determining a first shard size corresponding to the IoT terminal based on access information of the IoT terminal's historical IoT data; If the number of historical data uploads of the IoT terminal is less than or equal to a preset number threshold, based on access information of similar IoT terminals, determine a first shard size corresponding to the IoT terminal; Slice the IoT data based on the first shard size; The data preprocessing node determines the first shard size corresponding to the IoT terminal based on the access information of the historical IoT data of the IoT terminal, including: Calculate a first access count mean and a first access count dispersion parameter based on the access count of each piece of historical IoT data of the IoT terminal; Determine a first shard size corresponding to the Internet of Things terminal based on the first access number mean and the first access number dispersion parameter; The data preprocessing node determines the similar Internet of Things terminal, including: Obtain multiple keywords of the IoT data; Acquire multiple keywords of each of the IoT terminals, specifically, extract multiple keywords from historical IoT data of the IoT terminal; For each of the IoT terminals, based on a plurality of keywords of the IoT data and a plurality of keywords of the IoT terminals, determining the similarity of the IoT terminals, and based on the similarity of the IoT terminals, determining whether the IoT terminals are similar IoT terminals; The data preprocessing node determines the first fragment size corresponding to the IoT terminal based on the access information of the similar IoT terminal, including: For each of the similar Internet of Things terminals, calculating a first fragment size corresponding to the similar Internet of Things terminal; Calculate a fragment size dispersion parameter based on the first fragment size corresponding to each of the similar Internet of Things terminals; When the fragment size dispersion parameter is less than a preset fragment size dispersion parameter threshold, calculating the average of the first fragment sizes corresponding to the similar Internet of Things terminals as the first fragment size corresponding to the Internet of Things terminal; When the shard size dispersion parameter is greater than or equal to a preset shard size dispersion parameter threshold, according to the adjustment rule, based on the mean of the first shard sizes corresponding to the similar IoT terminals and the shard size dispersion parameter, the first shard size corresponding to the IoT terminal is determined. Specifically, the first shard size corresponding to the similar IoT terminal is calculated by the mean of the first access times of the similar IoT terminals and the first access times dispersion parameter; The data preprocessing node determines whether to cache the IoT data, including: Determine whether the number of historical data uploads of the IoT terminal is greater than a preset number threshold; If the number of times the IoT terminal uploads historical data is greater than a preset number threshold, based on the access information of the IoT terminal's historical IoT data, determining whether to cache the IoT data; If the number of historical data uploads of the IoT terminal is less than or equal to a preset number threshold, based on access information of similar IoT terminals, determining whether to cache the IoT data; the data preprocessing node determining whether to cache the IoT data based on the access information of the historical IoT data of the IoT terminal, including: For each piece of the historical IoT data, determining a first access time interval mean and a first access time interval dispersion parameter based on a time interval between two adjacent accesses to the historical IoT data; Determine a second access time interval mean and a second access time interval dispersion parameter based on a first access time interval mean and a first access time interval dispersion parameter corresponding to each piece of the historical Internet of Things data; Based on the second access time interval mean and the second access time interval dispersion parameter, it is determined whether to cache the Internet of Things data.

2. A big data distributed storage system for the Internet of Things according to claim 1, characterized in that: The data preprocessing node is also used for: After determining to cache the IoT data based on the access information of the historical IoT data of the IoT terminal, determine the second data slice size and cache time based on the second access time interval mean and the second access time interval dispersion parameter; Slice the IoT data based on the second data slice size; The optimal data cache node is also used to store the Internet of Things data after being segmented and processed based on the second data slice size based on the cache time.

3. A big data distributed storage system for the Internet of Things according to claim 2, characterized in that: The cache scheduling node allocates multiple optimal data cache nodes for the pre-processed IoT data, including: For each of the data cache nodes, based on the cache time of the IoT data stored in the data cache node, predict the load of the data cache node at multiple future time points, and calculate the load mean and load discrete parameters of the data cache node; Based on the load mean and load discrete parameters of each of the data cache nodes, multiple optimal data cache nodes are allocated to the preprocessed Internet of Things data.

4. A big data distributed storage method for the Internet of Things, characterized in that: A big data distributed storage system for the Internet of Things as described in any one of claims 1 to 3, comprising: Receive IoT data collected by multiple IoT terminals and perform storage preprocessing; The computing power scheduling node allocates a data preprocessing node for the IoT data; The data preprocessing node performs storage preprocessing on the IoT data, and determines whether to cache the IoT data; The storage scheduling node allocates multiple optimal data storage nodes for the pre-processed IoT data; The multiple optimal data storage nodes store the pre-processed IoT data; After the data preprocessing node determines to cache the IoT data, the cache scheduling node allocates a plurality of optimal data cache nodes for the preprocessed IoT data; The multiple optimal data caching nodes cache the pre-processed IoT data; The computing power scheduling node allocates a data preprocessing node for a data query request initiated by a user terminal; The data preprocessing node retrieves the target IoT data corresponding to the data query request from the data storage module or the data cache module.

Images