An IoT cloud data storage management system and method based on measurement model

Abstract

The present invention relates to the field of data storage technology, and specifically to an IoT cloud data storage management system and method based on a measurement model. The management method comprises the following steps: obtaining the usage of historical cleared data and establishing a data clearing probability estimation model; obtaining the usage of each historical data and determining whether to perform a clearing mark; classifying valid data to obtain a plurality of valid data sets, and obtaining a retrieval order for each valid data set during the next data retrieval; analyzing the storage time interval between each valid data set in the valid data set to obtain the single transmission data volume of the valid data set, and allocating storage capacity to the valid data set; if new data is currently transmitted, analyzing the usage of each valid data set in the valid data set, and adjusting the storage capacity allocated to the valid data set according to the addition of clearing marks for the valid data and the actual data volume currently allocated.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to an IoT cloud data storage management system and method based on a measurement model. Background Art

[0002] The literal translation of iot is "Internet of Things", and the accurate translation should be "Internet of Things"; it refers to a huge network formed by combining various information sensing devices, such as radio frequency identification devices, infrared sensors, global positioning systems, laser scanners and other devices with the Internet; its purpose is to connect all objects to the network for easy identification and management.

[0003] With the increase in the number of IoT devices and the expansion of data scale, the management of IoT data will bring huge costs; the data obtained by IoT devices are generally time-series data, which may no longer be used after a period of time. Failure to clean up these useless data in a timely manner will increase the burden of data storage management; in the unified management of data, if the data space cannot be reasonably allocated, the data storage will be very chaotic. With the input of more and more IoT data, the data storage will become more and more chaotic, seriously increasing the data management burden and reducing the data storage efficiency and processing efficiency. Summary of the Invention

[0004] The purpose of the present invention is to provide an IoT cloud data storage management system and method based on a measurement model to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: an IoT cloud data storage management method based on a measurement model, the management method comprising the following steps:

[0006] Step S100: monitoring the usage of each piece of historical cleansed data during the storage process, building a data cleansing probability estimation model, and training the data cleansing probability estimation model;

[0007] Step S200: Obtain the usage of each historical data item, and determine whether to mark it for purge using a data purge probability estimation model; define data items without purge marks as valid data, classify the valid data items, and obtain a number of valid data sets; and determine the retrieval order of each valid data item in the valid data set during the next data retrieval based on the usage of each valid data item in the valid data set;

[0008] Step S300: Obtain the total amount of data in each valid data set, analyze the storage time intervals between each valid data in the valid data set, divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set;

[0009] Step S400: If new data is currently being transmitted, analyze the usage of each valid data in the valid data set and add clear marks to some valid data; based on the addition of clear marks for the valid data and the actual amount of data currently allocated, obtain the remaining storage capacity of the valid data set and adjust the storage capacity allocated to the valid data set.

[0010] Furthermore, step S100 includes the following steps:

[0011] Step S101: Obtain the storage time point and cleaning time point of the i-th data in all the historical cleaned data, and obtain the usage time of the i-th data as Δt i ; Get the usage time Δt i The number of times the i-th data is used is u i , the average usage frequency of the i-th data is f i =u i / Δt i The average usage frequency can reveal the overall usage of the data. Combined with the recent usage frequency, if the data has not been used for a long time, it can be preliminarily analyzed to determine whether the data is due to low usage frequency or invalid data. This can also provide a reference for the subsequent calculation of the removal probability.

[0012] Step S102: Set the i-th data to be cleared at the j-th data clearing, and obtain the clearing time interval ΔT between the j-1-th data clearing and the j-th data clearing j,j-1 ; Extract the number of times the i-th data is used during the cleaning time interval The most recent usage frequency of the i-th data is

[0013] Step S103: Obtain the unused time ut from the last time the i-th data is cleared i , establish the formula as the processing logic of the data clearing probability estimation model:

[0014]

[0015] Among them, a, b, c and d are constant coefficients, a<0, b<0, c>0; the cleaning probability P of the i-th data is calculated i ; Set a cleaning probability threshold ε, if P i ≥ε, then clear the mark of the i-th data;

[0016] Step S104: Select several cleaned data as the training set of the data purge probability estimation model. In the training set, the average usage frequency, recent usage frequency and non-use time of the cleaned data are used as the input data of the data purge probability estimation model, and a purge probability P is output, where P≥ε, and the constant coefficients a, b, c and d are determined; the remaining cleaned data are used as the test set. If the output results of the cleaned data do not satisfy P≥ε, the unsatisfactory cleaned data are put into the training set, and the data purge probability estimation model is retrained until the output results of all cleaned data satisfy P≥ε; set a purge probability threshold, and when the purge probability exceeds the threshold, it needs to be marked. Therefore, when the model is trained on historical cleaned data, as long as the output purge probability exceeds the threshold, it is sufficient.

[0017] Furthermore, step S200 includes the following steps:

[0018] Step S201: Set the average usage frequency of the kth data in the currently stored data to f k and the most recent frequency of use is Get the unused time ut of the kth data from the last use k , the data removal probability estimation model is used to calculate the removal probability, and the removal probability of the kth data is obtained as P k , if P k ≥ε, then the kth data is marked for clearing;

[0019] Step S202: Set the data without clear marks as valid data, summarize all valid data, obtain the data source of each valid data, classify the valid data according to the data source, and generate several valid data sets;

[0020] Step S203: Randomly select a valid data set and obtain the average usage frequency of the k′th valid data in the valid data set as f k′ and the most recent frequency of use is According to the formula:

[0021]

[0022] Among them, P k′ is the probability of clearing the k′th valid data in the valid data set, and the retrieval priority Y of the k′th valid data in the valid data set is calculated. k′; Obtain the estimated retrieval priority of all valid data in a valid data set, sort the valid data from large to small according to the numerical value, and use it as the retrieval order for the next data retrieval; Based on the frequency of data use, it is estimated whether the data will be used in the next data retrieval. Therefore, the estimated retrieval priority of the data can be calculated, and the data with high priority can be placed at the front of the retrieval position. In this way, when the user retrieves data, the time to retrieve the target data will be shortened, thereby improving the retrieval efficiency;

[0023] Furthermore, step S300 includes the following steps:

[0024] Step S301: Obtain the data volume D of the k′th valid data in the Lth valid data set k′ , the total amount of data in the Lth valid data set is Q L ;

[0025] Step S302: Obtain the storage time point of each valid data in the Lth valid data set, and sort the valid data in the Lth valid data set according to the order of the storage time points; obtain the time interval between two adjacent valid time points as Δt′, set a time interval threshold τ, if Δt′>τ, then divide the two valid time points into two time periods, if Δt′≤τ, then divide the two valid time points into one time period; divide all valid time points to obtain several time periods of the Lth valid data set, and set the number of valid time points in the eth time period as n e , according to the formula:

[0026]

[0027] Where r is the number of time periods divided in the Lth valid data set; the single transmission data volume of the Lth valid data set is calculated as The storage capacity allocated to the Lth valid data set is

[0028] Furthermore, step S400 includes the following steps:

[0029] Step S401: Distribute the currently transmitted data according to the data source, and set the actual data volume distributed to the Lth valid data set as Q' L ; Get the average usage frequency f of the k′th valid data in the Lth valid data set k′ , recent usage frequency The unused time since the last use k′ , put it into the data clearing probability estimation model for calculation, and get the clearing probability P of the k′th valid data k′ ; Set the cleaning probability threshold to ε, if Pk′ >ε, then add a clear mark to the k′th valid data;

[0030] Step S402: Set the number of valid data with clear marks in the current Lth valid data set to U L , the amount of valid data with clear marks is Q″ L =U L ×x L , where x L is the amount of data of a single data in the Lth valid data set; the total amount of data originally obtained from the Lth valid data set is Q L , adjust the total amount of new data to Q′ L =Q L +Q′ L -Q″ L ;

[0031] Step S403: Obtain the single transmission data volume of the Lth valid data set and the allocated storage capacity S L ,like Then send a reminder of insufficient storage capacity, and the storage capacity to be supplemented is Get all the remaining valid data sets whose storage capacity is less than 0. The remaining storage capacity of the L′th valid data set is Q′ L′ is the total amount of new data in the L′th valid data set, is the single transmission data volume of the L′th valid data set, S L′ The storage capacity allocated for the L′th valid data set; if Then the remaining storage capacity of the L′th valid data set is added to the Lth valid data set;

[0032] When the storage capacity of a valid data set is insufficient, priority is given to the remaining storage capacity of other valid data sets rather than the storage capacity provided by the system, so that the storage capacity of the data is reduced as much as possible, the scope of data management is reduced, and the management burden is reduced.

[0033] In order to better implement the above method, an IoT cloud data storage management system based on a measurement model is also proposed. The management system includes a clean data analysis module, a historical data cleanup module, a storage capacity allocation module, and a storage capacity adjustment module.

[0034] The cleansing data analysis module is used to monitor the usage of each cleansed data in the storage process, build a data cleansing probability estimation model, and train the data cleansing probability estimation model;

[0035] The historical data cleaning module is used to obtain the usage of each historical data and determine whether to mark it for clearing through the data clearing probability estimation model; the data without clearing marks is set as valid data, and the valid data is classified to obtain several valid data sets; based on the usage of each valid data in the valid data set, the retrieval order of each valid data in the valid data set is obtained in the next data retrieval;

[0036] The storage capacity allocation module is used to obtain the total amount of data in each valid data set, analyze the storage time interval between each valid data in the valid data set, divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set;

[0037] The storage capacity adjustment module is used to analyze the usage of each valid data in the valid data set if new data is currently transmitted, and add clearance marks to several valid data; based on the addition of clearance marks for valid data and the actual amount of data currently allocated, it obtains the remaining storage capacity of the valid data set and adjusts the storage capacity allocated to the valid data set.

[0038] Furthermore, the data cleaning module includes a data usage analysis unit and a model training unit;

[0039] The data usage analysis unit is used to monitor the usage of each historical cleansed data during the storage process; the model training unit is used to build a data cleansing probability estimation model and train the data cleansing probability estimation model.

[0040] Furthermore, the historical data cleaning module includes a clearing judgment unit and a valid data storage unit;

[0041] The clearing judgment unit is used to obtain the usage of each historical data and determine whether to mark it for clearing through the data clearing probability estimation model; the valid data storage unit is used to set the data without clearing mark as valid data, classify the valid data, and obtain several valid data sets; according to the usage of each valid data in the valid data set, the retrieval order of each valid data in the valid data set during the next data retrieval is obtained.

[0042] Furthermore, the storage capacity allocation module includes a storage time analysis unit and a capacity requirement calculation unit;

[0043] The storage time analysis unit is used to obtain the total amount of data in each valid data set and analyze the storage time interval between each valid data in the valid data set; the capacity requirement calculation unit is used to divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set.

[0044] Furthermore, the storage capacity adjustment module includes a clear mark adding unit and a remaining capacity analysis unit;

[0045] The clear mark adding unit is used to analyze the usage of each valid data in the valid data set and add clear marks to some valid data if new data is currently transmitted; the remaining capacity analysis unit is used to obtain the remaining storage capacity of the valid data set based on the clear mark addition status of the valid data and the actual amount of data currently allocated, and adjust the storage capacity allocated to the valid data set.

[0046] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention calculates the probability of clearing each data by establishing a data clearing probability estimation model, automatically clears some data that has not been used for a long time, and eliminates the burden of useless data on data storage and management; (2) The present invention divides the valid data and arranges the retrieval order according to each data in each category, giving priority to retrieving the data with higher frequency of use, which can help users to retrieve the target data in advance to a certain extent and improve the retrieval efficiency; (3) The present invention preliminarily allocates storage space to the valid data set, and then adjusts the subsequent storage space allocation according to the transmission and clearing of subsequent data, and gives priority to adjusting the remaining storage space of each data set, so that the data management capacity is as fixed as possible, which is convenient for data management and reduces the problem of excessive management pressure caused by excessive data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 It is a schematic diagram of the steps of an IoT cloud data storage management method based on a measurement model;

[0049] Figure 2 It is a structural diagram of an IoT cloud data storage management system based on a measurement model. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] See also Figures 1 to 2The present invention provides a technical solution: an IoT cloud data storage management method based on a measurement model, the management method comprising the following steps:

[0052] Step S100: monitoring the usage of each piece of historical cleansed data during the storage process, building a data cleansing probability estimation model, and training the data cleansing probability estimation model;

[0053] Wherein, step S100 includes the following steps:

[0054] Step S101: Obtain the storage time point and cleaning time point of the i-th data in all the historical cleaned data, and obtain the usage time of the i-th data as Δt i ; Get the usage time Δt i The number of times the i-th data is used is u i , the average usage frequency of the i-th data is f i =u i / Δt i ;

[0055] Step S102: Set the i-th data to be cleared at the j-th data clearing, and obtain the clearing time interval ΔT between the j-1-th data clearing and the j-th data clearing j,j-1 ; Extract the number of times the i-th data is used during the cleaning time interval The most recent usage frequency of the i-th data is

[0056] Step S103: Obtain the unused time ut from the last time the i-th data is cleared i , establish the formula as the processing logic of the data clearing probability estimation model:

[0057]

[0058] Among them, a, b, c and d are constant coefficients, a<0, b<0, c>0; the cleaning probability P of the i-th data is calculated i ; Set a cleaning probability threshold ε, if P i ≥ε, then clear the mark of the i-th data;

[0059] Step S104: Select several cleaned data as the training set of the data purge probability estimation model. In the training set, the average usage frequency, recent usage frequency and non-use time of the cleaned data are used as the input data of the data purge probability estimation model, and a purge probability P is output, where P≥ε, and the constant coefficients a, b, c and d are determined; the remaining cleaned data are used as the test set. If the output results of the cleaned data do not satisfy P≥ε, the unsatisfactory cleaned data are put into the training set, and the data purge probability estimation model is retrained until the output results of all cleaned data satisfy P≥ε.

[0060] Step S200: Obtain the usage of each historical data item, and determine whether to mark it for purge using a data purge probability estimation model; define data items without purge marks as valid data, classify the valid data items, and obtain a number of valid data sets; and determine the retrieval order of each valid data item in the valid data set during the next data retrieval based on the usage of each valid data item in the valid data set;

[0061] Wherein, step S200 includes the following steps:

[0062] Step S201: Set the average usage frequency of the kth data in the currently stored data to f k and the most recent frequency of use is Get the unused time ut of the kth data from the last use k , the data removal probability estimation model is used to calculate the removal probability, and the removal probability of the kth data is obtained as P k , if P k ≥ε, then the kth data is marked for clearing;

[0063] Step S202: Set the data without clear marks as valid data, summarize all valid data, obtain the data source of each valid data, classify the valid data according to the data source, and generate several valid data sets;

[0064] Step S203: Randomly select a valid data set and obtain the average usage frequency of the k′th valid data in the valid data set as f k′ and the most recent frequency of use is According to the formula:

[0065]

[0066] Among them, P k′ is the probability of clearing the k′th valid data in the valid data set, and the retrieval priority Y of the k′th valid data in the valid data set is calculated. k′; Get the estimated retrieval priority of all valid data in a valid data set, sort the valid data from large to small according to the value, and use it as the retrieval order for the next data retrieval.

[0067] Step S300: Obtain the total amount of data in each valid data set, analyze the storage time intervals between each valid data in the valid data set, divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set;

[0068] Wherein, step S300 includes the following steps:

[0069] Step S301: Obtain the data volume D of the k′th valid data in the Lth valid data set k′ , the total amount of data in the Lth valid data set is Q L ;

[0070] Step S302: Obtain the storage time point of each valid data in the Lth valid data set, and sort the valid data in the Lth valid data set according to the order of the storage time points; obtain the time interval between two adjacent valid time points as Δt′, set a time interval threshold τ, if Δt′>τ, then divide the two valid time points into two time periods, if Δt′≤τ, then divide the two valid time points into one time period; divide all valid time points to obtain several time periods of the Lth valid data set, and set the number of valid time points in the eth time period as n e , according to the formula:

[0071]

[0072] Where r is the number of time periods divided in the Lth valid data set; the single transmission data volume of the Lth valid data set is calculated as The storage capacity allocated to the Lth valid data set is

[0073] Step S400: If new data is currently being transmitted, the usage of each valid data in the valid data set is analyzed, and clear marks are added to some valid data. Based on the clear mark addition status of the valid data and the actual amount of data currently allocated, the remaining storage capacity of the valid data set is obtained, and the storage capacity allocated to the valid data set is adjusted.

[0074] Step S400 includes the following steps:

[0075] Step S401: Distribute the currently transmitted data according to the data source, and set the actual data volume distributed to the Lth valid data set as Q' L; Get the average usage frequency f of the k′th valid data in the Lth valid data set k′ , recent usage frequency The unused time since the last use l′ , put it into the data clearing probability estimation model for calculation, and get the clearing probability P of the k′th valid data l′ ; Set the cleaning probability threshold to ε, if P k′ >ε, then add a clear mark to the k′th valid data;

[0076] Step S402: Set the number of valid data with clear marks in the current Lth valid data set to U L , the amount of valid data with clear marks is Q″ L =U L ×x L , where x L is the amount of data of a single data in the Lth valid data set; the total amount of data originally obtained from the Lth valid data set is Q L , adjust the total amount of new data to Q′ L =Q L +Q′ L -Q″ L ;

[0077] Step S403: Obtain the single transmission data volume of the Lth valid data set and the allocated storage capacity S L ,like Then send a reminder of insufficient storage capacity, and the storage capacity to be supplemented is Get all the remaining valid data sets whose storage capacity is less than 0. The remaining storage capacity of the L′th valid data set is Q′ L′ is the total amount of new data in the L′th valid data set, is the single transmission data volume of the L′th valid data set, S L′ The storage capacity allocated for the L′th valid data set; if The remaining storage capacity of the L′th valid data set is added to the Lth valid data set.

[0078] An IoT cloud data storage management system based on a measurement model, the management system includes a clean data analysis module, a historical data cleanup module, a storage capacity allocation module, and a storage capacity adjustment module;

[0079] The cleansing data analysis module is used to monitor the usage of each cleansed data in the storage process, build a data cleansing probability estimation model, and train the data cleansing probability estimation model;

[0080] The historical data cleaning module is used to obtain the usage of each historical data and determine whether to mark it for clearing through the data clearing probability estimation model; the data without clearing marks is set as valid data, and the valid data is classified to obtain several valid data sets; based on the usage of each valid data in the valid data set, the retrieval order of each valid data in the valid data set is obtained in the next data retrieval;

[0081] The storage capacity allocation module is used to obtain the total amount of data in each valid data set, analyze the storage time interval between each valid data in the valid data set, divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set;

[0082] The storage capacity adjustment module is used to analyze the usage of each valid data in the valid data set if new data is currently transmitted, and add clearance marks to several valid data; based on the addition of clearance marks for valid data and the actual amount of data currently allocated, it obtains the remaining storage capacity of the valid data set and adjusts the storage capacity allocated to the valid data set.

[0083] Among them, the data cleaning module includes a data usage analysis unit and a model training unit;

[0084] The data usage analysis unit is used to monitor the usage of each historical cleansed data during the storage process; the model training unit is used to build a data cleansing probability estimation model and train the data cleansing probability estimation model.

[0085] Among them, the historical data cleaning module includes a clearing judgment unit and a valid data storage unit;

[0086] The clearing judgment unit is used to obtain the usage of each historical data and determine whether to mark it for clearing through the data clearing probability estimation model; the valid data storage unit is used to set the data without clearing mark as valid data, classify the valid data, and obtain several valid data sets; according to the usage of each valid data in the valid data set, the retrieval order of each valid data in the valid data set during the next data retrieval is obtained.

[0087] The storage capacity allocation module includes a storage time analysis unit and a capacity demand calculation unit;

[0088] The storage time analysis unit is used to obtain the total amount of data in each valid data set and analyze the storage time interval between each valid data in the valid data set; the capacity requirement calculation unit is used to divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set.

[0089] The storage capacity adjustment module includes a clear mark adding unit and a remaining capacity analysis unit;

[0090] The clear mark adding unit is used to analyze the usage of each valid data in the valid data set and add clear marks to some valid data if new data is currently transmitted; the remaining capacity analysis unit is used to obtain the remaining storage capacity of the valid data set based on the clear mark addition status of the valid data and the actual amount of data currently allocated, and adjust the storage capacity allocated to the valid data set.

[0091] Obtain two historical data cleaning processes, select the storage time point and cleaning time point of one of the cleaned data in one of the data cleaning processes, obtain the usage time as 10 hours, the number of times used within the usage time as 10 times, the average usage frequency as 1 time / hour, the recent usage frequency of the cleaned data as 0.2 times / hour, and the unused time since the last use as 5 hours, and establish the formula: Set the cleanup probability threshold to 80%; obtain the usage of the remaining cleaned data during the data cleanup process, substitute the data into the formula, determine the values of a, b, c, and d, and generate a data cleanup probability estimation model;

[0092] Get the currently stored data, calculate the purge probability of each stored data, exclude the data with the purge mark to obtain valid data, classify the valid data according to the corresponding data source, that is, the sensor, to obtain several types of valid data sets, and arbitrarily select a valid data set. Assume that there are 3 data in total. The average usage frequency of the first data is 1, the most recent usage frequency is 0.5, and the purge probability is 0.5. The average usage frequency of the second data is 2, the most recent usage frequency is 1, and the purge probability is 0.4. The average usage frequency of the third data is 1, the most recent usage frequency is 0.2, and the purge probability is 0.6. According to the formula:

[0093]

[0094] The estimated retrieval priority of the first data is 1×2=2, the estimated retrieval priority of the second data is 1×2.5=2.5, and the estimated retrieval priority of the third data is 0.25×1 / 0.6=0.417. Sorting by numerical value yields the retrieval order for the next data retrieval. That is, when performing data retrieval, the second data is retrieved first.

[0095] Get a valid data set with a total data volume of 100 bits. Divide the valid data in the valid data set to obtain a single transmission data volume of the Lth valid data set of 100 bits. The storage capacity allocated to the Lth valid data set is S L =200 bits, that is, 200 bits of storage capacity are allocated to the valid data set;

[0096] When new data is transmitted, analysis shows that the amount of data with clear marks in the valid data set is 50 bits, the amount of new transmission is 100 bits, and the new data amount of the valid data set is 150 bits. The subsequent storage capacity required to be supplemented is 50 bits.

[0097] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0098] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An IoT cloud data storage management method based on a measurement model, characterized by: The management method comprises the following steps: Step S100: monitoring the usage of each piece of historical cleansed data during the storage process, building a data cleansing probability estimation model, and training the data cleansing probability estimation model; Step S200: Monitor the usage of each historical data item and determine whether to mark it for purge using a data purge probability estimation model; define data items without purge marks as valid data, classify the valid data items, and obtain a number of valid data sets; and determine the retrieval order of each valid data item in the next data retrieval based on the usage of each valid data item in the valid data set; Step S300: Obtain the total amount of data in each valid data set, analyze the storage time intervals between each valid data in the valid data set, divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set; Step S400: If new data is currently being transmitted, the usage of each valid data in the valid data set is analyzed, and clear marks are added to some valid data. Based on the clear mark addition status of the valid data and the actual amount of data currently allocated, the remaining storage capacity of the valid data set is obtained, and the storage capacity allocated to the valid data set is adjusted. The step S100 includes the following steps: Step S101: Obtain the storage time point and cleaning time point of the i-th data in all the historical cleaned data, and obtain the usage time of the i-th data as Δt i ; Get the usage time Δt i The number of times the i-th data is used is u i , the average usage frequency of the i-th data is f i =u i / Δt i Step S102: Set the i-th data to be cleared at the j-th data clearing, and obtain the clearing time interval ΔT between the j-1-th data clearing and the j-th data clearing j,j-1 ; Extract the number of times the i-th data is used during the cleaning time interval as u re i , the most recent usage frequency of the i-th data is f re i =u re i / ΔT j,j-1 ; Step S103: Obtain the unused time ut from the last time the i-th data is cleared i , establish the formula as the processing logic of the data clearing probability estimation model: ； Among them, a, b, c and d are constant coefficients, a<0, b<0, c>0; the cleaning probability P of the i-th data is calculated i ; Set a cleaning probability threshold ε, if P i ≥ε, then clear the mark of the i-th data; Step S104: Select several cleaned data as the training set of the data purge probability estimation model. In the training set, the average usage frequency, recent usage frequency and non-use time of the cleaned data are used as the input data of the data purge probability estimation model, and a purge probability P is output, where P≥ε, and the constant coefficients a, b, c and d are determined; the remaining cleaned data are used as the test set. If the output results of the cleaned data do not satisfy P≥ε, the unsatisfactory cleaned data are put into the training set, and the data purge probability estimation model is retrained until the output results of all cleaned data satisfy P≥ε.

2. The IoT cloud data storage management method based on a measurement model according to claim 1, characterized in that: The step S200 includes the following steps: Step S201: Set the average usage frequency of the kth data in the currently stored data to f k and the most recent frequency is f re k , get the unused time ut of the kth data from the last use k , the data removal probability estimation model is used to calculate the removal probability, and the removal probability of the kth data is obtained as P k , if P k ≥ε, then the kth data is marked for clearing; Step S202: Set the data without clear marks as valid data, summarize all valid data, obtain the data source of each valid data, classify the valid data according to the data source, and generate several valid data sets; Step S203: Randomly select a valid data set and obtain the kth ’ The average usage frequency of valid data is and the most recent frequency of use is , according to the formula: ； in, is the kth ’ The probability of clearing valid data is calculated to get the kth ’ Estimated priority for retrieval of valid data ; Get the estimated retrieval priority of all valid data in a valid data set, sort the valid data from large to small according to the value, and use it as the retrieval order for the next data retrieval.

3. The IoT cloud data storage management method based on a measurement model according to claim 2, characterized in that: The step S300 includes the following steps: Step S301: Get the Lth valid data set, the kth ’ The amount of valid data , the total amount of data in the Lth valid data set is Q L ; Step S302: Obtain the storage time point of each valid data in the Lth valid data set, and sort the valid data in the Lth valid data set according to the order of the storage time points; obtain the time interval between two adjacent valid time points as Δt', set a time interval threshold τ, if Δt'>τ, then divide the two valid time points into two time periods; if Δt'≤τ, then divide the two valid time points into one time period; divide all valid time points to obtain several time periods of the Lth valid data set, and set the number of valid time points in the eth time period as n e , according to the formula: ； Where r is the number of time periods divided in the Lth valid data set; the single transmission data volume of the Lth valid data set is calculated as ; Then the storage capacity allocated to the Lth valid data set is .

4. The IoT cloud data storage management method based on a measurement model according to claim 3, characterized in that: The step S400 includes the following steps: Step S401: Distribute the currently transmitted data according to the data source, and set the actual data volume distributed to the Lth valid data set to Q ’ L ; Get the Lth valid data set, the kth ’ The average usage frequency of valid data , recent usage frequency and the unused time since the last use , put it into the data clearing probability estimation model for calculation, and get the The probability of cleaning valid data ; Set the cleaning probability threshold to ε, if , then for the Add a clear mark to valid data; Step S402: Set the number of valid data with clear marks in the current Lth valid data set to U L , the amount of valid data with clear marks is Q ’’ L =U L ×x L , where x L is the amount of data of a single data in the Lth valid data set; the total amount of data originally obtained from the Lth valid data set is Q L , adjust the total amount of new data to Q ’ L =Q L +Q ’ L -Q ’’ L ; Step S403: Obtain the single transmission data volume of the Lth valid data set and the allocated storage capacity S L ,like , then send a reminder of insufficient storage capacity, and the storage capacity to be supplemented is ; Get all other valid data sets with storage capacity less than 0 to be supplemented, among which the Lth ’ The remaining storage capacity of valid data sets is , For L ’ The total amount of new data in the valid dataset, For L ’ The amount of data transmitted in a single transaction for a valid data set, For L ’ The storage capacity allocated for a valid data set; if , then the L ’ The remaining storage capacity of the Lth valid data set is added to the Lth valid data set.

5. An IoT cloud data storage management system applied to an IoT cloud data storage management method based on a measurement model according to any one of claims 1 to 4, characterized in that: The management system includes a cleanup data analysis module, a historical data cleanup module, a storage capacity allocation module and a storage capacity adjustment module; The cleansing data analysis module is used to monitor the usage of each piece of cleansed data in the storage process, build a data cleansing probability estimation model, and train the data cleansing probability estimation model; The historical data cleaning module is used to obtain the usage of each historical data, determine whether to mark it for clearing through the data clearing probability estimation model; set the data without clearing mark as valid data, classify the valid data to obtain several valid data sets; and determine the retrieval order of each valid data in the valid data set in the next data retrieval based on the usage of each valid data in the valid data set; The storage capacity allocation module is used to obtain the total amount of data in each valid data set, analyze the storage time interval between each valid data in the valid data set, divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set; The storage capacity adjustment module is used to analyze the usage of each valid data in the valid data set when new data is currently transmitted, and add clearance marks to some valid data; according to the addition of clearance marks of valid data and the actual amount of data currently allocated, it obtains the remaining storage capacity of the valid data set and adjusts the storage capacity allocated to the valid data set.

6. The IoT cloud data storage and management system according to claim 5, characterized in that: The data cleaning module includes a data usage analysis unit and a model training unit; The data usage analysis unit is used to monitor the usage of each historical cleared data during the storage process; the model training unit is used to build a data clearing probability estimation model and train the data clearing probability estimation model.

7. The IoT cloud data storage and management system according to claim 5, characterized in that: The historical data cleaning module includes a clearing judgment unit and a valid data storage unit; The clearing judgment unit is used to obtain the usage of each historical data and determine whether to mark it for clearing through a data clearing probability estimation model; the valid data storage unit is used to set the data without clearing marks as valid data, classify the valid data, and obtain several valid data sets; based on the usage of each valid data in the valid data set, the retrieval order of each valid data in the valid data set during the next data retrieval is obtained.

8. The IoT cloud data storage and management system according to claim 5, characterized in that: The storage capacity allocation module includes a storage time analysis unit and a capacity demand calculation unit; The storage time analysis unit is used to obtain the total amount of data in each valid data set and analyze the storage time interval between each valid data in the valid data set; The capacity requirement calculation unit is used to divide the valid data into time periods, obtain the single transmission data volume of the valid data set, and allocate storage capacity to the valid data set.

9. The IoT cloud data storage and management system according to claim 5, characterized in that: The storage capacity adjustment module includes a clear mark adding unit and a remaining capacity analysis unit; The clear mark adding unit is used to analyze the usage of each valid data in the valid data set and add clear marks to some valid data if new data is currently transmitted; the remaining capacity analysis unit is used to obtain the remaining storage capacity of the valid data set based on the clear mark addition status of the valid data and the actual amount of data currently allocated, and adjust the storage capacity allocated to the valid data set.

Images