Big data storage and backup management system based on cloud service

Abstract

The invention discloses a big data storage and backup management system based on cloud service, relates to the field of big data backup management, and is used for solving the problems that parameters collected in a parameter library cannot be dynamically adjusted to balance the stability of the system, and the cloud storage environment adaptability of long-term operation of the system is reduced. A weighted analysis model is established by collecting a deviation value between an estimated backup time interval and a reference value, a deviation value between an estimated backup data volume and a reference data volume and an overall round duration, the total number of times of backup interval adjustment is obtained, parameters influencing the backup interval adjustment range are collected and integrated into a parameter library, and the backup interval adjustment range is adjusted according to the parameter library. Setting a data cooling and heating mechanism according to the network delay effect corresponding to all the parameters in the parameter library and the data change frequency, classifying the parameters in the parameter library, formulating a group of fuzzy rules for fuzzy reasoning according to the frequency characteristics of the cold data parameters in the backup interval, determining the calling result of the cold data parameters, and optimizing the utilization rate of storage resources.

Description

Technical Field

[0001] The present invention relates to the field of big data backup management, and more specifically, to a big data storage and backup management system based on cloud services. Background Art

[0002] In the cloud-based big data storage and backup management system, backup is an important means to ensure data security. A reasonable backup interval can ensure that the system has recent backup data when unforeseen data failures or anomalies occur, thereby minimizing the risk of data loss. Reasonable backup interval adjustment is crucial to system stability, optimal use of storage resources, and efficient data recovery.

[0003] The prior art has the following deficiencies:

[0004] At present, the system stability and backup performance are balanced by dynamically adjusting the backup frequency. However, when facing sudden data growth or abnormal network fluctuations, the parameters related to the backup time interval cannot be accurately called for calculation, which results in the inability to dynamically adjust the parameters collected and stored in the parameter library to balance the system stability, reducing the adaptability of the system to the cloud storage environment for long-term operation and increasing the waste of computing resources caused by unreasonable adjustments. Therefore, a big data storage and backup management system based on cloud services is proposed.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a big data storage and backup management system based on cloud services, which solves the problems raised in the above-mentioned background technology by using different product inspection methods.

[0007] To achieve the above object, the present invention provides the following technical solution: a cloud service-based big data storage and backup management system, including a data acquisition module, an adjustment determination module, a parameter management module and a data call module; signal connections between the modules;

[0008] The data collection module is used to collect backup feature information and backup round information, and send it to the adjustment determination module;

[0009] The adjustment determination module is used to obtain backup feature information and backup round information, establish a weighted analysis model, obtain the total number of backup interval adjustments, and collect parameters that affect the backup interval adjustment range, integrate them into a parameter library and send them to the parameter management module;

[0010] The parameter management module is used to obtain the total number of backup interval adjustments and the parameter library, obtain all parameters in the parameter library, and set the data hot and cold mechanism according to the corresponding data of all parameters in the parameter library, classify the parameters in the parameter library, and send them to the data call module;

[0011] The data calling module is used to obtain parameter classification results, and according to the frequency characteristics of cold data parameters within the backup interval, the fuzzy logic is substituted to determine the cold data parameter calling results.

[0012] In a preferred embodiment, the backup characteristic information includes the deviation between the estimated backup time interval and the reference value and the deviation between the estimated backup data volume and the reference data volume;

[0013] The estimated backup time interval is calculated by combining the backup time interval data recorded in the previous backup operation with the current system storage utilization, and then subtracted from the preset backup time interval reference value to obtain the deviation between the estimated backup time interval and the reference value;

[0014] Based on the amount of backup data in the previous round, the current data growth rate, and the frequency of data changes, an estimated amount of backup data is obtained, and the difference is subtracted from the preset reference amount of backup data to obtain the deviation between the estimated amount of backup data and the reference amount of data;

[0015] The overall round duration is calculated based on real-time monitoring data.

[0016] In a preferred embodiment, the deviation between the estimated backup time interval and the reference value, and the deviation between the estimated backup data volume and the reference data volume are obtained to establish a weighted analysis model and generate a deviation coefficient;

[0017] Substitute the deviation coefficient and the overall round duration into the adaptive backup interval adjustment formula to obtain the total number of backup interval adjustments.

[0018] In a preferred embodiment, the parameter library includes a data growth ratio, a time deviation ratio, and a system load deviation ratio;

[0019] The backup interval is a preset fixed backup interval during the first run, and the length of subsequent backup intervals is determined based on the corresponding parameters in the parameter library;

[0020] The data growth ratio, time deviation ratio and system load deviation ratio are all the accumulated data growth ratio, time deviation ratio and system load deviation ratio of the second time to the first time after the first initial backup interval of the current round has been run;

[0021] The network delay effect and data change frequency corresponding to all parameters in the parameter library jointly determine the adjustment of data growth rate, time deviation ratio and system load deviation ratio.

[0022] In a preferred embodiment, the network delay effect is obtained by weighting the round-trip time and the one-way transmission delay and calculating the ratio of the round-trip time to the current backup data volume and the current network bandwidth;

[0023] The data change amount within a unit time is obtained and the ratio is calculated with the preset cumulative time intervals of multiple unit times to obtain the data change frequency.

[0024] In a preferred embodiment, the network delay effect and the data change frequency corresponding to the data growth ratio, the time deviation ratio and the system load deviation ratio are multiplied and substituted into the data growth ratio, the time deviation ratio and the system load deviation ratio to obtain the data hot and cold evaluation standard;

[0025] The data hot and cold evaluation standards corresponding to the data growth ratio, the time deviation ratio, and the system load deviation ratio are sorted in descending order, and the two parameters with the highest sorting are used as hot data parameters, and the parameter with the lowest sorting is used as a cold data parameter.

[0026] In a preferred embodiment, the frequency characteristics of the cold data parameters within the backup interval include the cold data parameter growth frequency and the number of cold data parameter backup intervals;

[0027] Record the value of the cold data parameter, calculate the growth rate based on the cold data parameter values ​​backed up twice in a row, and then calculate the ratio of two adjacent growth rates to obtain the cold data parameter growth frequency;

[0028] Count the number of times hot data parameters are generated to obtain the number of cold data parameter backup intervals.

[0029] In a preferred embodiment, the cold data parameter growth frequency and the cold data parameter backup interval times are defined as input variables, and are divided into different fuzzy sets respectively;

[0030] The cold data parameter call results are defined as output variables and divided into fuzzy sets;

[0031] Formulate fuzzy rules to describe the impact of cold data parameter growth frequency and cold data parameter backup interval on cold data parameter call results;

[0032] Perform fuzzy reasoning based on fuzzy rules to determine the cold data parameter calling results.

[0033] Technical effects and advantages of the present invention:

[0034] 1. The present invention establishes a weighted analysis model by collecting the deviation between the estimated backup time interval and the reference value, the deviation between the estimated backup data volume and the reference data volume, and the overall round duration, obtains the total number of backup interval adjustments, and collects parameters that affect the backup interval adjustment range and integrates them into a parameter library. According to the network delay effect and data change frequency corresponding to all parameters in the parameter library, a data hot and cold mechanism is set, hot data parameters and cold data parameters are classified, the dynamic adaptability of the backup strategy is improved, the storage resource utilization is optimized, the backup interval is dynamically adjusted, and the system stability is balanced.

[0035] 2. The present invention obtains hot data parameters and cold data parameters, and obtains the cold data parameter growth frequency and the number of cold data parameter backup intervals based on the frequency characteristics of the cold data parameters in the backup interval, formulates a set of fuzzy rules for fuzzy reasoning, determines the cold data parameter call results, adapts to large-scale data backup environments, avoids long-term non-backup of cold data, and reduces the risk of data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a module diagram of the big data storage and backup management system based on cloud services of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Example 1

[0039] The present invention discloses a big data storage and backup management system based on cloud services, such as Figure 1 As shown, it includes a data acquisition module, an adjustment determination module, a parameter management module and a data calling module; and the signal connections between the modules;

[0040] The data collection module is used to collect backup feature information and backup round information, obtain the deviation between the estimated backup time interval and the reference value, the deviation between the estimated backup data volume and the reference data volume, and the overall round duration through data processing, and send them to the adjustment determination module;

[0041] Among them, data processing includes data type conversion, which refers to converting the raw data collected by the system into standardized data types; missing value processing, which refers to filling the missing data that may occur during the system collection process through interpolation and mean filling to ensure the continuity and integrity of the data, etc. The purpose of data processing is to make the data clearer and easier to understand, so that the system can calculate faster;

[0042] Specifically, the above data operation methods are all existing technologies and will not be described in detail here;

[0043] The backup characteristic information includes the deviation between the estimated backup time interval and the reference value and the deviation between the estimated backup data volume and the reference data volume, and the backup round information includes the overall round duration;

[0044] Specifically, the system backup operation is divided into multiple rounds, each round contains multiple backup times, and the initial backup interval is set first. The specific round number and initial backup interval are determined by the experimenter based on historical data and big data coverage, and will not be elaborated here;

[0045] The logic for obtaining the deviation between the estimated backup time interval and the reference value is to calculate the estimated backup time interval based on the backup time interval data recorded in the previous round of backup operations, combined with the current system storage utilization, according to the trend prediction algorithm, and then pre-set the backup time interval reference value according to the system design requirements and historical operation statistics, and subtract the estimated backup time interval from the backup time interval reference value to obtain the deviation between the estimated backup time interval and the reference value. ;

[0046] The trend prediction algorithm uses the exponential smoothing method to calculate the estimated next backup time interval. Suppose the actual time interval of the i-th backup in the previous round is , and assume that there are n backup records, and these data form a time series:

[0047] ;

[0048] Specifically, exponential smoothing is a simple and effective trend prediction algorithm. Its core idea is to give higher weights to the latest observations and lower weights to earlier data. Suppose the smoothing coefficient is (in ), the prediction model is described as:

[0049] ;

[0050] In the formula, is the predicted value of the i+1th backup time interval, is the actual backup time interval of the ith time, is the predicted value of the i-th backup time interval;

[0051] It should be noted that the initial value Can be set to the first actual observation Or the experimenter selects a reasonable value based on prior knowledge, etc., which will not be elaborated here;

[0052] Specifically, regarding the subscripts and superscripts in the above calculation formula, b indicates that the parameter is related to "backup", that is, it refers to the time interval in the backup operation, and p indicates "estimated", that is, the estimated backup time interval obtained by the prediction model;

[0053] Using the above exponential smoothing formula, the previous round of data is iteratively calculated to obtain the predicted time intervals from the second to the n+1th backup in turn, and the last iterative calculation result is obtained. Then, use it as the estimated backup time interval;

[0054] It should be noted that the estimated backup interval is actually the first backup interval of the current round, which can be considered as the initial backup interval of the current round;

[0055] Among them, the smoothing coefficient controls the weight of the latest observation value in the prediction. A larger smoothing coefficient makes the prediction more sensitive to the latest data, while a smaller smoothing coefficient makes the predicted value smoother. The optimal smoothing coefficient is determined by historical data fitting and error analysis, so the smoothing coefficient is continuously updated iteratively;

[0056] Subtract the estimated backup interval from the backup interval reference value. The specific formula is:

[0057] ;

[0058] In the formula, is the backup time interval reference value, where the subscript re indicates a “reference” value, i.e., a preset backup time interval reference value;

[0059] The logic for obtaining the deviation between the estimated backup data volume and the reference data volume is based on the previous round of backup data volume, the current data growth rate, and the frequency of data changes. The estimated backup data volume is calculated by substituting it into the prediction model, and then the reference backup data volume is preset based on the system storage plan and historical statistical data. The estimated backup data volume is subtracted from the reference backup data volume to obtain the deviation between the estimated backup data volume and the reference data volume. ;

[0060] Specifically, the amount of backup data in the previous round is the amount of historical backup data obtained by weighted calculation of the backup file size and the number of backup records;

[0061] Subtract the estimated backup data volume from the reference backup data volume. The specific formula is:

[0062] ;

[0063] In the formula, The amount of backup data for reference;

[0064] It should be noted that the estimated amount of data to be backed up next time can be calculated by substituting the prediction model into the prediction model, which is not described in detail here.

[0065] The overall round duration refers to the time span from the start to the end of a round, including a series of backup operations and possible backup interval adjustments in the cloud-based big data storage and backup management system. The acquisition logic calculates the overall round duration based on real-time monitoring data, and substitutes the backup data volume of the previous round and the target data transmission rate of the previous round into the optimization model to obtain the overall round duration. ;

[0066] Specifically, the calculation formula of the optimization model is:

[0067] ;

[0068] In the formula, is the amount of backup data in the previous round, is the target data transmission rate of the previous round, t is the duration of the previous round, To find the value of t that minimizes the objective function;

[0069] It should be noted that a round represents all relevant scheduling, adjustment, and monitoring operations in the entire backup cycle. The entire optimization process aims to select a suitable time window for the new round based on the actual operation data of the previous round, so as to achieve adaptive tuning and efficient operation of the system.

[0070] Specifically, the reference values ​​corresponding to all the above parameters are obtained by the experimenters based on historical data, and will not be elaborated here;

[0071] The adjustment determination module is used to obtain the deviation between the estimated backup time interval and the reference value, the deviation between the estimated backup data volume and the reference data volume, establish a weighted analysis model, obtain the deviation coefficient, substitute the deviation coefficient and the overall round duration into the calculation, obtain the total number of backup interval adjustments, and collect parameters that affect the backup interval adjustment range, and integrate them into a parameter library and the total number of backup interval adjustments and send them to the parameter management module;

[0072] Specifically, the weighted analysis model refers to generating the deviation coefficient through weighted calculation;

[0073] Usually, in the weighted calculation process, the above parameters are normalized first, and the parameters of different dimensions and dimensions are normalized to be in the same relative scale range, so as to perform reasonable weighted calculation;

[0074] Obtain the deviation between the estimated backup time interval and the reference value, and the deviation between the estimated backup data volume and the reference data volume to establish a weighted analysis model and generate a deviation coefficient , based on the formula:

[0075] ;

[0076] In the formula, is the coefficient of variation, as well as are respectively preset proportional coefficients of the deviation between the estimated backup time interval and the reference value and the ratio of the deviation between the estimated backup data volume and the reference data volume to the corresponding reference value, and as well as All are greater than 0;

[0077] Substitute the deviation coefficient and the overall round duration into the calculation, and the specific formula is expressed as:

[0078] ;

[0079] In the formula, Adjust the total number of times for the backup interval; is the number of reference backup cycles theoretically included in this round;

[0080] Furthermore, in order to prevent the system from being under-regulated due to a negative deviation coefficient or too low an adjustment number, at least one adjustment is usually set, i.e. ;

[0081] It should be noted that when the system is in the initial state of round operation, different initial fixed backup intervals are set. The specific fixed backup time can be determined according to the transmission status or security level of the big data. The specific security level can be obtained according to the preset hash algorithm. The transmission status of the big data can be obtained according to the transmission network status and the data size, etc., which will not be elaborated here.

[0082] Specifically, the parameter library includes data growth ratio, time deviation ratio, and system load deviation ratio;

[0083] From the above, it can be seen that the backup interval is a preset fixed backup interval during the first run, and the length of subsequent backup intervals is determined according to the corresponding parameters in the parameter library;

[0084] At the same time, according to the length of the subsequent backup interval and the above calculation, the total number of backup interval adjustments is obtained, so that the parameters selected and called in the parameter library are different in each backup interval. The total number of backup interval adjustments represents that in the current round, the parameters obtained by the prediction model are substituted into the data model for calculation, the deviation coefficient is determined, and then the total number of backup interval adjustments for the current round is determined in advance by calculating with the overall round length;

[0085] It should be noted that the total number of backup interval adjustments and the total number of backup intervals are not the same concept. The total number of backup intervals is related to parameters such as the current round duration, and usually includes the number of backup intervals that do not need to be adjusted.

[0086] The parameter management module is used to obtain the total number of backup interval adjustments, and obtain all parameters in the parameter library by connecting to the parameter library. According to the network delay effect and data change frequency corresponding to all parameters in the parameter library, the data hot and cold mechanism is set, and hot data parameters and cold data parameters are classified and sent to the data call module.

[0087] The parameter library includes data growth ratio, time deviation ratio and system load deviation ratio;

[0088] Specifically, the data growth ratio, time deviation ratio, and system load deviation ratio are all the accumulated data growth ratio, time deviation ratio, and system load deviation ratio of the second time to the first time after the first initial backup interval of the current round has been run;

[0089] For example, in the first case, the data volume is m, the time is t1, and the load is v1. In the second case, the data volume is u, the time is t2, and the load is v2. Then the data growth ratio is um / m, the time deviation ratio is t2-t1 / t1, the system load deviation ratio is v2-v1 / v1, and so on.

[0090] Among them, the network delay effect and data change frequency corresponding to all parameters in the parameter library can jointly determine the increase and decrease of the data growth rate, time deviation rate and system load deviation rate;

[0091] The logic of obtaining the network delay effect is to use the round-trip time and one-way transmission delay to measure the round-trip time from the backup system to the storage server through the ping command, and then perform a weighted calculation on the ratio of the round-trip time to the current backup data volume and the current network bandwidth to obtain the network delay effect. ; Where i is the i-th parameter in the parameter library;

[0092] Specifically, when the network delay effect is greater, the data growth rate, time deviation rate and system load deviation rate, that is, the delay increases, resulting in a longer backup data transmission time, and the time deviation rate increases, resulting in data failure to be backed up on time. The amount of data at the next backup becomes larger, and the data growth rate increases, resulting in a longer system waiting time. The CPU is occupied for a long time, and the system load deviation rate increases;

[0093] The logic of obtaining the frequency of data changes is to obtain the data change amount per unit time by setting the unit time in the backup interval and then calculate the ratio with the preset time interval of multiple unit times to obtain the data change frequency. ;

[0094] Specifically, when the frequency of data changes increases, the amount of data per unit time increases. Therefore, compared with the last backup, the current amount of data will become larger, resulting in an increase in the data growth rate, which indirectly increases the time required for the backup process, thereby increasing the time deviation ratio. Secondly, the storage system needs to process a larger amount of data, increasing CPU calculations, disk and network bandwidth usage, thereby increasing system load;

[0095] It should be noted that the preset unit time is set by the experimenter according to the total number of specific backup intervals, the current backup interval, and the total number of backup intervals in the previous round. Specifically, the preset unit time in the data change frequency needs to be less than the current backup interval. Since the backup interval is in an adjustment state, the preset unit time will also be adjusted with the adjustment of the current backup interval. The specific adjustment mechanism is not limited and will not be elaborated here.

[0096] The network delay effect and data change frequency corresponding to the data growth ratio, time deviation ratio and system load deviation ratio are multiplied and substituted into the data growth ratio, time deviation ratio and system load deviation ratio to obtain the data hot and cold evaluation standard;

[0097] The specific data cold and hot evaluation standard calculation formula is:

[0098] ;

[0099] Since this example shows that the parameter library contains three parameters, in actual application scenarios, multiple parameters can be used for calculation, and the number of parameters in the specific parameter library is not limited;

[0100] in, as well as The network delay effect and data change frequency corresponding to the data growth rate parameter, is the data hot and cold evaluation standard corresponding to the data growth ratio, then as well as The network delay effect and data change frequency corresponding to the time deviation ratio parameter, is the data hot / cold evaluation standard corresponding to the time deviation ratio, then as well as The network delay effect and data change frequency corresponding to the system load deviation ratio parameter, The data hot and cold evaluation criteria corresponding to the system load deviation ratio;

[0101] The data hot and cold evaluation criteria corresponding to the data growth ratio, the time deviation ratio, and the system load deviation ratio are sorted in descending order, and the two parameters with the highest order are used as hot data parameters, and the parameter with the lowest order is used as a cold data parameter;

[0102] For example, , as well as Sort them from largest to smallest and get , the time deviation ratio and system load deviation ratio are obtained as hot data, and the data growth ratio is obtained as cold data. Then, the adjustment instruction is executed at the next backup interval, and the time deviation ratio and system load deviation ratio are substituted into the calculation to determine the adjustment range of the backup interval, etc., which will not be elaborated here;

[0103] The present invention establishes a weighted analysis model by collecting the deviation between the estimated backup time interval and the reference value, the deviation between the estimated backup data volume and the reference data volume, and the overall round duration, obtains the total number of backup interval adjustments, and collects parameters that affect the backup interval adjustment range and integrates them into a parameter library. According to the network delay effect and data change frequency corresponding to all parameters in the parameter library, a data hot and cold mechanism is set, hot data parameters and cold data parameters are classified, the dynamic adaptability of the backup strategy is improved, the storage resource utilization is optimized, the backup interval is dynamically adjusted, and the system stability is balanced.

[0104] Example 2

[0105] In Example 1 of the present invention, examples are given to illustrate the collection of the deviation between the estimated backup time interval and the reference value, the deviation between the estimated backup data volume and the reference data volume, and the overall round duration, and to establish a weighted analysis model to obtain the total number of backup interval adjustments, and to collect parameters that affect the backup interval adjustment range and integrate them into a parameter library. According to the network delay effect and data change frequency corresponding to all parameters in the parameter library, a data hot and cold mechanism is set, and the operation strategies of hot data parameters and cold data parameters are classified. However, in Example 1, only the hot data parameters are analyzed. Obviously, if the cold data parameters are ignored for a long time, the cold data will not be backed up for a long time, the risk of data loss will increase, and the system backtracking capability will be reduced. In view of the above problems, Example 2 of the present invention is further refined;

[0106] The data calling module is used to obtain hot data parameters and cold data parameters, and obtain the cold data parameter growth frequency and the number of cold data parameter backup intervals according to the frequency characteristics of the cold data parameters in the backup interval, and substitute the fuzzy logic to determine the cold data parameter calling result;

[0107] Specifically, the frequency characteristics of the cold data parameters within the backup interval include the cold data parameter growth frequency and the number of cold data parameter backup intervals;

[0108] The logic for obtaining the cold data parameter growth frequency is to record the value of the cold data parameter in each backup interval, use the cold data parameter values ​​of two consecutive backups as a comparison basis, calculate the growth rate, and then calculate the ratio of two adjacent growth rates to obtain the cold data parameter growth frequency;

[0109] The logic for obtaining the number of cold data parameter backup intervals is to record the number and corresponding status of the current backup interval in the current round, count the number of times the hot data parameters are generated, and record it as the number of cold data parameter backup intervals;

[0110] For example, "High", "Low", "Medium" for the cold data parameter growth frequency, "More", "Less", "Average" for the cold data parameter backup interval times;

[0111] Formulate a set of fuzzy rules to describe the impact of different input variables on output variables. The definition of rules can be based on professional knowledge or obtained through data analysis and experiments. For example:

[0112] The cold data parameter growth frequency is marked as X, the cold data parameter backup interval is marked as U, and the cold data parameter call result is marked as C_Public;

[0113] Then we can define:

[0114] Rule 1: IF (X is High) AND (U is Less) THEN (C_Public is High)

[0115] Rule 2: IF (X is Low) AND (U is More) THEN (C_Public is Low) ...

[0117] Perform fuzzy reasoning based on fuzzy rules to determine the cold data parameter call results;

[0118] It should be noted that the division of fuzzy sets can be adjusted according to actual conditions. For example, although this embodiment takes three fuzzy sets as examples, the cold data parameter growth frequency and the number of cold data parameter backup intervals can actually be divided into more than three sets to facilitate better classification of data packets according to different life cycle stages for precise adjustment.

[0119] Further, for the high, medium and low judgment of the cold data parameter growth frequency and the cold data parameter backup interval times, thresholds can be set according to actual conditions for judgment. For example, when the cold data parameter growth frequency exceeds 80%, it is marked as "High", and when the cold data parameter backup interval times are higher than 75%, it is marked as "More", etc., which will not be elaborated here;

[0120] It should be noted that the result of calling the cold data parameter is "High", which means that when the adjustment instruction is executed at the next backup interval, not only the hot data parameters are called for calculation, but also the cold data is called for calculation as the basis for adjusting the backup interval;

[0121] The present invention obtains hot data parameters and cold data parameters, and obtains the cold data parameter growth frequency and the number of cold data parameter backup intervals according to the frequency characteristics of the cold data parameters in the backup interval, formulates a set of fuzzy rules for fuzzy reasoning, determines the cold data parameter call results, adapts to large-scale data backup environments, avoids long-term non-backup of cold data, and reduces the risk of data loss.

[0122] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0123] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0124] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0125] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0131] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A cloud-based big data storage and backup management system, characterized by: It includes data acquisition module, adjustment and determination module, parameter management module and data calling module; signal connection between each module; The data collection module is used to collect backup feature information and backup round information, and send them to the adjustment determination module; The adjustment determination module is used to obtain backup feature information and backup round information, establish a weighted analysis model, obtain the total number of backup interval adjustments, and collect parameters that affect the backup interval adjustment range, integrate them into a parameter library and send them to the parameter management module; The parameter management module is used to obtain the total number of backup interval adjustments and the parameter library, obtain all parameters in the parameter library, and set the data hot and cold mechanism according to the corresponding data of all parameters in the parameter library, classify the parameters in the parameter library, and send them to the data call module; The data calling module is used to obtain parameter classification results, and according to the frequency characteristics of cold data parameters within the backup interval, the fuzzy logic is substituted to determine the cold data parameter calling results.

2. The cloud service-based big data storage and backup management system according to claim 1, characterized in that: The backup characteristic information includes the deviation between the estimated backup time interval and the reference value and the deviation between the estimated backup data volume and the reference data volume; The estimated backup time interval is calculated by combining the backup time interval data recorded in the previous backup operation with the current system storage utilization, and then subtracted from the preset backup time interval reference value to obtain the deviation between the estimated backup time interval and the reference value; Based on the amount of backup data in the previous round, the current data growth rate, and the frequency of data changes, an estimated amount of backup data is obtained, and the difference is subtracted from the preset reference amount of backup data to obtain the deviation between the estimated amount of backup data and the reference amount of data; The overall round duration is calculated based on real-time monitoring data.

3. The cloud-based big data storage and backup management system according to claim 2, characterized in that: Obtain the deviation between the estimated backup time interval and the reference value, and the deviation between the estimated backup data volume and the reference data volume, establish a weighted analysis model, and generate a deviation coefficient; Substitute the deviation coefficient and the overall round duration into the adaptive backup interval adjustment formula to obtain the total number of backup interval adjustments.

4. The cloud-based big data storage and backup management system according to claim 3, characterized in that: The parameter library includes data growth ratio, time deviation ratio and system load deviation ratio; The backup interval is a preset fixed backup interval during the first run, and the length of subsequent backup intervals is determined based on the corresponding parameters in the parameter library; The data growth ratio, time deviation ratio and system load deviation ratio are all the accumulated data growth ratio, time deviation ratio and system load deviation ratio of the second time to the first time after the first initial backup interval of the current round has been run; The network delay effect and data change frequency corresponding to all parameters in the parameter library jointly determine the adjustment of data growth rate, time deviation ratio and system load deviation ratio.

5. The cloud-based big data storage and backup management system according to claim 4, characterized in that: Through the round-trip time and one-way transmission delay, the network delay effect is obtained by weighted calculation of the ratio of the round-trip time to the current backup data volume and the current network bandwidth; The data change amount within a unit time is obtained and the ratio is calculated with the preset cumulative time intervals of multiple unit times to obtain the data change frequency.

6. The cloud service-based big data storage and backup management system according to claim 5, characterized in that: The network delay effect and data change frequency corresponding to the data growth ratio, time deviation ratio and system load deviation ratio are multiplied and substituted into the data growth ratio, time deviation ratio and system load deviation ratio to obtain the data hot and cold evaluation standard; The data hot and cold evaluation standards corresponding to the data growth ratio, the time deviation ratio, and the system load deviation ratio are sorted in descending order, and the two parameters with the highest sorting are used as hot data parameters, and the parameter with the lowest sorting is used as a cold data parameter.

7. The cloud service-based big data storage and backup management system according to claim 6, characterized in that: The frequency characteristics of the cold data parameters within the backup interval include the cold data parameter growth frequency and the number of cold data parameter backup intervals; Record the value of the cold data parameter, calculate the growth rate based on the cold data parameter values ​​backed up twice in a row, and then calculate the ratio of two adjacent growth rates to obtain the cold data parameter growth frequency; Count the number of times hot data parameters are generated to obtain the number of cold data parameter backup intervals.

8. The cloud service-based big data storage and backup management system according to claim 7, characterized in that: The cold data parameter growth frequency and the cold data parameter backup interval times are defined as input variables and divided into different fuzzy sets respectively; The cold data parameter call results are defined as output variables and divided into fuzzy sets; Formulate fuzzy rules to describe the impact of cold data parameter growth frequency and cold data parameter backup interval on cold data parameter call results; Perform fuzzy reasoning based on fuzzy rules to determine the cold data parameter calling results.

Images