Data backup and compression method and apparatus

Abstract

The application relates to a data backup and compression method and device, which comprises the following steps: collecting ETCD cluster operation data to form a monitoring index sequence; inputting a pre-trained LSTM time sequence prediction model to output a predicted value in a next time window, including a predicted TPS peak value, a data increment size and a potential fault probability; dynamically determining a snapshot backup interval and a compression scheme according to the predicted value: shortening the backup interval when the predicted load is increased, increasing the compression level when the storage is tight or the data increment is increased, and shortening the interval or triggering a temporary snapshot when the fault probability exceeds a threshold; executing snapshot backup and compression and recording actual data; comparing the actual data with the predicted value, updating model parameters based on a combined loss function, and forming a closed loop. The scheme can reduce the data loss risk, save storage space, reduce the influence on system performance, and realize adaptive optimization of the backup strategy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to data backup and compression methods and apparatus. Background Technology

[0002] ETCD is a highly available distributed key-value store database widely used in container orchestration systems such as Kubernetes to store cluster state data. Because the data stored in ETCD is crucial for the stable operation of the cluster and business continuity, data loss due to catastrophic failures will have a severe impact. Therefore, operations and maintenance personnel typically need to perform regular snapshot backups of ETCD data to mitigate the risk of data loss.

[0003] Currently, most existing technologies back up ETCDs by calling snapshot commands at fixed intervals via scheduled tasks, and then compress the generated snapshot files using a fixed compression level. However, this static backup scheduling and compression strategy is difficult to adapt to dynamic changes in cluster load: if the backup interval is set too long, the potential data loss window between snapshots is large, reducing data security; if the backup interval is set too short, frequent snapshot operations will occur, resulting in high system overhead and storage resource consumption. Furthermore, fixed compression strategies cannot be flexibly adjusted based on snapshot size and available storage resources. In addition, existing solutions are all based on current or historical states and passively respond, lacking the ability to predict future cluster load trends, and thus failing to achieve proactive backup scheduling and resource optimization. Therefore, how to dynamically adjust backup scheduling and compression strategies based on predictions of cluster load trends while ensuring data security has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a data backup and compression method and apparatus.

[0005] According to a first aspect of the embodiments of this application, a data backup and compression method is provided, the method comprising: Collect real-time and historical operational data of the ETCD cluster to form a series of monitoring indicators; The monitoring index sequence is input into a pre-trained LSTM (Long Short-Term Memory) time series prediction model. The LSTM layer in the LSTM time series prediction model encodes the temporal dependencies of the monitoring index sequence and outputs the predicted value in the next time window. The predicted value includes the predicted peak TPS (Transactions Per Second), the data increment size, and the probability of potential failure. Based on the predicted values, determine the target interval and compression scheme for the next snapshot backup cycle; ETCD snapshot backup and compression operations are performed according to the snapshot backup target interval and compression scheme, and the actual execution data, including actual TPS, actual data increment, actual failure occurrence, backup time, snapshot file size and resource usage, are recorded. The deviation is obtained by comparing the actual TPS with the predicted peak TPS, the actual data increment with the predicted data increment, and the actual fault occurrence with the predicted potential fault probability. Based on the deviation, the gradient is calculated using a combined loss function consisting of a weighted sum of continuous value prediction error terms and fault classification error terms, and the parameters of the LSTM time series prediction model are updated online incrementally. The actual TPS, actual data increment, and actual fault occurrence are then incorporated into the monitoring indicator sequence as new historical data. At the same time, resource usage data is fed back to correct the compression scheme selection for the next cycle, forming a closed-loop control.

[0006] According to a second aspect of the embodiments of this application, a data backup and compression apparatus is provided, the apparatus comprising: The data acquisition module is used to collect real-time and historical operating data of the ETCD cluster and form a series of monitoring indicators. The prediction module is used to input the monitoring index sequence into the pre-trained LSTM time series prediction model. The LSTM layer in the LSTM time series prediction model encodes the temporal dependency of the monitoring index sequence and outputs the predicted value in the next time window. The predicted value includes the predicted TPS peak, the data increment size and the potential failure probability. The determination module is used to determine the target interval and compression scheme for the next period of snapshot backup based on the predicted values. The backup and compression module is used to perform ETCD snapshot backup and compression operations according to the snapshot backup target interval and compression scheme, and record the actual execution data including actual TPS, actual data increment, actual failure occurrence, backup time, snapshot file size and resource usage. The update module compares the actual TPS with the predicted peak TPS in the forecast values, the actual data increment with the predicted data increment in the forecast values, and the actual fault occurrence with the potential fault probability in the forecast values ​​to obtain the deviation. Based on the deviation, the gradient is calculated using a combined loss function consisting of a weighted sum of continuous value prediction error terms and fault classification error terms, and the parameters of the LSTM time series prediction model are updated online incrementally. The actual TPS, actual data increment, and actual fault occurrence are then incorporated as new historical data into the monitoring indicator sequence. At the same time, resource usage data is fed back to correct the compression scheme selection for the next cycle, forming a closed-loop control.

[0007] According to a third aspect of the embodiments of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the method described above.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the methods described above in this application.

[0009] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above in this application.

[0010] The data backup and compression method and apparatus provided in this application use an AI prediction model to predict the future load and failure risk of the ETCD cluster, and dynamically adjust the snapshot backup frequency and compression strategy accordingly. At the same time, a hard threshold is used as a fallback guarantee to form a closed-loop adaptive control. It can automatically increase the backup frequency and shorten the data loss window during high load or frequent data change periods, and reduce the backup frequency to save resources during low load periods. It can also dynamically select a compression scheme based on the predicted snapshot size and storage status to balance storage space and CPU overhead. By comparing the actual execution data with the predicted value and continuously updating the model parameters, the prediction accuracy is continuously optimized as the cluster runs. This significantly reduces storage costs and system performance overhead while ensuring data security, and achieves long-term adaptive optimization of backup and compression strategies as the cluster status changes. Attached Figure Description

[0011] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A schematic diagram of the architecture of a data backup and compression system provided in an exemplary embodiment of this application; Figure 2 A flowchart illustrating a data backup and compression method provided in an exemplary embodiment of this application; Figure 3 A flowchart illustrating a data backup and compression method provided in another exemplary embodiment of this application; Figure 4 A schematic block diagram of the functional modules of a data backup and compression apparatus provided in an exemplary embodiment of this application; Figure 5 A structural block diagram of an electronic device provided in an exemplary embodiment of this application; Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of this application. Detailed Implementation

[0012] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0013] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this application's technical solution, based on the prompt message.

[0018] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation of this application; other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0019] like Figure 1 The above, Figure 1 This is a schematic diagram of the architecture of a data backup and compression system provided in an embodiment of this application. The system can realize backup and compression based on AI prediction. The system may specifically include: a data monitoring module 11, an AI prediction module 12, a strategy decision module 13, and an execution feedback module 14.

[0020] The data monitoring module 11 continuously collects real-time operational data and historical monitoring metrics of the ETCD cluster using monitoring tools (such as Prometheus combined with Grafana), which serve as input data for the AI ​​prediction module 12. The collected data includes, but is not limited to, the current cluster load, ETCD write transaction frequency, and data change volume. Specifically, the monitoring items are shown in Table 1: Data monitoring module: Continuously collects operational data from the ETCD cluster using monitoring tools (such as Prometheus + Grafana) as input for the AI ​​model. This data includes, but is not limited to: current cluster load (e.g., CPU and memory usage), ETCD write transaction frequency (e.g., TPS), data change volume (e.g., the number of key-value pairs added / modified within a certain period), ETCD database size growth trend, and other relevant system metrics (e.g., disk utilization, network traffic). See Table 1 for the monitoring items and their descriptions.

[0021] Table 1:

[0022] The data monitoring module 11 uses the collected monitoring indicator sequences of the current and historical times as input data for the subsequent AI prediction module 12.

[0023] The AI ​​prediction module 12 deploys a trained machine learning model to predict the cluster state over a future period. Preferably, this AI model employs an LSTM-based time series prediction model, with historical cluster metric data as training input and actual load or write activity at the corresponding time point as training output. Through offline training and continuous online learning, the model can capture periodic patterns and sudden behaviors in the business.

[0024] The model input consists of current and historical monitoring indicator sequences; the model output consists of predicted key indicator values ​​for the next time window (e.g., the next hour or more granular), including the expected peak TPS, the size of the data increment, and the probability of potential failures.

[0025] In this embodiment, an LSTM-based prediction network is used as an example for illustration, where the prediction model is a function. : (1) (2) in, The predicted vector output by the model includes estimates of multiple future indicators in (1). Indicates time arrive Input sequence at time (length) (time window) These are the model parameters.

[0026] The final hidden state is obtained after calculation by the LSTM layer. Then, the output layer is mapped to the desired prediction metric: TPS forecast: (3) Data increment prediction: (4) Failure probability prediction: (5) in , , and corresponding , , The output layer parameters are used for fault probability prediction. (sigmoid) activation is used to guarantee the probability of the output being between 0 and 1.

[0027] Supervised learning is used during training, and model parameters are adjusted based on historical data. Define the actual value as , and actual fault indication (A flag indicating whether a fault has occurred within the prediction window). A combined loss function can be constructed. (θ), the cumulative error of each prediction: (6) in, (7) It is the binary cross-entropy, used to measure the error in predicting the probability of failure. The weighting coefficients are used to balance the magnitude of the loss for different prediction tasks. The loss function takes the sum of the squares of the differences between the predicted and actual values, as well as the weighted sum of the fault classification errors, thereby optimizing the accuracy of continuous value prediction and probabilistic prediction simultaneously during training.

[0028] In this embodiment, the model parameters can be trained by minimizing the loss function using historical data. The parameters are updated iteratively using gradient descent. (8) in, For learning rate, This represents the gradient of the loss function with respect to the parameters.

[0029] Training can be divided into two stages: first, the model is trained offline using historical monitoring data to capture the periodic patterns and sudden behaviors of the business; then, online learning is continuously performed during system operation, and the model parameters are fine-tuned at a certain frequency using newly generated data. Through this online update, the model can always reflect the latest cluster status.

[0030] Based on the prediction results output by the AI ​​prediction module 12, the strategy decision module 13 calculates and adjusts the backup and compression strategies in real time, specifically including two aspects: dynamic adjustment of snapshot backup strategy and dynamic adjustment of snapshot compression strategy.

[0031] (1) Snapshot backup strategy is dynamically adjusted.

[0032] The strategy decision module 13 sets a target RPO range, which is the desired timeframe for controlling the data loss window. For example, the default target could be no more than 30 minutes of data loss. The strategy decision module 13 then dynamically adjusts the scheduling time of the next snapshot based on the predicted write frequency and load level.

[0033] Specifically, when high load or a large amount of data writing is predicted in the near future, the snapshot interval is shortened and the snapshot frequency is increased, and a temporary snapshot is triggered immediately if necessary; conversely, when a period of low load and stability is predicted, the snapshot interval is appropriately extended. At the same time, the strategy decision module 13 considers the impact on system performance and tries to avoid performing backups at the predicted peak moment, for example, scheduling backups at a time when the load is slightly lower, in order to reduce the impact of the snapshot process on online business.

[0034] The snapshot backup target interval can be dynamically calculated using the following formula: (9) in, The baseline interval time corresponding to the target RPO. , It is an adjustment coefficient used to balance the weight of TPS and data increment on the interval. The next stage of TPS (transaction frequency) predicted by the AI ​​model. This refers to the predicted data increment (data change) for the next stage. and These are the thresholds for TPS and data increment, respectively. When or When it increases, it will lead to Decreasing the denominator increases the frequency; conversely, when the forecast load is low and data changes little, the denominator is close to 1. This means the interval tends to approach the default value (or can be slightly extended). This strategy can automatically shorten the backup interval under high load and keep the interval close to the upper limit under low load, thereby dynamically meeting the balance between data security and resource conservation.

[0035] Building upon the aforementioned AI dynamic scheduling, the strategy decision module 13 also incorporates a hard threshold interception mechanism as a safety net for data security. Regardless of the AI ​​model's prediction results, the strategy decision module 13 will immediately trigger a snapshot backup if any of the following conditions are met: The cumulative amount of incremental data since the last snapshot exceeds a preset hard limit (e.g., 50GB). The time interval since the last snapshot has reached the preset hard limit (e.g., 60 minutes).

[0036] Hard thresholds can be configured independently based on the business data protection level and have a higher priority than AI dynamic prediction results, ensuring that the data loss window will not exceed the tolerance range under extreme load or prediction deviation.

[0037] The strategy decision module 13 adjusts the compression scheme based on the predicted growth of snapshot size and storage resources, including selecting the compression algorithm type, adjusting compression level parameters, and determining the snapshot retention and elimination strategy.

[0038] When it is predicted that data growth will be rapid in the near future, snapshot files will become significantly larger, or the system's remaining storage space is limited, the strategy decision module 13 increases the compression ratio to save space to the greatest extent possible, and can compress older, uncompressed snapshots in advance to free up space. Conversely, if it is predicted that data changes will not be significant in the short term and storage is sufficient, a low compression or no compression strategy is adopted to reduce CPU overhead and speed up backup or recovery.

[0039] In addition, the strategy decision module 13 can also determine the incremental backup scheme. When the predicted change is small, incremental snapshots or differential compression are adopted, saving only the difference from the previous snapshot, further reducing the backup size and time. For older snapshots, the strategy decision module 13 intelligently adjusts the retention period based on the predicted future recovery needs: snapshots at important time points are retained for a long time, while redundant intermediate snapshots are deleted after compression or dumped to inexpensive storage media.

[0040] The specific implementation of compression level selection is as follows: make This indicates the predicted size of the next snapshot file (uncompressed). This indicates the current remaining available storage space. Set the selected compression algorithm and its level parameters to... The size of the compressed snapshot is: (10) in, Indicates compression level The corresponding compression factor (the ratio of the output file size to the original size).

[0041] To minimize CPU overhead while meeting storage constraints, an optimization objective function is established, taking into account both the space occupancy ratio after compression and CPU utilization. : (11) in, Indicates the adoption of levels The CPU resources consumed during compression, such as compression time or CPU utilization. Typically... The higher, The larger the value, the more complex and time-consuming the compression algorithm. This indicates the CPU reserve currently available for compression in the system. This is used to adjust the weight of the two, for example, it can be increased when storage space is tight. It tends to save space and can reduce CPU usage when resources are scarce. To reduce the compression load, the final dynamic compression strategy is to select from the available set of compression levels. (Including different algorithms and compression levels), select to use Minimal optimization solution: (12) The execution feedback module 14 performs specific operations according to the frequency and strategy given by the strategy decision module 13, controlled by the backup execution engine. This includes: triggering ETCD snapshots at set times and performing snapshot backups by calling the etcd API or the etcdctl binary tool; and processing the snapshot files according to the specified compression algorithm and compression level after obtaining them. During execution, information such as backup time, file size, and CPU usage are monitored synchronously, and this actual data is fed back to the data monitoring module 11 and the AI ​​prediction module 12 to correct subsequent strategies.

[0042] For the online updates of AI prediction module 12, set at time The predicted output is The actual observed value is Then it can be based on the previously defined loss function. Calculate the gradient and perform an online update on the model parameters: (13) Through the aforementioned feedback and update measures, the system forms a closed-loop control of "prediction → decision-making → execution → feedback". After each snapshot is executed, not only are subsequent snapshots and compression strategies adjusted, but the AI ​​prediction module 12 itself also absorbs new data and corrects parameters. After multiple iterations, the model's prediction accuracy for a specific ETCD cluster continuously improves, making the dynamic adjustment of snapshot frequency and compression strategies more accurate and efficient, maximizing the saving of storage and performance resources while ensuring data security.

[0043] See Figure 2 , Figure 2 This is a flowchart illustrating a data backup and compression method provided in an embodiment of this application. This method can be applied to... Figure 1 The system architecture shown is executed collaboratively by the data monitoring module 11, the AI ​​prediction module 12, the strategy decision-making module 13, and the execution feedback module 14. Figure 2 As shown, the method includes the following steps: S201, the data monitoring module 11 continuously monitors and collects the operating data of the ETCD cluster, which is used as input for AI model training.

[0044] In this embodiment, the operational data includes TPS, data increment, CPU utilization, memory utilization, disk utilization, and network bandwidth. The data monitoring module 11 organizes the collected real-time and historical operational data into a monitoring index sequence according to the sampling timestamp. This monitoring index sequence serves as both the training input for the LSTM time series prediction model and the online prediction input. The specific construction method of the monitoring index sequence can be found in the description of the foregoing embodiment, and will not be repeated here.

[0045] S202, AI prediction module 12 deploys a trained AI model to predict the cluster state over a future period of time.

[0046] Specifically, the AI ​​prediction module 12 inputs the monitoring indicator sequence into a pre-trained LSTM time series prediction model. The LSTM layer encodes the temporal dependencies of the monitoring indicator sequence, and after calculating the final hidden state, the predicted value for the next time window is output through a fully connected output layer. The predicted value includes the predicted peak TPS, the data increment size, and the probability of potential faults. The specific calculation process of the LSTM time series prediction model can be found in the description of the corresponding formulas mentioned above, and will not be repeated here.

[0047] S203, the strategy decision module 13 calculates and adjusts the backup and compression strategies based on the prediction results of the AI ​​model.

[0048] Specifically, the backup and compression strategy includes the target interval time and compression level for snapshot backups. Based on the predicted value output in S202, the strategy decision module 13 calculates the target interval time and compression level for the next period of snapshot backups in real time, and performs the judgments in S204 and S207 respectively.

[0049] S204, determine if the prediction is high load; if not, proceed to S205; if yes, proceed to S206.

[0050] In this embodiment, high load can be determined by at least one of the following methods: the predicted TPS peak or data increment size is larger than the current period or exceeds a preset load threshold; the potential failure probability exceeds a preset failure threshold.

[0051] S205, extend the snapshot interval and provide feedback on the execution results.

[0052] Specifically, when the predicted high load is not the case, it indicates that the ETCD cluster is in a low load state in the next time window. The strategy decision module 13 will extend or maintain the target interval of the snapshot backup to the preset default interval in order to reduce the resource overhead of the backup operation on the ETCD cluster. At the same time, the execution feedback module 14 will feed back the execution results of this cycle to the data monitoring module 11, and the process will return to S201 to enter the next cycle.

[0053] S206, shorten the snapshot interval, trigger temporary snapshots immediately when necessary, and provide feedback on the execution results.

[0054] Specifically, when a high load is predicted, the strategy decision module 13 shortens the target interval for snapshot backups; when the potential failure probability exceeds a preset failure threshold, a temporary snapshot backup is immediately triggered to ensure data security. The specific calculation of the target interval for snapshot backups and its hard threshold constraints (such as the cumulative increment limit and the maximum time interval limit) can be found in the description of the corresponding formulas mentioned above. The execution feedback module 14 feeds back the execution result of this step to the data monitoring module 11, and the process continues to execute S207.

[0055] S207, predict whether the amount of data will change significantly; if not, proceed to S208; if so, proceed to S209.

[0056] Specifically, when the predicted data increment is significantly larger than the current period or exceeds a preset change threshold, it is determined that the predicted data volume has changed significantly.

[0057] S208 uses low-compression or incremental backups and provides feedback on the execution results.

[0058] Specifically, when the amount of predicted data does not change significantly, the strategy decision module 13 selects a lower compression level to compress the snapshot or uses incremental backup to save CPU overhead; the execution feedback module 14 feeds back the execution results to the data monitoring module 11, and the process returns to S201 to enter the next cycle.

[0059] S209 uses a high compression ratio to compress snapshots and provides feedback on execution results.

[0060] Specifically, when the predicted data volume changes significantly, the strategy decision module 13 selects a higher compression level to compress the snapshot with a high compression ratio, thereby reducing storage space usage. Optionally, selecting a higher compression level also requires that the current CPU utilization rate is below a preset compression threshold to avoid high compression operations competing for CPU resources with the ETCD cluster. The specific rules for selecting the compression level can be found in the description of the compression optimization objective function mentioned above. The execution feedback module 14 feeds back the execution result of this step, and after this cycle ends, the process returns to S201 to enter the next cycle.

[0061] In this embodiment, the execution feedback module 14 records actual execution data, including actual TPS, actual data increment, actual fault occurrence, backup time, snapshot file size, and resource usage data. The execution feedback module 14 compares the actual execution data with the predicted value to obtain the deviation, calculates the gradient based on the combined loss function consisting of a weighted sum of continuous value prediction error terms and fault classification error terms, and performs online incremental updates to the parameters of the LSTM time series prediction model. Simultaneously, the actual TPS, actual data increment, and actual fault occurrence are incorporated as new historical data into the monitoring indicator sequence, and the resource usage data is fed back to correct the compression level selection for the next cycle, thus forming a closed-loop control of "prediction-decision-execution-feedback". The specific calculations of the combined loss function and parameter updates can be found in the descriptions of the corresponding formulas above, and will not be repeated here.

[0062] It is understandable that the above steps S201-S209 are executed cyclically, and the execution result of each cycle is returned to S201. The data monitoring module 11 continuously collects the operating data of the ETCD cluster, so that the backup and compression strategies are continuously and dynamically optimized as the cluster status changes. It should also be understood that in other embodiments, the backup interval adjustment in S204-S206 and the compression level adjustment in S207-S209 can also be executed in parallel or in reverse order. Figure 2 The execution order shown is merely illustrative and does not constitute a limitation of this application.

[0063] Based on the above embodiments, this application also provides a data backup and compression method, such as... Figure 3 As shown, the method may include the following steps: In step S310, real-time and historical operating data of the ETCD cluster are collected to form a monitoring indicator sequence.

[0064] In this embodiment, the monitoring metric sequence includes at least TPS, data increment, CPU utilization, and disk utilization, and may also include at least one of memory utilization and network bandwidth. TPS is the current and historical transactions per second of the cluster; data increment reflects the number of key-value pairs added or modified per unit time; CPU utilization and disk utilization are the CPU utilization and disk utilization of Member nodes, respectively. The data monitoring module continuously collects the above operational data using Prometheus in conjunction with monitoring tools such as Grafana, and uses the monitoring metric sequence at the current and historical times as input to the AI ​​prediction module.

[0065] In step S320, the monitoring index sequence is input into the pre-trained LSTM time series prediction model. The LSTM layer in the LSTM time series prediction model encodes the temporal dependency of the monitoring index sequence and outputs the predicted value in the next time window. The predicted value includes the predicted TPS peak, the data increment size, and the potential failure probability.

[0066] Specifically, a historical monitoring index sequence with a time window length of L is input into the LSTM model. After the hidden state is calculated by the LSTM layers, the predicted peak TPS, data increment size, and potential fault probability are mapped through a fully connected output layer. The mapping of the potential fault probability uses a sigmoid activation function to ensure that the output value is between 0 and 1. This LSTM time series prediction model is pre-trained offline using historical monitoring data and continuously learns online during system operation.

[0067] In step S330, the target interval and compression scheme for the next period of snapshot backup are determined based on the predicted values.

[0068] Specifically, when the predicted peak TPS or data increment size increases compared to the current period, the snapshot backup target interval is shortened. When the predicted data increment size increases compared to the current period or the current disk utilization exceeds a preset storage threshold, a higher compression level is selected, provided the current CPU utilization is below a preset compression threshold. When the potential failure probability exceeds a preset failure threshold, the snapshot backup target interval is shortened or a temporary snapshot backup is triggered.

[0069] In this embodiment, under the condition that the current CPU utilization rate is lower than the preset compression threshold, the compression level that minimizes the optimization objective function J(L) is selected from the set of available compression algorithms and compression levels based on the predicted data increment size, the current remaining storage space and the current CPU utilization rate.

[0070] For details, please refer to the embodiments corresponding to the above formulas (9) and (11), which will not be repeated here.

[0071] By using the above methods, we can achieve "more backups during busy periods and fewer backups during idle periods," and automatically increase the compression level when storage is tight or data is growing rapidly, while avoiding over-compression when CPU resources are limited.

[0072] In step S340, ETCD snapshot backup and compression operations are performed according to the snapshot backup target interval and compression scheme, and the actual execution data, including actual TPS, actual data increment, actual failure occurrence, backup time, snapshot file size and resource usage, are recorded.

[0073] Specifically, the execution and feedback module calls the ETCD application programming interface or the etcdctl binary tool to trigger a snapshot backup. After obtaining the original snapshot file, it compresses the snapshot file according to the selected compression algorithm and compression level. During the execution process, it synchronously monitors and records actual execution data such as the actual backup time, the actual size of the generated file, CPU usage, disk I / O, and actual failure occurrences.

[0074] In step S350, the actual execution data is compared with the predicted value to obtain the deviation. Based on this, the LSTM model parameters are updated, and the actual data is fed back to adjust the strategy to form a closed-loop control.

[0075] The deviation can be obtained by comparing the actual TPS with the predicted peak TPS in the predicted value, comparing the actual data increment with the predicted data increment in the predicted value, and comparing the actual fault occurrence with the potential fault probability in the predicted value. Based on the deviation, the gradient is calculated using the combined loss function, which is composed of the weighted sum of the continuous value prediction error term and the fault classification error term, and the parameters of the LSTM time series prediction model are updated online incrementally. The actual TPS, actual data increment, and actual fault occurrence are included as new historical data in the monitoring indicator sequence, and the resource usage data is fed back to correct the compression scheme selection for the next cycle, thereby forming a closed-loop control of prediction-decision-execution-feedback. The combined loss function can be found in formula (6) in the above embodiment; in addition, the update of the parameters of the LSTM time series prediction model can be found in formula (13) above, which will not be repeated here.

[0076] This application's embodiments use an AI prediction model to predict the future load and failure risks of the ETCD cluster, and dynamically adjust the snapshot backup frequency and compression strategy accordingly. A hard threshold is used as a fallback guarantee, forming a closed-loop adaptive control. This automatically increases the backup frequency and shortens the data loss window during periods of high load or frequent data changes, and reduces the backup frequency to save resources during periods of low load. It also dynamically selects a compression scheme based on the predicted snapshot size and storage status to balance storage space and CPU overhead. By comparing actual execution data with predicted values ​​and continuously updating model parameters, the prediction accuracy is continuously optimized as the cluster operates. This significantly reduces storage costs and system performance overhead while ensuring data security, achieving long-term adaptive optimization of backup and compression strategies as the cluster status changes.

[0077] Based on the above embodiments, in another embodiment provided in this application, the above LSTM time series prediction model includes at least one LSTM layer and a fully connected output layer; the LSTM layer encodes the input monitoring index sequence to obtain the hidden state; the fully connected output layer maps the hidden state to the predicted TPS peak, the data increment size, and the potential fault probability, respectively; wherein, the mapping of the potential fault probability adopts the sigmoid activation function, so that the output value is between 0 and 1.

[0078] In another embodiment provided in this application, the aforementioned LSTM time series prediction model may include at least one LSTM layer and a fully connected output layer. The LSTM layer encodes the input monitoring indicator sequence, models the temporal dependencies in the sequence through its internal forget gate, input gate, and output gate, and outputs a hidden state. The fully connected output layer receives this hidden state and maps it to the predicted peak TPS, the data increment size, and the potential failure probability, respectively.

[0079] Specifically, please refer to formulas (3)-(5) in the above embodiments, which will not be repeated here.

[0080] This embodiment clarifies the hierarchical structure of the LSTM time series prediction model, effectively encodes the temporal dependencies of historical monitoring indicator sequences using the LSTM layer, and then maps the hidden states to multiple prediction targets through a fully connected output layer, thereby achieving multi-dimensional prediction of the future load and failure risk of the ETCD cluster. The failure probability is output as a probability value between 0 and 1 using the sigmoid activation function, providing a quantitative risk basis for the subsequent strategy decision module, making the adjustment of backup frequency and compression strategy more reasonable and reliable.

[0081] Based on the above embodiments, in another embodiment provided in this application, the method may further include the following steps: In step S360, it is determined whether the cumulative incremental data volume since the last snapshot exceeds a preset hard upper limit value, or whether the time interval reaches a preset hard duration upper limit value.

[0082] Specifically, while executing the AI ​​dynamic scheduling strategy, the strategy decision-making module simultaneously monitors the cumulative incremental data volume and time interval since the last snapshot. Hard upper limits and hard duration limits can be configured independently based on the business data protection level; for example, the cumulative incremental data volume limit can be set to 50GB, and the time interval limit can be set to 60 minutes. When the cumulative incremental data volume exceeds 50GB, or the time interval since the last snapshot reaches 60 minutes, the mandatory backup conditions are deemed met.

[0083] In step S370, if any condition is met, an immediate backup is forcibly triggered, and the priority of the forced backup is higher than the backup strategy dynamically determined based on the predicted value.

[0084] Regardless of the AI ​​model's prediction, as long as any of the aforementioned hard threshold conditions are met, the strategy decision-making module will immediately trigger a snapshot backup operation without waiting for the scheduling time dynamically calculated by the AI. This hard threshold interception mechanism serves as a safety net for data security, ensuring that the data loss window will not exceed a preset tolerance range under AI prediction bias or extreme load conditions.

[0085] By setting a hard threshold fallback mechanism, this embodiment further ensures data security based on AI prediction and dynamic adjustment of backup strategies, avoiding data loss due to AI prediction deviations or sudden extreme loads, and enhancing the robustness and reliability of the backup strategy.

[0086] Based on the above embodiments, in another embodiment provided in this application, in order to describe in detail how to perform ETCD snapshot backup and compression operations according to the snapshot backup target interval and compression scheme, the above step S340 may specifically include the following steps: In step S341, the ETCD application interface or the etcdctl tool is invoked to trigger a snapshot backup and obtain the original snapshot file.

[0087] Specifically, the backup execution engine in the execution feedback module, based on the scheduling time determined by the policy decision module, calls the ETCD application programming interface (API) or the etcdctl binary tool to initiate a snapshot backup request to the ETCD cluster. The ETCD cluster exports the current data state as a snapshot file and returns it to the backup execution engine. This snapshot file is an uncompressed raw snapshot file, and its size reflects the current total amount of data in the ETCD database.

[0088] In step S342, the original snapshot file is compressed according to the selected compression algorithm and compression level.

[0089] After obtaining the original snapshot file, the backup execution engine compresses it according to the compression algorithm and compression level selected by the strategy decision module. Compression algorithms can include, but are not limited to, gzip, lz4, and zstd, and the compression level can be determined based on the optimization results of the strategy decision module. During compression, the backup time, the actual size of the compressed file, CPU usage, and disk I / O are monitored and recorded synchronously for subsequent feedback and online model updates.

[0090] This embodiment clarifies the specific method for triggering snapshot backup by calling the ETCD application programming interface or the etcdctl tool, and details the specific process of compressing the original snapshot file according to the selected compression algorithm and compression level. This makes the execution steps of ETCD snapshot backup and compression operations clear and feasible. At the same time, it records actual data such as backup time, file size, and CPU usage during the execution process, providing a reliable data foundation for subsequent online updates of the AI ​​model and policy adjustments, ensuring the effective operation of the entire closed-loop control.

[0091] Based on the above embodiments, in another embodiment provided in this application, the method may further include the following steps: In step S380, when the predicted data increment is less than the preset increment threshold, incremental snapshot or differential compression is used for backup, and only the difference from the previous snapshot is saved.

[0092] Specifically, after obtaining the predicted data increment size for the next time window from the AI ​​prediction module, the strategy decision-making module compares this predicted value with a preset increment threshold. The preset increment threshold can be flexibly configured according to the actual business scenario, for example, set to 1GB or 5% of the previous snapshot size. When the predicted data increment size is less than the preset increment threshold, it indicates that the data change in the ETCD cluster will be relatively small in the future. If a full snapshot backup is still performed at this time, a large amount of duplicate data will be generated, resulting in unnecessary storage space consumption and backup time consumption.

[0093] Therefore, the strategy decision module decides whether to use incremental snapshots or differential compression for backup. When performing a backup, the execution feedback module first obtains the baseline data from the previous snapshot, calculates the differences between the current data state and the previous snapshot, and then saves only the differences as the incremental snapshot file; alternatively, it uses a differential compression algorithm to compress and store only the differences. In this way, the backup file contains only the data that has changed since the last snapshot, rather than a complete copy of the database, thus effectively reducing the amount of backup data.

[0094] This embodiment predicts the size of the data increment and intelligently selects incremental snapshot or differential compression method. When the data change is small, only the difference from the previous snapshot is saved, which significantly reduces the amount of backup data, reduces storage space occupation and backup time, and further reduces the impact of backup operations on the performance of online services of ETCD cluster, thus improving the economy and adaptability of backup strategy.

[0095] Based on the above embodiments, in another embodiment provided in this application, the method may further include the following steps: In step S391, the retention period of old snapshots is dynamically adjusted based on the predicted future recovery needs.

[0096] Specifically, after obtaining the predicted information such as future load and failure probability output by the AI ​​prediction module, the strategy decision-making module dynamically adjusts the retention period of each old snapshot based on the importance of the data status corresponding to the snapshot at different time points to business recovery. For example, for snapshots generated around important business change nodes, version release nodes, or predicted high-risk periods, the strategy decision-making module marks them as important snapshots and extends their retention period accordingly; for intermediate redundant snapshots generated during periods of low load stability, their retention period can be appropriately shortened to reduce long-term accumulated storage consumption.

[0097] In step S392, snapshots of important time points are permanently retained, while redundant intermediate snapshots are deleted after compression or dumped to inexpensive storage media.

[0098] Specifically, for the critical snapshots identified in step S391, the strategy decision module controls the execution feedback module to retain these snapshots long-term, ensuring a rapid rollback to the critical data state when needed. For intermediate snapshots deemed redundant, the execution feedback module first compresses them using a selected compression algorithm to reduce their temporary storage size. Then, according to a preset eviction policy, they are either directly deleted after compression or transferred to inexpensive storage media such as object storage or cold storage for long-term archiving. This approach preserves the recoverability of critical snapshots while preventing a large number of redundant snapshots from continuously consuming high-speed storage resources.

[0099] This embodiment dynamically adjusts the retention period of old snapshots based on predicted future recovery needs, retains snapshots from important time points for a long time, and deletes or transfers redundant intermediate snapshots after compression to inexpensive storage media. This achieves refined management of the snapshot storage structure and effective control of storage costs. While ensuring the recoverability of critical data, it significantly reduces the occupation of high-speed storage space by long-term backup data, and improves the utilization efficiency and overall economy of backup storage resources.

[0100] In summary, compared with related technologies, the embodiments of this application have the following significant advantages: (1) Reduce the risk of data loss: Fine-grained backup scheduling is achieved through AI prediction, which can increase the snapshot frequency in a timely manner during high-risk periods to ensure that important data is backed up in a timely manner. This solution is not limited to a fixed time interval, which greatly shortens the data loss window in the worst case. When the cluster fails, the recoverable data state is closer to the time of failure, thereby significantly improving the system's disaster recovery capability.

[0101] (2) Saving storage space and resources: Through dynamic compression strategies, the storage space occupied by snapshot files can be minimized. The system performs high-cost compression only when needed and reduces the compression intensity when it can be relaxed, thereby balancing CPU overhead and storage savings. Compared with fixed compression configuration, it can reduce storage consumption by several times (depending on the data mode) and greatly extend the life of existing backup storage.

[0102] (3) Reduce the impact on system performance: By using intelligent scheduling to avoid peak business periods for backup or selecting incremental backup, the backup process can minimize the interference to the ETCD cluster while ensuring data security. For example, the embodiments of this application can quickly capture snapshots during the short idle period after the predicted high-concurrency business transactions end, thereby capturing peak data without increasing the system load during peak periods.

[0103] (4) Adaptability and versatility: The embodiments of this application adopt a learning algorithm, which can automatically adjust the backup strategy as the business model changes, adapting to different cluster sizes and workload characteristics. The model training data and strategy rules can be customized according to specific deployment scenarios. In addition, the method is transparent to the external environment, does not depend on ETCD source code modification, and can be integrated into platforms such as Kubernetes as an independent scheduling service, linking with existing backup tools to achieve intelligent upgrades.

[0104] The AI-predictive ETCD cluster data backup and compression method provided in this application can be applied to products such as container services. This method can adapt to different cluster sizes and workload characteristics, reducing the risk of cluster data loss by dynamically adjusting snapshot backup frequency and compression strategies. In the event of a cluster failure, because the backup data is closer to the time of the failure, data and services can be restored more quickly, thereby enhancing the resilience and business continuity of container service products. Simultaneously, the dynamic snapshot compression strategy effectively saves system storage space and resources, reducing the storage costs of container service products. Furthermore, the entire system achieves integrated automation of the monitoring, analysis, and execution processes, reducing the need for manual intervention and improving operational efficiency. Therefore, the technical solution of this application has high application value and promising prospects for widespread adoption.

[0105] By dividing the data into functional modules corresponding to each function, embodiments of this application provide a data backup and compression device. This data backup and compression device can be a server, a terminal, or a chip applied to a server. Specifically, it can be applied to… Figure 1 The system in the middle. Figure 4 A schematic block diagram of the functional modules of a data backup and compression apparatus provided for an exemplary embodiment of this application. Figure 4 As shown, the data backup and compression device includes: The data monitoring module 41 is used to collect real-time and historical operating data of the ETCD cluster and form a series of monitoring indicators. AI prediction module 42 is used to input the monitoring index sequence into the pre-trained LSTM time series prediction model, encode the temporal dependency of the monitoring index sequence through the LSTM layer in the LSTM time series prediction model, and output the predicted value in the next time window. The predicted value includes the predicted TPS peak, the data increment size and the potential failure probability. The strategy decision module 43 is used to determine the target interval and compression scheme for the next period of snapshot backup based on the predicted value. The execution feedback module 44 is used to perform ETCD snapshot backup and compression operations according to the snapshot backup target interval and compression scheme, and record the actual execution data including actual TPS, actual data increment, actual failure occurrence, backup time, snapshot file size and resource usage data. The execution feedback module 44 is also used to compare the actual TPS with the predicted peak TPS in the predicted values, compare the actual data increment with the predicted data increment in the predicted values, and compare the actual fault occurrence with the potential fault probability in the predicted values ​​to obtain the deviation. Based on the deviation, the gradient is calculated based on the combined loss function composed of the weighted sum of the continuous value prediction error term and the fault classification error term, and the parameters of the LSTM time series prediction model are updated online incrementally. The actual TPS, actual data increment, and actual fault occurrence are included as new historical data in the monitoring indicator sequence. At the same time, the resource usage data is fed back to correct the compression scheme selection for the next cycle, forming a closed-loop control.

[0106] In another embodiment provided in this application, the strategy decision module 43 is specifically used to: shorten the snapshot backup target interval time when the predicted TPS peak or data increment size increases compared to the current period; select a compression scheme with a higher compression level when the predicted data increment size increases compared to the current period or the current disk utilization exceeds a preset storage threshold, provided that the current CPU utilization is lower than a preset compression threshold; and shorten the snapshot backup target interval time or trigger a temporary snapshot backup when the potential failure probability exceeds a preset failure threshold. In another embodiment provided in this application, the LSTM time series prediction model includes at least one LSTM layer and a fully connected output layer; the LSTM layer encodes the input monitoring index sequence to obtain the hidden state; the fully connected output layer maps the hidden state to the predicted TPS peak, the data increment size, and the potential fault probability, respectively; wherein, the mapping of the potential fault probability adopts the sigmoid activation function, so that the output value is between 0 and 1.

[0107] In another embodiment provided in this application, the combined loss function is:

[0108] in, , , These represent the predicted peak TPS, the size of the data increment, and the probability of potential failures, respectively. , , These represent the actual TPS, actual data increment, and actual fault occurrence, respectively. BCE is the binary cross-entropy function, and β is the weighting coefficient.

[0109] In another embodiment provided in this application, the strategy decision module 43 is further used for: Based on the predicted peak TPS and the size of the data increment, the snapshot backup target interval Δt(t) is calculated according to the following formula:

[0110] in, The baseline interval time corresponding to the target RPO. and For adjustment coefficients, To predict the next stage of peak TPS, To predict the size of the data increment in the next stage, and These are the TPS threshold and the data increment threshold, respectively.

[0111] In another embodiment provided in this application, the execution feedback module 44 is further used for: Determine whether the cumulative incremental data volume since the last snapshot exceeds the preset hard limit, or whether the time interval reaches the preset hard duration limit. If any condition is met, an immediate backup is forcibly triggered, and this forced backup has a higher priority than the backup strategy dynamically determined based on predicted values.

[0112] In another embodiment provided in this application, the strategy decision module 43 is further used for: Under the condition that the current CPU utilization is below the preset compression threshold, based on the predicted data increment size, the current remaining storage space, and the current CPU utilization, the compression level that minimizes the optimization objective function J(L) is selected from the set of available compression algorithms and compression levels; where,

[0113] The size of the snapshot after compression using compression level L. This represents the current remaining available storage space. This refers to the CPU resources consumed when using level L compression. λ represents the CPU margin available for compression in the system, and λ is the weighting coefficient.

[0114] In another embodiment provided in this application, ETCD snapshot backup and compression operations are performed according to the snapshot backup target interval and compression scheme, including: Call the ETCD application programming interface or the etcdctl tool to trigger a snapshot backup and obtain the original snapshot file; The original snapshot file is compressed according to the selected compression algorithm and compression level.

[0115] In another embodiment provided in this application, the execution feedback module 44 is further used for: The parameters of the LSTM time series prediction model are updated using the gradient descent method according to the following formula:

[0116] in For learning rate, This is the gradient of the combined loss function with respect to the model parameters.

[0117] In another embodiment provided in this application, the execution feedback module 44 is further used for: When the predicted data increment is less than the preset increment threshold, incremental snapshot or differential compression is used for backup, and only the difference from the previous snapshot is saved.

[0118] In another embodiment provided in this application, the device further includes an adjustment module, specifically used for: The retention period for old snapshots is dynamically adjusted based on predicted future recovery needs. Snapshots of important time points are permanently retained, while redundant intermediate snapshots are deleted after compression or dumped to inexpensive storage media.

[0119] The data backup and compression device provided in this application uses an AI prediction model to predict the future load and failure risks of the ETCD cluster, and dynamically adjusts the snapshot backup frequency and compression strategy accordingly. At the same time, it uses a hard threshold as a fallback guarantee to form a closed-loop adaptive control. It can automatically increase the backup frequency and shorten the data loss window during periods of high load or frequent data changes, and reduce the backup frequency to save resources during periods of low load. It also dynamically selects a compression scheme based on the predicted snapshot size and storage status to balance storage space and CPU overhead. By comparing the actual execution data with the predicted values ​​and continuously updating the model parameters, the prediction accuracy is continuously optimized as the cluster runs. This significantly reduces storage costs and system performance overhead while ensuring data security, and achieves long-term adaptive optimization of backup and compression strategies as the cluster status changes.

[0120] This application also provides an electronic device, including: at least one processor; a memory for storing executable instructions of the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the method disclosed in the embodiments of this application.

[0121] Figure 5 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. For example... Figure 5 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can perform the corresponding steps in the methods disclosed in the embodiments of this application.

[0122] The processor 1801 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this application can be implemented by the integrated logic circuitry in the hardware of the processor 1801 or by instructions in software form. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1802, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1801 reads information from the memory 1802 and, in conjunction with its hardware, completes the steps of the above method.

[0123] Furthermore, the various operations / processes according to this application, when implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 6 The computer system 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 6A structural block diagram of a computer system provided for an exemplary embodiment of this application.

[0124] Computer System 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0125] like Figure 6 As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computer system 1900. The computing unit 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0126] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disks and optical disks. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0127] The computing unit 1901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in the embodiments of this application can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the methods disclosed in the embodiments of this application by any other suitable means (e.g., by means of firmware).

[0128] This application also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this application.

[0129] The computer-readable storage medium in this application embodiment may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0130] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0131] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this application.

[0132] In embodiments of this application, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] The modules, components, or units described in the embodiments of this application can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0135] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0136] The above description is merely an embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0137] While specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this application. The scope of this application is defined by the appended claims.

Claims

1. A data backup and compression method, characterized in that, The method includes: Collect real-time and historical operational data of the ETCD cluster to form a series of monitoring indicators; The monitoring index sequence is input into a pre-trained LSTM time series prediction model. The LSTM layer in the LSTM time series prediction model encodes the temporal dependency of the monitoring index sequence and outputs the predicted value in the next time window. The predicted value includes the predicted TPS peak, the data increment size, and the potential failure probability. Based on the predicted values, determine the target interval and compression scheme for the next snapshot backup cycle; ETCD snapshot backup and compression operations are performed according to the snapshot backup target interval and compression scheme, and the actual execution data, including actual TPS, actual data increment, actual failure occurrence, backup time, snapshot file size and resource usage, are recorded. The actual TPS is compared with the predicted peak TPS in the predicted values, the actual data increment is compared with the predicted data increment in the predicted values, and the actual fault occurrence is compared with the potential fault probability in the predicted values ​​to obtain the deviation. Based on the deviation, the gradient is calculated using a combined loss function consisting of a weighted sum of continuous value prediction error terms and fault classification error terms, and the parameters of the LSTM time series prediction model are updated online incrementally. The actual TPS, actual data increment, and actual fault occurrence are then incorporated as new historical data into the monitoring indicator sequence. Simultaneously, the resource usage data is fed back to correct the compression scheme selection for the next cycle, forming a closed-loop control.

2. The method according to claim 1, characterized in that, The LSTM time series prediction model includes at least one LSTM layer and a fully connected output layer. The LSTM layer encodes the input monitoring index sequence to obtain the hidden state. The fully connected output layer maps the hidden state to the predicted TPS peak, the data increment size, and the potential fault probability, respectively. The mapping of the potential fault probability uses the sigmoid activation function, so that the output value is between 0 and 1.

3. The method according to claim 1, characterized in that, According to the method of claim 1, the combined loss function is: in, , , These represent the predicted peak TPS, the size of the data increment, and the probability of potential failures, respectively. , , These represent the actual TPS, actual data increment, and actual fault occurrence, respectively. BCE is the binary cross-entropy function, and β is the weighting coefficient.

4. The method according to claim 1, characterized in that, The step of determining the target interval and compression scheme for the next snapshot backup based on the predicted value includes: When the predicted peak TPS or the data increment size increases compared to the current period, the target interval for the snapshot backup is shortened. When the predicted data increment is larger than the current period or the current disk utilization exceeds the preset storage threshold, a higher compression level compression scheme is selected if the current CPU utilization is lower than the preset compression threshold. When the potential failure probability exceeds a preset failure threshold, the snapshot backup target interval is shortened or a temporary snapshot backup is triggered.

5. The method according to claim 4, characterized in that, When the predicted TPS peak or data increment size increases compared to the current period, shortening the snapshot backup target interval includes: Based on the predicted peak TPS and the size of the data increment, the target interval time Δt(t) for the snapshot backup is calculated according to the following formula: in, The baseline interval time corresponding to the target RPO. and For adjustment coefficients, To predict the next stage of peak TPS, To predict the size of the data increment in the next stage, and These are the TPS threshold and the data increment threshold, respectively.

6. The method according to claim 5, characterized in that, The method further includes: Determine whether the cumulative incremental data volume since the last snapshot exceeds the preset hard limit, or whether the time interval reaches the preset hard duration limit. If any condition is met, an immediate backup is forcibly triggered, and the priority of this forced backup is higher than the backup strategy dynamically determined based on the predicted value.

7. The method according to claim 4, characterized in that, The selection of a compression scheme with a higher compression level includes: Under the condition that the current CPU utilization is below the preset compression threshold, based on the predicted data increment size, the current remaining storage space, and the current CPU utilization, the compression level that minimizes the optimization objective function J(L) is selected from the set of available compression algorithms and compression levels; where, The size of the snapshot after compression using compression level L. This represents the current remaining available storage space. This refers to the CPU resources consumed when using level L compression. λ represents the CPU margin available for compression in the system, and λ is the weighting coefficient.

8. The method according to claim 1, characterized in that, The step of performing ETCD snapshot backup and compression operations according to the snapshot backup target interval and compression scheme includes: Trigger a snapshot backup to obtain the original snapshot file; The original snapshot file is compressed according to the selected compression algorithm and compression level.

9. The method according to claim 1, characterized in that, The step of calculating the gradient based on the combined loss function consisting of a weighted sum of continuous value prediction error terms and fault classification error terms, and then incrementally updating the parameters of the LSTM time series prediction model online, includes: The parameters of the LSTM time series prediction model are updated using the gradient descent method according to the following formula: in For learning rate, The gradient of the combined loss function with respect to the model parameters is given.

10. The method according to claim 1, characterized in that, The method further includes: When the predicted data increment is less than the preset increment threshold, incremental snapshot or differential compression is used for backup, and only the difference from the previous snapshot is saved.

11. The method according to claim 1, characterized in that, The method further includes: The retention period for old snapshots is dynamically adjusted based on predicted future recovery needs. Snapshots of important time points are permanently retained, while redundant intermediate snapshots are deleted after compression or dumped to inexpensive storage media.

12. A data backup and compression device, characterized in that, The device includes: The data acquisition module is used to collect real-time and historical operating data of the ETCD cluster and form a series of monitoring indicators. The prediction module is used to input the monitoring index sequence into a pre-trained LSTM time series prediction model, encode the temporal dependency of the monitoring index sequence through the LSTM layer in the LSTM time series prediction model, and output the predicted value in the next time window. The predicted value includes the predicted TPS peak, the data increment size, and the potential failure probability. The determination module is used to determine the target interval and compression scheme for the next period of snapshot backup based on the predicted value. The backup and compression module is used to perform ETCD snapshot backup and compression operations according to the snapshot backup target interval and compression scheme, and record the actual execution data including actual TPS, actual data increment, actual failure occurrence, backup time, snapshot file size and resource usage data. The update module is used to compare the actual TPS with the predicted TPS peak value in the predicted value, compare the actual data increment with the predicted data increment value, and compare the actual fault occurrence with the potential fault probability in the predicted value to obtain the deviation. Based on the deviation, the gradient is calculated using a combined loss function consisting of a weighted sum of continuous value prediction error term and fault classification error term, and the parameters of the LSTM time series prediction model are updated online incrementally. The actual TPS, actual data increment, and actual fault occurrence are incorporated into the monitoring indicator sequence as new historical data, and the resource usage data is fed back to correct the compression scheme selection for the next cycle, forming a closed-loop control.

13. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images