Data backup method and device, equipment, storage medium and product

Abstract

The invention discloses a data backup method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring an environment parameter and a working parameter of at least one file to be backed up in a data backup process in a previous period; determining working parameters of the current period of the data backup process based on the environment parameters and the working parameters of the previous period and a preset optimization model; and based on the working parameters of the current period, driving a streaming backup tool to execute data backup. According to the data backup method and device, no local disk falling in the backup process can be achieved through data stream processing of stream type backup, the uploading speed can be increased, delay can be reduced, the network resource utilization rate can be optimized by dynamically adjusting the working parameters of the stream type backup tool, and then the efficiency and reliability of data backup are improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data backup method, apparatus, device, storage medium, and product. Background Technology

[0002] In related technologies, cloud providers' database products typically store users' backup files in storage products with massive storage capacity, such as object storage. For example, they store them in EOS (Elastic Object Storage), which has the characteristics of high performance, high reliability, security, and low cost. It provides storage services for unstructured data (image, audio, video, text, and other file formats) through the standard S3 interface, meeting users' needs to manage and access data from anywhere via the Internet.

[0003] Traditional file upload mechanisms use a single thread, resulting in long upload times and low system resource utilization. Furthermore, with large data volumes, timeouts or overwriting of the original file can occur, leading to backup file upload failures and potentially causing service interruptions, downtime, and data loss. In large-scale clusters, to improve the success rate and transmission speed of backup file uploads, a common industry solution is to split the target large file into N smaller files. When requesting upload via the API, parallel processing logic is invoked. After all requests receive responses, the files are merged and verified on the server side. Failed portions can be retransmitted individually instead of the entire large file. This method reduces user waiting time and alleviates server pressure; it is known as file chunking upload.

[0004] However, the above-mentioned file chunking upload often has drawbacks such as increased overhead due to an unreasonable number of chunks and insufficient network bandwidth utilization, which affect the efficiency and reliability of data backup. Summary of the Invention

[0005] In view of this, embodiments of this application provide a data backup method, apparatus, device, storage medium, and product, aiming to improve the efficiency and reliability of data backup.

[0006] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a data backup method for performing fragmented data backup of at least one file to be backed up using a streaming backup tool, the method comprising: Obtain the environmental and operational parameters of at least one file to be backed up in the previous cycle during the data backup process; Based on the environmental parameters, operating parameters, and preset optimization model of the previous cycle, the operating parameters for the current cycle of the data backup process are determined. Based on the current cycle's working parameters, the streaming backup tool is driven to perform the data backup. The environmental parameters include one or more of the following: a first environmental parameter characterizing network performance and a second environmental parameter characterizing the input and output performance of the client to which the file to be backed up belongs; the working parameters include one or more of the following: the fragment size for partitioning, the number of concurrent threads for concurrent upload, and the retry strategy for fragment retransmission.

[0007] The method in the above scheme further includes: Initialize the working parameters based on the backup type of the at least one file to be backed up; Based on the initialized operating parameters, the streaming backup tool is driven to start the data backup.

[0008] In the above scheme, initializing the working parameters based on the backup type of the at least one file to be backed up includes: Obtain the backup type and historical data backup information of the at least one file to be backed up; The working parameters are initialized based on the backup type, the backup information, and the heuristic solution method. The backup information includes the working parameters and backup upload rate for at least one period of historical data backups.

[0009] In the above scheme, the preset optimization model adopts the Fruit Fly Optimization Algorithm (FOA) or an optimization algorithm combining FOA and simulated annealing.

[0010] In the above scheme, the preset optimization model adopts an optimization algorithm that combines FOA and simulated annealing algorithm. The environmental parameters also include annealing temperature and annealing rate.

[0011] The method in the above scheme further includes: The period is determined based on the interval at which heartbeat packets are sent to the storage server; the storage server is used to generate backup files for the data backup. The step of obtaining the environmental parameters of at least one file to be backed up during the data backup process in the previous cycle includes: The first environmental parameter is generated based on the network latency and / or packet loss rate obtained from the heartbeat packet; The second environment parameter is generated based on the number of input / output operations per second (IOPS) and / or throughput of the client in the previous cycle.

[0012] In the above scheme, determining the working parameters for the current cycle of the data backup process based on the environmental parameters, working parameters, and a preset optimization model from the previous cycle includes: The first environmental parameter is converted into a first input value based on a first function, and the second environmental parameter is converted into a second input value based on a second function; wherein, the first function represents the correlation between the first environmental parameter and the fragment size, and the second function represents the correlation between the second environmental parameter and the fragment size; The first input quantity, the second input quantity, the fragment size of the previous cycle, and the file size of the file to be backed up are input into the preset optimization model to obtain the working parameters of the current cycle of the data backup process.

[0013] In the above scheme, driving the streaming backup tool to perform the data backup based on the working parameters of the current period includes: The data in the data backup process is divided into multiple backup tasks based on the fragment size of the current period. The multiple backup tasks are added to the task queue in a specified order, and the thread pool is invoked to execute the backup tasks in the task queue concurrently based on the number of concurrent threads in the current period.

[0014] The method in the above scheme further includes: If it is determined that the upload of any backup task's corresponding fragment fails, the failed backup task is added to the task queue based on the retry policy of the current period; wherein, the retry policy includes: the number of retries and / or the retry interval.

[0015] In the above scheme, if the backup type of the file to be backed up is a binary log backup type, the method further includes: Obtain the High Frequency Data Distribution Rate (HFDDR) corresponding to the shards of the binary log backup type during the data backup process of the previous cycle. The retry strategy for sharding the binary log backup type is determined based on the high-frequency data distribution rate.

[0016] Secondly, embodiments of this application also provide a data backup apparatus for performing fragmented data backup of at least one file to be backed up based on a streaming backup tool, the apparatus comprising: The acquisition module is used to acquire the environmental parameters and working parameters of at least one file to be backed up in the previous cycle during the data backup process; An optimization module is used to determine the working parameters for the current cycle of the data backup process based on the environmental parameters, the working parameters, and a preset optimization algorithm from the previous cycle. The driver module is used to drive the streaming backup tool to perform the data backup based on the working parameters of the current cycle; The environmental parameters include one or more of the following: a first environmental parameter characterizing network performance and a second environmental parameter characterizing the input and output performance of the client to which the file to be backed up belongs; the working parameters include one or more of the following: the fragment size for partitioning, the number of concurrent threads for concurrent upload, and the retry strategy for fragment retransmission.

[0017] Thirdly, embodiments of this application also provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor is used to run the computer program, it performs the steps of the method described in the first aspect above.

[0018] Fourthly, embodiments of this application also provide a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0019] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0020] The technical solution provided in this application embodiment obtains the environmental parameters and working parameters of at least one file to be backed up in the previous cycle during the data backup process; based on the environmental parameters, working parameters, and a preset optimization model of the previous cycle, the working parameters of the current cycle of the data backup process are determined; based on the working parameters of the current cycle, the streaming backup tool is driven to perform data backup; wherein, the environmental parameters include one or more of the following: a first environmental parameter characterizing network performance and a second environmental parameter characterizing the input and output performance of the client to which the file to be backed up belongs, and the working parameters include one or more of the following: the fragment size used for partitioning, the number of concurrent threads used for concurrent upload, and the retry strategy used for fragment retransmission. In this way, the data stream processing of streaming backup can achieve backup without local disk persistence, which is suitable for scenarios with low storage resource consumption and high network efficiency requirements. Furthermore, by dynamically adjusting the working parameters of the streaming backup tool, the upload speed can be improved, latency reduced, and network resource utilization optimized, thereby improving the efficiency and reliability of data backup. Attached Figure Description

[0021] Figure 1This is a flowchart illustrating the data backup method according to an embodiment of this application; Figure 2 This is a schematic diagram of the overall architecture of the backup device in an application embodiment of this application; Figure 3 This is a schematic diagram of the data backup device according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0022] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0024] While file chunking can improve the success rate and transfer speed of backup file uploads, related technologies still have the following drawbacks: 1) The additional overhead of the fragmentation method is not considered. Large file fragmentation inherently incurs additional overhead. Because a file needs to be split into multiple smaller files before transmission, the process of how to divide them and the size of each fragment involves additional steps such as file splitting, indexing, and tagging. Simultaneously, at the receiving end, these smaller files need to be reassembled into the original file, which also requires additional processing time and resources.

[0025] As the number of fragments increases, the number of network requests will also increase accordingly, accompanied by additional network overhead and latency. Too many requests may lead to a decrease in transmission efficiency.

[0026] 2) Network bandwidth utilization was not fully considered. Most existing solutions focus on the file itself, discussing how to efficiently split large files to improve transmission efficiency. However, in large-scale cloud cluster environments, network bandwidth is a crucial factor that directly affects data transmission efficiency, system response speed, and overall performance. Current technologies have not yet considered the impact of network bandwidth on data backup performance in cloud environments.

[0027] 3) Limited by the client's local disk I / O rate As the client initiates the backup process, the upload speed of the cloud database instance backup is also limited by the client's local disk I / O (input and output) performance. If local disk performance cannot be improved, optimizations in other areas are necessary.

[0028] Based on this, in various embodiments of this application, by introducing a preset optimization model and combining the environmental and working parameters of the previous cycle of streaming backup, the optimal working parameters for the current cycle of streaming backup are determined. On the one hand, key parameters such as fragment size and number of concurrent threads can be dynamically adjusted to adapt to real-time environmental changes, thereby improving data backup efficiency. On the other hand, by introducing an optimization model, multi-dimensional factors can be comprehensively considered to achieve a balance between resource utilization and transmission success rate. Compared with the fixed or empirical parameter setting method in the prior art, this significantly improves the stability and reliability of data backup.

[0029] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: 1) Environmental parameters: A set of data used to characterize the external environmental conditions during data backup, including but not limited to network performance (such as network latency and packet loss rate) and client input / output performance (such as IOPS and throughput). These parameters reflect the current system operating status and are an important basis for optimizing upload strategies.

[0030] 2) Operating parameters: These refer to adjustable operational parameters during the data backup process, used to control the specific behavior of chunked uploads, including but not limited to chunk size, number of concurrent threads, and retry strategy. By properly setting these parameters, backup efficiency can be improved and the failure rate reduced.

[0031] 3) Pre-defined optimization model: A swarm intelligence optimization algorithm is used to output the optimal operating parameters for the current period. It can employ algorithms such as FOA (Field-Oriented Analysis), artificial fish swarming, or heuristic search algorithms to dynamically adjust the operating parameters for the current period based on historical environmental and operating parameters. This model can find the optimal solution in a multi-dimensional parameter space, thereby achieving efficient and stable data backup.

[0032] 4) Streaming backup tools: These tools support a backup-as-you-go mechanism, enabling the uploading of file chunks to the target storage server even before the file is fully generated. This significantly improves backup efficiency and reduces transmission failures caused by incomplete file writing.

[0033] 5) Chunking: The process of dividing a large file into several smaller chunks according to certain rules; each chunk is called a chunk. Chunking can reduce the amount of data in a single request, reduce the risk of transmission failure, and improve the overall upload speed.

[0034] 6) Retry Strategy: This refers to the strategy for deciding whether and how to re-upload after a fragment upload fails. It typically includes two aspects: the number of retries and the retry interval. A reasonable retry strategy can effectively address network fluctuations and other issues, improving the upload success rate.

[0035] 7) Heartbeat: A control message packet used to detect network connection status. By periodically sending heartbeat packets, key indicators such as network latency and packet loss rate can be obtained, thereby assessing the current network condition and adjusting upload strategies accordingly.

[0036] 8) IOPS (Input / Output Operations Per Second): This is one of the important indicators for measuring the performance of storage devices. Higher IOPS means faster read and write capabilities, which helps improve backup efficiency.

[0037] 9) Throughput: The amount of data transmitted per unit of time, usually measured in MB / s or GB / s. Higher throughput indicates higher network bandwidth utilization, which helps to speed up the backup process.

[0038] 10) Fruit Fly Optimization Algorithm (FOA): A swarm intelligence optimization algorithm that simulates the foraging behavior of fruit flies and searches for the optimal solution iteratively. It features low computational cost and fast convergence speed, making it suitable for solving optimization problems.

[0039] 11) Heuristic solution method: An approximate solution method based on experience or existing knowledge, often used for fast solutions to complex problems. In this application, the heuristic solution method is used to initialize the parameters in the optimization model to accelerate the convergence process.

[0040] 12) Simulated Annealing (SA): A probabilistic global optimization algorithm based on the principle of physical annealing, capable of avoiding getting trapped in local optima. In this application, SA is introduced into the FOA algorithm to enhance its global search capability and robustness.

[0041] It should be noted that the embodiments of this application can be applied to various cloud database backup scenarios, such as full backup, incremental backup, logical backup, and binary log (binlog) backup. The following will use specific backup types as examples to illustrate how this application optimizes the upload strategy through a preset optimization model to improve backup efficiency and stability.

[0042] In related technologies, traditional backup methods often use fixed fragment sizes and thread counts, lacking adaptability to environmental changes, resulting in low backup efficiency and high failure rates. To address these issues, this application provides an improved data backup method that dynamically adjusts the upload strategy (i.e., working parameters) for the current period based on historical environmental parameters and working parameters, thereby improving backup efficiency, reducing retransmissions, and fully utilizing system resources.

[0043] This application provides a data backup method for performing fragmented data backup of at least one file to be backed up using a streaming backup tool. This data backup method can be executed by a backup device configured with a streaming backup tool for backing up the file to a cloud database. (Refer to...) Figure 1 The data backup method includes: Step 101: Obtain the environmental parameters and working parameters of at least one file to be backed up in the previous cycle during the data backup process.

[0044] Here, environmental parameters refer to a set of data used to characterize the external environmental conditions during data backup. For example, environmental parameters include at least one of the following: a first environmental parameter characterizing network performance and a second environmental parameter characterizing the input and output performance of the client to which the file to be backed up belongs. Exemplarily, environmental parameters include, but are not limited to, network performance (such as network latency and packet loss rate) and client input / output performance (such as IOPS and throughput). Environmental parameters reflect the current system operating status and can serve as an important basis for optimizing upload strategies. For example, when a user performs a full backup operation, environmental parameters may include the current network bandwidth usage and local disk read / write speed; when a user performs an incremental backup operation, environmental parameters may include network fluctuations and the existence of I / O bottlenecks. Working parameters refer to adjustable operating parameters during data backup, used to control the specific behavior of chunked uploads, including but not limited to chunk size, number of concurrent threads, and retry strategy. By reasonably setting working parameters, the system can improve backup efficiency and reduce failure rate. For example, in a backup, working parameters may include a chunk size of 5MB, a number of concurrent threads of 10, and a retry strategy of up to 3 retransmissions with a 2-second interval between each retransmission.

[0045] In practice, backup devices can be operated collaboratively by a file analyzer, a FOA optimizer, and a task executor. The file analyzer extracts the environmental and operational parameters from the previous backup cycle from historical backup records, for example, through logs or monitoring tools. The advantage of this operation is that it provides the FOA optimizer with the information from the previous cycle, allowing it to predict the optimal configuration for the current cycle based on past experience data. This improves upload efficiency, reduces failure rates, and avoids resource waste.

[0046] Step 102: Based on the environmental parameters, working parameters, and preset optimization model of the previous cycle, determine the working parameters for the current cycle of the data backup process.

[0047] Here, the preset optimization model is a swarm intelligence optimization algorithm used to output the optimal working parameters for the current cycle. For example, this preset optimization model can employ a mathematical model based on an improved FOA algorithm, used to dynamically adjust the working parameters for the current cycle according to historical environmental parameters and working parameters. The preset optimization model can find the optimal solution in a multi-dimensional parameter space, and the system can achieve efficient and stable data backup. The FOA algorithm simulates fruit fly foraging behavior, iteratively searching for the optimal solution. The FOA algorithm has the characteristics of low computational cost and fast convergence speed, making it suitable for solving optimization problems. To enhance the global search capability and robustness of the FOA algorithm, this application introduces heuristic initial solution generation and simulated annealing algorithms. Heuristic initial solution generation accelerates the process of entering the effective solution space by collecting a large training set and selecting the groups with the fastest average rate as initial inputs, rather than random initialization. The simulated annealing algorithm further explores the subtle structure of the solution space by probabilistically accepting non-optimal solutions, in order to find a solution closer to the global optimum. For example, in a large-scale cluster backup task, assuming the previous cycle's environmental parameters showed low network bandwidth and high local disk I / O, the optimization model might suggest increasing the shard size, reducing the number of concurrent threads, and using a longer retry interval. By increasing the shard size, reducing the number of concurrent threads, and using a longer retry interval, the number of requests can be minimized under limited bandwidth conditions, improving the transmission efficiency of a single shard. In actual implementation, the FOA optimizer will output the optimal combination of working parameters for the current cycle after multiple iterations of calculation based on the previous cycle's environment and working parameters. By combining historical data with the current environment, adaptive parameter adjustments can be achieved, thereby maximizing upload efficiency and resource utilization while ensuring system stability.

[0048] Step 103: Based on the working parameters of the current cycle, drive the streaming backup tool to perform the data backup.

[0049] Here, a streaming backup tool refers to a tool that supports a backup-while-upload mechanism, enabling it to begin uploading chunks of the file to be backed up to the target storage server even before the file to be backed up is fully generated. Streaming backup tools significantly improve backup efficiency and reduce transmission failures caused by incomplete file writing. For example, xtrabackup, as a tool supporting a backup-while-upload mechanism, can package data blocks in real time during database backup and transmit them to the object storage service via xbstream, thus achieving efficient integrated backup and upload operations. Streaming backup tools can include xtrabackup, mysqldump, and mysql. Specifically, xtrabackup can be used for full and incremental backups; mysqldump can be used for logical backups; and mysql can be used for Binlog backups. The choice between xtrabackup, mysqldump, and mysql depends on the specific backup type and application scenario. In actual implementation, the task executor starts the task queue based on the working parameters output by the FOA optimizer and executes the upload task according to the specified chunk size, number of concurrent threads, and retry strategy. If an upload of a segment fails, it will be re-queued according to priority and re-transmitted by an idle thread. The entire process is managed through a thread pool to ensure efficient execution of upload tasks and reasonable utilization of resources.

[0050] In practice, different backup types have different tool requirements. For example, full backups typically involve copying large amounts of physical data, making binary-level backup tools like xtrabackup more suitable; while logical backups focus more on the structure and integrity of the data, making script-based tools like mysqldump more appropriate. Furthermore, MySQL's binary logs (binlogs) can be parsed and backed up using the mysql command-line tool to preserve complete transaction records.

[0051] In summary, the embodiments of this application provide an improved data backup method. This method obtains the environmental and operational parameters from the previous cycle, combines them with a preset optimization model, dynamically adjusts the operational parameters for the current cycle, and drives a streaming backup tool to perform data backup. This not only improves backup efficiency but also reduces the failure rate, achieving optimal resource utilization.

[0052] In some embodiments, the method further includes: Initialize the working parameters based on the backup type of the at least one file to be backed up; Based on the initialized operating parameters, the streaming backup tool is driven to start the data backup.

[0053] In this application, backup type refers to different categories based on the nature and purpose of the files to be uploaded, such as full backups, incremental backups, logical backups, and binlogs file backups. Each backup type has different file characteristics. For example, full backups typically involve large amounts of data and are suitable for xbstream streaming backups; while logical backups may contain more structured data, and the sharding strategy needs to consider the compression ratio and parsing efficiency of logical backups. Initialization parameters refer to a series of configuration parameters set according to the aforementioned backup types for subsequent backup operations, including but not limited to shard size, number of concurrent threads, retry strategy, and priority settings. The initialization parameter process relies on the file analyzer reading and evaluating file attributes (such as size, type, and historical upload performance) and dynamically adjusting them in conjunction with real-time environmental information such as current network conditions and local I / O performance. By using backup type as the basis for initialization, the upload requirements of different file characteristics can be matched more accurately, improving the overall system resource utilization and upload efficiency, thereby reducing the risk of upload failures or timeouts due to misconfiguration, and improving user experience and operational efficiency.

[0054] Here, initiating data backup using a streaming backup tool refers to calling the tool's interface and starting the backup process after initializing the parameters, according to their specific configuration. This process typically includes initializing the backup channel, establishing a connection, and starting data streaming. Simultaneously, the streaming backup tool will split large files into smaller chunks based on the configured chunking strategy and concurrency parameters, and upload them sequentially or by priority. By using initialization parameters to drive the streaming backup tool, seamless integration between the backup and upload processes can be achieved, improving stability and efficiency during large-scale data uploads. Furthermore, streaming backup effectively addresses network fluctuations or disk I / O performance bottlenecks, reducing the risk of interruptions during data uploads and ultimately enhancing the overall robustness and reliability of the system.

[0055] In practical implementation, the initialized working parameters provide specific operation instructions for subsequent calls to the streaming backup tool, ensuring that the backup process conforms to the optimization goals defined by initializing working parameters based on backup type and driving the streaming backup tool to start data backup. In summary, in this embodiment, by initializing working parameters based on backup type and driving the streaming backup tool to start data backup, the method of initializing working parameters based on backup type can more efficiently match the upload requirements of different files. Driving the streaming backup tool to start data backup can optimize the upload strategy and reduce the failure rate. The above method can improve the overall performance and stability of large-scale data uploads in a cloud database environment.

[0056] In some embodiments, initializing the working parameters based on the backup type of the at least one file to be backed up includes: Obtain the backup type and historical data backup information of the at least one file to be backed up; The working parameters are initialized based on the backup type, the backup information, and the heuristic solution method. The backup information includes the working parameters and backup upload rate for at least one period of historical data backups.

[0057] Here, the heuristic solution method is an experience- or rule-based search strategy often used to find approximate optimal solutions to complex problems. In this application, the heuristic solution method is used to generate the initial working parameter configuration, and the system avoids the inefficient search process caused by complete randomization by adopting this method. The heuristic solution method uses successful cases in the backup information of historical data backups as reference points to quickly locate initial parameter combinations with high probability. The heuristic solution method can significantly shorten the convergence time of the algorithm and increase the probability of finding an effective solution. In actual implementation, the heuristic solution method combines the backup type and historical data in the backup information of historical data backups to make decisions. For example, if the current backup type is a logical backup, the system can prioritize parameter combinations with high success rates used in past logical backup processes. In this way, the system can speed up the generation of initial parameters and improve subsequent execution efficiency. In the embodiments of this application, by obtaining the backup type and the backup information of historical data backups, and combining them with the heuristic solution method to initialize the working parameters, a reasonable and efficient initial parameter configuration can be generated quickly without relying on large-scale computation. By acquiring backup information based on backup type and historical data backups, and combining this with heuristic solution methods, the time cost of blind searching can be effectively reduced, thereby accelerating the startup speed of the entire backup upload process and improving overall upload performance and resource utilization.

[0058] In summary, the steps described above work together to construct an efficient and intelligent initial parameter configuration system. First, by clearly defining the type of file to be backed up, its characteristics and upload requirements are determined. Second, by incorporating backup information from historical data backups, existing successful experiences and data foundations are obtained. Finally, through heuristic solution methods, this information is transformed into specific initial parameter configurations. This progressive approach not only enhances the system's intelligence level but also ensures the scientific rigor and practicality of the parameter configuration.

[0059] In some embodiments, the preset optimization model employs FOA or an optimization algorithm combining FOA and simulated annealing.

[0060] Fruit fly auto-analysis (FOA) is a swarm intelligence optimization algorithm inspired by the foraging behavior of fruit flies. FOA searches for the optimal solution to a problem through random search and iterative updates of the fruit fly population. FOA has advantages such as low computational cost and fast convergence speed, making it particularly suitable for multidimensional nonlinear optimization problems. In this application, FOA is used to dynamically adjust multiple parameters in the upload strategy to optimize file chunk size, number of concurrent threads, and retry strategy, thereby improving upload efficiency and reducing transmission latency. The main steps of FOA include: initializing the position of the fruit fly population; continuously updating the fruit fly position through olfactory search (local search) and visual search (global search); and performing iterative optimization operations until a preset termination condition is reached. In this application, FOA is used to find the optimal combination of file upload strategy parameters, including file chunk size, number of concurrent threads, and retry interval. By dynamically adjusting the file chunk size, number of concurrent threads, and retry interval parameters, this application can adapt to different network environments and changes in local disk performance, thereby improving overall upload efficiency.

[0061] To further enhance the optimization capability and stability of FOA, this application also introduces the Simulated Annealing (SA) algorithm. Simulated Annealing is a probabilistic search algorithm based on the physical annealing process, which effectively avoids getting trapped in local optima. In this application, SA serves as an auxiliary mechanism to balance the global and local search capabilities of FOA. Specifically, in each iteration, when FOA generates a new solution, SA decides whether to accept this new solution based on the current temperature and fitness difference. The mechanism by which SA decides whether to accept a new solution helps prevent premature convergence and increases the probability of finding the global optimum. The combination of FOA and SA enables the optimization model to run more stably in complex environments and provides higher optimization accuracy. Applying the combination of FOA and SA to the system allows the system to continuously adjust its upload strategy based on actual operating conditions, ensuring good upload performance in different scenarios.

[0062] In this embodiment, an optimization algorithm combining FOA and SA is employed to dynamically adjust key parameters in the upload strategy, optimizing file chunk size, number of concurrent threads, and retry strategy. The system can significantly improve upload efficiency, reduce transmission latency, and lower the failure rate caused by network fluctuations or resource contention. The system achieves a more stable upload process, improving overall system performance and user experience, and better adapts to backup needs in large-scale clusters.

[0063] In practice, the synergy between FOA and SA lies in the following: FOA is responsible for quickly exploring the solution space and providing initial solutions, while SA is used to evaluate and select initial solutions, and SA can avoid getting trapped in local optima. After combining FOA and SA, the optimization model has both strong global search capabilities and the ability to make fine adjustments locally, thereby improving the optimization quality of the uploaded policy parameters.

[0064] In summary, this embodiment achieves dynamic optimization of upload strategy parameters by constructing an optimization model that includes FOA and / or a combination of FOA and SA, thereby improving the efficiency and stability of the data backup process.

[0065] In some embodiments, the preset optimization model employs an optimization algorithm that combines FOA with simulated annealing, wherein the environmental parameters further include annealing temperature and annealing rate.

[0066] Here, annealing temperature refers to a control parameter in the simulated annealing algorithm, used to adjust the probability of accepting inferior solutions during the search process. Setting a higher annealing temperature in the initial stage helps the algorithm to explore a wide area in the early stages; as the number of iterations increases, the annealing temperature is gradually decreased, making the algorithm more inclined to converge to the optimal solution. Annealing temperature is usually expressed numerically, such as T=1000, T=100, etc. A larger value makes the algorithm more likely to accept inferior solutions, while a smaller value makes the algorithm more rigorous in selecting superior solutions. Dynamic adjustment of the annealing temperature plays a crucial role in helping the algorithm escape local optima.

[0067] The annealing rate is a parameter in simulated annealing algorithms that determines the rate at which the annealing temperature decreases, typically ranging from 0 to 1. For example, if the annealing rate is 0.95, the annealing temperature will be multiplied by 0.95 after each iteration, thus achieving a gradual temperature decay. When the annealing rate is high, the annealing process proceeds more slowly, allowing the simulated annealing algorithm to retain more global search capabilities; when the annealing rate is low, the annealing process proceeds more quickly, and the simulated annealing algorithm tends to converge to a local optimum. By setting the annealing rate appropriately, the efficiency of the simulated annealing algorithm can be ensured while improving the quality of its optimal solution search.

[0068] There is a close logical relationship between annealing temperature and annealing rate. Annealing temperature determines the probability of accepting a poor solution in the current iteration, while annealing rate determines the rate of change of annealing temperature. Both jointly affect the convergence behavior of the algorithm. Specifically, in the early stages of the algorithm, a higher annealing temperature allows for the acceptance of poorer solutions, thus expanding the search space; as the annealing temperature decreases (controlled by the annealing rate), the algorithm gradually shrinks the search radius, entering a local fine-grained search phase. This mechanism effectively balances the contradiction between global search and local search.

[0069] In practical implementation, the combined use of annealing temperature and annealing rate can enhance the robustness and stability of the FOA algorithm. In cloud database backup file upload tasks, the system dynamically adjusts the annealing temperature and annealing rate based on the current network status, disk I / O performance, and historical upload data performance. By adjusting annealing parameters, the system optimizes the number of concurrent threads, fragment size, and retry strategy, ultimately achieving the goals of shortest upload time, highest resource utilization, and fewest retransmissions.

[0070] In this embodiment, annealing temperature and annealing rate are introduced as key parameters of the optimization model into the FOA algorithm. Introducing annealing temperature and annealing rate as key parameters of the optimization model into the FOA algorithm enhances the global search capability of the FOA algorithm and avoids premature convergence, thereby improving the optimization effect and more effectively solving the complex optimization problem of backup file upload in large-scale clusters.

[0071] In summary, this embodiment combines the fruit fly optimization algorithm with the simulated annealing algorithm, utilizing two key parameters—annealing temperature and annealing rate—to regulate the algorithm's search behavior. In practical implementation, the optimization process first initiates a wide-area search with a high annealing temperature, gradually accepting solutions from different approaches. Subsequently, as the annealing temperature decreases according to a set annealing rate, the algorithm gradually focuses on the local optimum, completing a refined search. Simultaneously, the annealing temperature and annealing rate parameters are dynamically adjusted based on the real-time operating environment to ensure that the optimization results more closely reflect the needs of real-world application scenarios. Therefore, the method of combining the fruit fly optimization algorithm with simulated annealing not only improves the algorithm's adaptability but also enhances its practical value in complex systems.

[0072] In some embodiments, the method further includes: The period is determined based on the interval at which heartbeat packets are sent to the storage server; the storage server is used to generate backup files for the data backup. The step of obtaining the environmental parameters of at least one file to be backed up during the data backup process in the previous cycle includes: The first environmental parameter is generated based on the network latency and / or packet loss rate obtained from the heartbeat packet; The second environment parameter is generated based on the client's IOPS and / or throughput from the previous cycle.

[0073] Here, network latency refers to the time spent in the process of network transmission, usually measured in milliseconds (ms). For example, 100ms means that the network latency is 100 milliseconds.

[0074] Packet loss rate refers to the proportion of data packets lost during network transmission out of the total number of data packets. It is usually expressed as a percentage (%), for example, 1% means that the packet loss rate is 1%.

[0075] IOPS refers to the number of read and write operations per second, and is used to measure the performance of storage devices.

[0076] Throughput refers to the amount of data transmitted per unit of time, usually measured in bytes per second (B / s) or megabytes per second (MB / s).

[0077] In this embodiment, the period is determined based on the heartbeat packet interval, and network latency and packet loss rate are obtained to generate first environmental parameters, while IOPS and throughput are obtained to generate second environmental parameters. The system can evaluate the current network conditions and storage performance in real time, thereby dynamically adjusting the sharding strategy and improving the success rate and efficiency of data backup.

[0078] In practice, the above steps are closely related. First, the system determines the cycle by setting the heartbeat packet sending interval, and accordingly sets periodic tasks, making the entire data backup process predictable and controllable in terms of time. Next, within each cycle, the system uses network latency and packet loss rate information obtained from heartbeat packets, combined with the client's IOPS and throughput metrics from the previous cycle, to comprehensively generate parameters reflecting the overall environment and operational status. These parameters, reflecting the overall environment and operational status, constitute a comprehensive evaluation of the system's operating status, thereby supporting subsequent intelligent adjustments to the sharding strategy and ultimately achieving more efficient and stable data backup results.

[0079] In summary, this embodiment, by reasonably setting the heartbeat packet sending frequency and combining multi-dimensional indicators of network quality and storage performance, achieves accurate perception and dynamic response to environmental parameters during data backup, thereby improving the system's intelligence level and reliability.

[0080] In some embodiments, determining the working parameters for the current cycle of the data backup process based on the environmental parameters, the working parameters, and a preset optimization model from the previous cycle includes: The first environmental parameter is converted into a first input value based on a first function, and the second environmental parameter is converted into a second input value based on a second function; wherein, the first function represents the correlation between the first environmental parameter and the fragment size, and the second function represents the correlation between the second environmental parameter and the fragment size; The first input quantity, the second input quantity, the fragment size of the previous cycle, and the file size of the file to be backed up are input into the preset optimization model to obtain the working parameters of the current cycle of the data backup process.

[0081] For example, the first function can represent the positive correlation between fragment size and packet loss rate and network latency, and the second function can represent the correlation between fragment size and IOPS and throughput. Thus, by transforming the first environmental parameters using the first function to obtain the first input quantity, and by transforming the second environmental parameters using the second function to obtain the second input quantity, the first input quantity, the second input quantity, the fragment size of the previous cycle, and the file size of the file to be backed up are input into a preset optimization model. This allows for comprehensive consideration of multiple factors, achieving a balance between resource utilization and transmission success rate, and significantly improving the stability and reliability of data backup.

[0082] The preset optimization model is a mathematical model or algorithmic framework used to comprehensively evaluate and optimize the data backup process. This preset optimization model can include various input variables and output target parameters. Through the calculation and optimization of the preset optimization model, the working parameters for the current cycle can be determined more efficiently.

[0083] In this embodiment of the application, by introducing an optimization model of environmental parameters and working parameters, the method can achieve fine-grained control and optimization of the data backup process. The method can improve the success rate and efficiency of data upload, and can also better adapt to the complex environment and needs of large-scale clusters.

[0084] In some embodiments, driving the streaming backup tool to perform the data backup based on the current cycle's working parameters includes: The data in the data backup process is divided into multiple backup tasks based on the fragment size of the current period. The multiple backup tasks are added to the task queue in a specified order, and the thread pool is invoked to execute the backup tasks in the task queue concurrently based on the number of concurrent threads in the current period.

[0085] In this embodiment, the chunk size refers to the data capacity of each small block when the backup file to be uploaded is divided into several small blocks according to certain rules. The chunk size setting directly affects upload efficiency and resource utilization. Smaller chunks can improve concurrency but increase the number of requests, thus incurring additional overhead; while larger chunks reduce the number of requests but may result in higher retransmission costs when a single request fails. Therefore, a reasonable chunk size should be dynamically adjusted based on factors such as network conditions and disk I / O performance.

[0086] In practice, the system uses a monitoring module to obtain metrics such as network latency, packet loss rate, and local disk IOPS and throughput for the current period. Combined with the sharding strategy generated by the FOA optimizer, it calculates the optimal shard size and divides the entire backup file into multiple logical backup tasks based on this size. These multiple logical backup tasks can be processed, transmitted, and retried independently, thereby improving overall upload efficiency and fault tolerance.

[0087] Here, a task queue refers to a queue structure used to manage tasks awaiting execution. The role of the task queue is to ensure the orderliness and controllability of tasks. In this embodiment, multiple generated backup tasks are added to the task queue sequentially in a specified order (e.g., file order or priority sorting). The task queue supports First-In-First-Out (FIFO) or other scheduling strategies to ensure that tasks can be rationally arranged and executed efficiently.

[0088] Concurrent thread count refers to the number of threads that a system is allowed to run simultaneously within the current cycle, and it determines the system's parallel processing capability. The system configures the thread pool based on the optimal thread count provided by the FOA optimizer and applies this optimal thread count to the execution of the task queue. A thread pool is a pre-created and reused collection of threads, which reduces the overhead of thread creation and destruction, improving system response speed and resource utilization.

[0089] In practical applications, the system utilizes a multi-threaded mechanism to retrieve backup tasks from the task queue and submit them to the thread pool for execution. Each thread is responsible for handling one backup task, releasing its resources after completion. If a task fails to upload, it can be re-added to the task queue according to a preset retry strategy, awaiting the next execution opportunity. This multi-threaded backup upload mechanism not only improves the overall efficiency of backup uploads but also enhances the system's stability and fault tolerance.

[0090] In this embodiment, the system segments the backup file according to a dynamically adjusted segment size and adds these segment tasks to a task queue in an orderly manner. The system then uses a thread pool to achieve concurrent execution. This approach fully utilizes system resources, improves upload efficiency, reduces the failure rate caused by network fluctuations or resource contention, significantly reduces the time required for backup uploads, and enhances data reliability and business continuity in the cloud database environment.

[0091] In some embodiments, the method further includes: If it is determined that the upload of any backup task's corresponding fragment fails, the failed backup task is added to the task queue based on the retry policy of the current period; wherein, the retry policy includes: the number of retries and / or the retry interval.

[0092] When a segment upload fails, the system will decide whether to re-add the failed backup task to the task queue for retransmission based on a preset retry policy. The retry policy is a mechanism to control how to handle upload failures, and it includes two main parameters: the number of retries (i.e., the maximum number of attempts allowed) and the retry interval (i.e., the time interval between each retrieval). These two parameters can be set individually or in combination to adapt to different network environments and business needs.

[0093] For example, in poor network conditions, the system can improve the upload success rate by increasing the number of retries and appropriately extending the retry interval in the retry strategy. Conversely, in more stable network conditions, the system can reduce the number of retries and shorten the retry interval to speed up the overall upload speed. Adjusting the retry strategy configuration helps achieve optimal upload performance in different scenarios.

[0094] Specifically, when an upload of a data segment fails, the system first checks if the segment has reached its maximum retry limit. If the segment has not reached the maximum retry limit, the system waits for a specified period according to the retry interval in the set retry policy before putting the corresponding upload task back into the task queue for retransmission by the thread pool. If the segment has reached the maximum retry limit but still cannot be uploaded successfully, the system may record the specific information of the upload failure and trigger alarms or logging functions for subsequent manual intervention or further optimization of the upload strategy.

[0095] By introducing a retry-based fragment retransmission mechanism, this application effectively addresses upload failures caused by uncertainties such as network fluctuations and resource contention. Compared to the traditional full retransmission method, the current-cycle-based retry strategy only retransmits failed fragments, avoiding unnecessary duplicate transmissions and significantly improving upload efficiency and system stability. Furthermore, by dynamically adjusting relevant parameters in the retry strategy using an improved FOA algorithm, resource allocation and time management during the retransmission process can be further optimized, achieving even more efficient upload performance.

[0096] In summary, in this embodiment, the system adds failed backup tasks to the task queue using a retry strategy based on the current cycle. This strategy ensures that failed upload segments are retransmitted promptly, preventing the entire file upload from failing due to a single failed segment, thus improving the overall success rate and stability of backup uploads.

[0097] In some embodiments, if the backup type of the file to be backed up is a binary log backup type, the method further includes: Obtain the high-frequency data distribution rate corresponding to the shards of the binary log backup type during the data backup process of the previous cycle; The retry strategy for sharding the binary log backup type is determined based on the high-frequency data distribution rate.

[0098] Here, the high-frequency data distribution rate refers to the proportion of frequently accessed or transmitted data areas in the backup file within the entire data range over a certain period. Monitoring tools can statistically analyze the proportion of frequently read or transmitted data in each binary log backup type shard, thereby determining whether the binary log backup type shard belongs to a hotspot area. For example, during database operation, some transaction logs may be frequently modified or queried, and some transaction logs will exhibit a high access frequency during backup. The system can calculate the high-frequency data distribution rate of each binary log backup type shard by using the data access records of historical backup tasks.

[0099] A retry strategy refers to the mechanism adopted for re-uploading shards of different binary log backup types in the event of upload failure. The retry strategy includes the number of retransmissions, the retransmission interval, and priority adjustments. In this embodiment, the retry strategy for the binary log backup type shards will be dynamically formulated based on the high-frequency data distribution rate of the shards. Specifically, for shards of binary log backup types with a higher high-frequency data distribution rate, the corresponding retransmission priority is higher, and a shorter retransmission interval and a higher number of retransmissions may be used. Since high-frequency data usually has high business value, upload failure can lead to decreased recovery capabilities and affect system reliability.

[0100] In practical applications, when the upload of a certain fragment fails, the system first checks the high-frequency data distribution rate of the binary log backup type fragment corresponding to that fragment. If the binary log backup type fragment corresponding to that fragment belongs to the high-frequency data area, the system immediately adds the fragment to the high-priority queue and attempts multiple times according to the preset retry strategy; conversely, if the binary log backup type fragment corresponding to that fragment belongs to the low-frequency data area, the system can appropriately reduce the retransmission priority of that fragment and extend the retransmission interval of that fragment to save system resources and improve overall upload efficiency.

[0101] By employing dynamically defined retry policies, the system can prioritize the successful upload of critical data within limited network bandwidth and thread resources, thereby improving the stability and reliability of the entire backup process. Simultaneously, the dynamically defined retry policies can reduce unnecessary duplicate upload operations, preventing resource waste.

[0102] Furthermore, by combining the improved FOA algorithm, the system can dynamically adjust the upload parameters of each binary log backup type shard based on the current network status and local disk I / O performance, including the number of concurrent threads and shard size, thereby achieving more intelligent upload scheduling and optimization.

[0103] In summary, in this embodiment, by obtaining the high-frequency data distribution rate corresponding to the binary log backup type shards during the previous data backup process, and formulating a targeted retry strategy based on this high-frequency data distribution rate, priority can be given to ensuring the upload stability of the corresponding high-frequency data, thereby effectively improving the backup success rate and system reliability, and further optimizing the efficiency and resource utilization of the overall backup process.

[0104] The present application will now be described in further detail with reference to an application embodiment.

[0105] The backup device in this application example includes: a file analyzer, a FOA optimizer, and a task executor. A schematic diagram of the overall architecture is shown below. Figure 2As shown.

[0106] The file analyzer reads the file attributes of the file to be uploaded and, based on conditions such as file type, file size, and chunk size, splits the file task into N upload tasks.

[0107] The FOA optimizer employs an improved FOA optimization algorithm. The FOA optimizer iteratively updates the fragment size partitioned by the file analyzer and finds the appropriate number of concurrent threads, retries, and retry intervals.

[0108] The task executor adds the split upload tasks to the task queue in a certain order and executes them concurrently using a thread pool. If an upload task fails and the user has set up a retry mechanism, the system adds the failed upload task to the end of the task queue according to the retry policy output by the FOA optimizer.

[0109] Reference Figure 2 In large-scale clusters, cloud databases often have multiple instances executing upload tasks, such as instance A, instance B, and instance C. A file analyzer is used to divide the upload tasks of each instance based on its respective shard size, resulting in multiple upload tasks. For example, instance A corresponds to tasks A1~A3, instance B to task B1, and instance C to tasks C1~C2. A FOA optimizer is used to dynamically adjust the working parameters during the backup process, optimizing at least one of the following: shard size, number of concurrent threads, and retry strategy. A task executor adds the divided upload tasks to the task queue in a specified order (e.g., file order or priority sorting) and performs multi-threaded concurrent uploads based on a thread pool. Finally, on the target storage server, based on merge verification, backup file A is generated for instance A, backup file B is generated for instance B, and backup file C is generated for instance C. The target storage server can be a single server or a server cluster; this application does not limit this.

[0110] The data backup method in this application embodiment specifically includes the following steps: Step 1: Configure file strategies and divide file tasks in the file analyzer. Here, upload tasks are categorized into four main types based on different backup types: full backups, incremental backups, logical backups, and binlogs file backups. Within the upload task category, the general approach is as follows: because full backups and instance-generated binlogs files are typically large, the system uses xtrabackup's xbstream streaming backup method when performing backup operations.

[0111] In addition, the file analyzer initializes a connection channel and calculates the number of fragments based on the estimated size of the file to be backed up. This estimated size comes from the file size after the last successful transfer of a backup file of the same type. Furthermore, when transferring binlogs files, it is necessary to determine the high frequency data distribution rate (HFDDR) and calculate the fragment retransmission priority of each small file based on the size of the file to be uploaded, the start time, and the end time. For files with a higher high frequency data distribution rate, the fragment retransmission priority of that file is higher.

[0112] For example, using rate hf The distribution rate of high-frequency data is represented by the following formula (1): (1) in, Size (F) The size of the file to be uploaded. Start The start time, End End time, Duration This refers to the transmission duration.

[0113] In addition, the client can periodically send a heartbeat packet to the object storage server at set intervals 't'. This heartbeat packet does not contain actual file data and is only used to detect network connection status. The client can use the network latency, packet loss rate, and other parameters obtained through the heartbeat mechanism to assess the current network conditions in real time and dynamically adjust its sharding strategy.

[0114] For example, a mathematical model can be used to approximate the positive correlation between file fragment size and packet loss rate and network latency (i.e., the first function mentioned above).

[0115] For example, packet loss rate loss_rate With fragment size chunk_size The relationship is expressed by the following formula (2): (2) in, α、β These are the proportionality coefficients.

[0116] For example, network latency latency With fragment size chunk_size The relationship is expressed by the following formula (3): (3) in, γ δ represents the delay per unit fragment size, and δ is a fixed delay.

[0117] Furthermore, the aforementioned first function can be expressed using the following formula (4): (4) in, total_cost This is the overall cost corresponding to the fragment size determined based on network conditions (i.e., the aforementioned first input). w 1. w 2 is the weight value. w 1 represents the packet loss rate. loss_rate The relative weight in this comprehensive cost w 2 indicates network latency latency The relative weight in the overall cost.

[0118] Understandably, in the process of evaluating network communication performance, the system sets network packet loss rate and network latency as influencing factors, and assigns numerical weight parameters to these two factors to represent their relative importance in the overall cost.

[0119] For example, historical I / O performance metrics such as IOPS and throughput (TP) of the instance's local disk can be obtained through monitoring tools. Based on the current I / O performance, the system assesses whether the shard size needs to be adjusted.

[0120] For example, the aforementioned second function can be represented by the following formula (5): (5) in, IO_pi Corresponding to the second input quantity mentioned above, it can be IOPS capability and throughput. TP A set of.

[0121] It's important to note that the impact of shard size on IOPS is not linear. Generally, smaller shards may result in more IO requests because each shard requires a separate IO operation. However, as the shard size increases to a certain point, IOPS may become saturated due to limitations in disk physical performance.

[0122] in, IOPS_total The total IOPS capacity of the disk can be expressed by the following formulas (6) to (7): (6) (7) in, N It is the number of shards. N_max This is the upper limit of the number of shards that causes IOPS to reach saturation. IOPS_per_share This is the IOPS size for each shard (the average value can be obtained from cloud monitoring). total_data This is the total amount of data, size_ per_ share It is the size of each slice.

[0123] Similarly, throughput can be defined. TPThe relationship with the fragment size is shown in the following formula (8): (8) in, Bandwidth_limit It's a limitation of disk or network bandwidth.

[0124] If the system detects that I / O performance has become a bottleneck, and optimizing database queries, indexes, and other operations cannot effectively alleviate the performance problem, then adjusting the fragment size can be considered to improve upload performance.

[0125] For example, xbstream packages the backup content and transmits it to the compression tool following the pipeline. This compression uses a proprietary, efficient format to compress the information output by xtrabackup, helping to reduce disk space usage during the backup process and lower network bandwidth requirements. This application allows the code corresponding to this approach to be integrated and encapsulated in a software development kit (SDK) during the backup process design, thereby enabling simultaneous backup and upload of backup files. That is, while xbstream is working, the output of xbstream is input into the task executor for processing.

[0126] Step 2: Define the optimization objective and parameters The file analyzer processes the target file (the file to be backed up), collects file attributes and fragment information, and then sends this information as input data to the FOA optimizer. For example, the input data includes:

[0127] in, Input FOA For the input data set, Size(backupFiles) The size of the file to be backed up. chunk_size For the size of the slice, rate hf For high-frequency data distribution rate, total_cost The overall cost corresponding to the fragment size determined based on network conditions. IO_pi For IOPS capacity and throughput TP A set of.

[0128] In large-scale clusters, the FOA optimizer can determine the working parameters for the current cycle based on the input data, namely, at least one of the following: shard size, number of concurrent threads, and retry strategy. It can comprehensively consider multiple factors to achieve a balance between resource utilization and transmission success rate, thereby significantly improving the stability and reliability of data backup.

[0129] Here, Forecasting Algorithm (FOA) is an emerging swarm intelligence algorithm with powerful global optimization capabilities, relatively low computational cost, and low algorithmic complexity, making it highly suitable for optimizing the process of uploading cloud database backup files to object storage. FOA finds the optimal solution by adjusting multiple parameters in the system, thereby improving upload speed, reducing latency, and optimizing resource utilization efficiency.

[0130] In Free Optimization (FOA), the location of the food is analogous to the optimal solution to an optimization problem, while the fruit fly's current position represents a random solution. The FOA algorithm finds the optimal solution through continuous movement and iterative optimization of the fruit fly's position. The FOA algorithm has drawbacks, including its inability to handle negative optimal values ​​and its relatively weak ability to solve complex, high-dimensional, and nonlinear optimization problems. However, for the input set mentioned above, there are no negative solutions and it does not involve handling complex, high-dimensional problems.

[0131] It should be noted that traditional FOA algorithms, in practical applications, still have drawbacks such as insufficient local search capability, unstable optimization results, and a tendency to converge prematurely. Specifically, the traditional FOA algorithm consists of the following steps: Step 1: Initialize the fruit fly colony The system randomizes the location information of each individual in the fruit fly population based on the location range (LR) of the fruit fly population, as shown in the following formula (9).

[0132] (9) in, X axis , Y-axis This represents the location information of the randomized individual, corresponding to two coordinate positions.

[0133] Step 2: Olfactory search (local search) The system calculates the flavor concentration of each fruit fly in the current population by statistically analyzing the new location of individual fruit fly i, its distance from the origin, and the flavor concentration corresponding to its location. The fruit flies will then fly in random directions and distances (FR, Flying Range) from their current location to explore new locations and select the fruit fly with the best flavor concentration and location in the current population as a local optimum.

[0134] The new position of fruit fly individual i is represented by the following formula (10): (10) The new location of the individual fruit fly is as follows: Xi , Yi This corresponds to two coordinate positions.

[0135] Step 3: Visual Search (Global Search) A swarm of fruit flies searches for the location with the best taste and identifies it as the current local optimum. Individual fruit flies in other locations within the swarm gradually move towards and gather at this optimal taste location through visual perception.

[0136] Step 4: Iterative Optimization Repeat Step 2 and Step 3 until the maximum number of iterations, maxgen, is reached.

[0137] Because traditional FOA algorithms exhibit insufficient local search capabilities and unstable optimization results during the optimization process, and are prone to premature convergence, this application example designs an improved FOA algorithm. The aim is to enhance global search capabilities and stability by introducing a new mechanism, thereby achieving the goal of finding an optimal equilibrium solution.

[0138] For example, the following optimization objectives are determined: 1) Minimize the total time for uploading backup files to object storage; 2) Maximize network bandwidth utilization during the upload process; 3) Reduce the number of retries caused by network fluctuations or resource contention.

[0139] Optimize parameters: 1) Number of concurrent threads: Controls the number of file fragments uploaded simultaneously, determined by... IO_pi Decide.

[0140] 2) Chunk Size: Dynamically adjust the file chunk size for each upload task. The chunk size is determined by the input data in each round. Input SA-FOA ; Input SA-FOA This represents the input data for the improved FOA optimizer.

[0141] 3) Retry strategy: The retry interval and number of retries when an upload fails are determined by... rate hf Decide.

[0142] For example, the FOA model improvement scheme and its construction are as follows: Compared to the traditional FOA model construction steps, the improved FOA model incorporates the following optimizations: 1) Heuristic initialization is adopted, and a large amount of data on backup upload speed and performance under the current environment is collected based on current experience. Input FOA Information, select the groups with the fastest average rate. Input FOAFor the initial input, random location information will no longer be used. For example, in Company Instance A, with a data volume in the TB range and an average full backup time of 10.3 hours over the past 20 backups, the initial heartbeat packet sending interval... ,but .

[0143] 2) Each iteration updates the parameters based on the heartbeat packets, rather than being completely random, thus limiting the search radius of individual fruit flies in a single iteration. FR This allows for a better balance between the global and local search capabilities of the fruit fly optimization algorithm during the optimization process.

[0144] 3) Introduce the Simulated Annealing (SA) algorithm to further balance the capabilities of global and local searches. Specifically, in Step 1 of the aforementioned FOA algorithm, the initial annealing temperature of the annealing algorithm is randomly set. T 0. Repeat Step 4. When the fruit fly population generates a new solution in some way, calculate the fitness difference ΔE between the new solution and the current best solution. If the fitness difference ΔE is negative, the new solution is accepted unconditionally; if the fitness difference ΔE is positive, meaning the new solution is poor, the new solution is accepted with a certain probability according to the Metropolis criterion. This probability value is related to the fitness difference ΔE and the current annealing temperature. T 0 related.

[0145] Update the FOA optimizer input as follows:

[0146] in, λ The annealing rate is defined, with a value range of [0,1]. The number of perturbations for the simulated annealing algorithm is set to L = the upper limit of the number of iterations, maxgen.

[0147] For example, the FOA optimizer can be monitored and evaluated, including performance monitoring, effect evaluation, and feedback adjustment. Performance monitoring refers to real-time monitoring of changes in the fragment file transfer rate within each t-time interval; effect evaluation refers to comparing the optimized upload strategy with the original strategy to assess the optimization effect; and feedback adjustment refers to making adjustments to the FOA optimizer based on the evaluation results to further improve the optimization effect.

[0148] Step 3: Execute the task and upload. 1) Create a thread pool-based task executor Based on the parameters output by the FOA optimizer, determine the shard size and number of concurrent threads for each t interval, as well as the queue priority, start the task executor, and execute the tasks in the task queue.

[0149] 2) Adding and reading tasks Users add tasks; users add one or more tasks to the task queue of the task executor.

[0150] 3) Task Transfer The system executes file tasks in the task queue in parallel in sequence.

[0151] 4) Retry if it fails If a segment fails to upload, the system adds it to the task queue according to the priority of the failure, and the system will prioritize allocating an idle thread to retransmit the segment.

[0152] In order to implement the method of the embodiments of this application, the embodiments of this application also provide a data backup device, which corresponds to the data backup method described above. The steps in the embodiments of the data backup method described above are also fully applicable to the embodiments of this data backup device.

[0153] Reference Figure 3 The data backup device includes: an acquisition module 301, an optimization module 302, and a driver module 303. The acquisition module 301 acquires the environmental parameters and operating parameters of at least one file to be backed up in the previous backup cycle during the data backup process. The optimization module 302 determines the operating parameters of the current cycle of the data backup process based on the environmental parameters, the operating parameters, and a preset optimization algorithm from the previous cycle. The driver module 303 drives the streaming backup tool to perform the data backup based on the operating parameters of the current cycle. The environmental parameters include one or more of the following: a first environmental parameter characterizing network performance and a second environmental parameter characterizing the input and output performance of the client to which the file to be backed up belongs. The operating parameters include one or more of the following: the fragment size for partitioning, the number of concurrent threads for concurrent uploads, and the retry strategy for fragment retransmission.

[0154] In some embodiments, the optimization module 302 is further configured to: initialize the working parameters based on the backup type of the at least one file to be backed up; the driver module 303 is further configured to: drive the streaming backup tool to start the data backup based on the initialized working parameters.

[0155] In some embodiments, the optimization module 302 initializes the working parameters based on the backup type of the at least one file to be backed up, including: Obtain the backup type and historical data backup information of the at least one file to be backed up; The working parameters are initialized based on the backup type, the backup information, and the heuristic solution method. The backup information includes the working parameters and backup upload rate for at least one period of historical data backups.

[0156] In some embodiments, the preset optimization model employs FOA or an optimization algorithm combining FOA and simulated annealing.

[0157] In some embodiments, the preset optimization model employs an optimization algorithm that combines FOA with simulated annealing, wherein the environmental parameters further include annealing temperature and annealing rate.

[0158] In some embodiments, the acquisition module 301 is further configured to: The period is determined based on the interval at which heartbeat packets are sent to the storage server; the storage server is used to generate backup files for the data backup. The acquisition module 301 acquires the environmental parameters of at least one file to be backed up in the previous cycle during the data backup process, including: The first environmental parameter is generated based on the network latency and / or packet loss rate obtained from the heartbeat packet; The second environment parameter is generated based on the client's IOPS and / or throughput from the previous cycle.

[0159] In some embodiments, the optimization module 302 is specifically used for: The first environmental parameter is converted into a first input value based on a first function, and the second environmental parameter is converted into a second input value based on a second function; wherein, the first function represents the correlation between the first environmental parameter and the fragment size, and the second function represents the correlation between the second environmental parameter and the fragment size; The first input quantity, the second input quantity, the fragment size of the previous cycle, and the file size of the file to be backed up are input into the preset optimization model to obtain the working parameters of the current cycle of the data backup process.

[0160] In some embodiments, the driver module 303 is specifically used for: The data in the data backup process is divided into multiple backup tasks based on the fragment size of the current period. The multiple backup tasks are added to the task queue in a specified order, and the thread pool is invoked to execute the backup tasks in the task queue concurrently based on the number of concurrent threads in the current period.

[0161] In some embodiments, the driver module 303 is further configured to: If it is determined that the upload of any backup task's corresponding fragment fails, the failed backup task is added to the task queue based on the retry policy of the current period; wherein, the retry policy includes: the number of retries and / or the retry interval.

[0162] In some embodiments, if the backup type of the file to be backed up is a binary log backup type, the acquisition module 301 is further configured to: acquire the high frequency data distribution rate (HFDDR) corresponding to the shards of the binary log backup type during the data backup process of the previous cycle; the optimization module 302 is further configured to: determine the retry strategy of the shards of the binary log backup type based on the high frequency data distribution rate.

[0163] In practical applications, the acquisition module 301 (corresponding to the aforementioned file analyzer), the optimization module 302 (corresponding to the aforementioned FOA optimizer), and the driver module 303 (corresponding to the aforementioned task executor) can be implemented by the processor in the backup device. Of course, the processor needs to run the computer program in the memory to implement its functions.

[0164] It should be noted that the data backup device provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the data backup device and the data backup method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0165] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device, namely the aforementioned backup device. Figure 4 The diagram shows only an exemplary structure of the electronic device, not the entire structure; implementation is possible as needed. Figure 4 The structure shown may be part or all of the structure.

[0166] like Figure 4 As shown, the electronic device 400 provided in this application embodiment includes: at least one processor 401, a memory 402, a user interface 403, and at least one network interface 404. The various components in the electronic device 400 are coupled together via a bus system 405. It can be understood that the bus system 405 is used to implement communication between these components. In addition to a data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 The general designated all buses as Bus System 405.

[0167] The user interface 403 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0168] The memory 402 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0169] The data backup method disclosed in this application can be applied to or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the data backup method can be completed by integrated logic circuits in the hardware of processor 401 or by instructions in software form. The processor 401 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically memory 402. Processor 401 reads information from memory 402 and, in conjunction with its hardware, completes the steps of the data backup method provided in the embodiments of this application.

[0170] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0171] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0172] In an exemplary embodiment, this application also provides a computer storage medium, specifically a computer-readable storage medium, such as a memory 402 storing a computer program, which can be executed by a processor 401 of an electronic device 400 to complete the steps described in the method of this application embodiment. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0173] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 401 of an electronic device 400 to perform the steps described in the method of this application embodiment.

[0174] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0175] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0176] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data backup method, characterized in that, The method for performing fragmented data backup of at least one file to be backed up based on a streaming backup tool includes: Obtain the environmental and operational parameters of at least one file to be backed up in the previous cycle during the data backup process; Based on the environmental parameters, operating parameters, and preset optimization model of the previous cycle, the operating parameters for the current cycle of the data backup process are determined. Based on the current cycle's working parameters, the streaming backup tool is driven to perform the data backup. The environmental parameters include one or more of the following: a first environmental parameter characterizing network performance and a second environmental parameter characterizing the input and output performance of the client to which the file to be backed up belongs; the working parameters include one or more of the following: the fragment size for partitioning, the number of concurrent threads for concurrent upload, and the retry strategy for fragment retransmission.

2. The method according to claim 1, characterized in that, The method further includes: Initialize the working parameters based on the backup type of the at least one file to be backed up; Based on the initialized operating parameters, the streaming backup tool is driven to start the data backup.

3. The method according to claim 2, characterized in that, The initialization of the working parameters based on the backup type of the at least one file to be backed up includes: Obtain the backup type and historical data backup information of the at least one file to be backed up; The working parameters are initialized based on the backup type, the backup information, and the heuristic solution method. The backup information includes the working parameters and backup upload rate for at least one period of historical data backups.

4. The method according to claim 1, characterized in that, The preset optimization model adopts the Fruit Fly Optimization Algorithm (FOA) or an optimization algorithm combining FOA and simulated annealing.

5. The method according to claim 4, characterized in that, The preset optimization model adopts an optimization algorithm that combines FOA and simulated annealing algorithm. The environmental parameters also include annealing temperature and annealing rate.

6. The method according to claim 1, characterized in that, The method further includes: The period is determined based on the interval at which heartbeat packets are sent to the storage server; the storage server is used to generate backup files for the data backup. The step of obtaining the environmental parameters of at least one file to be backed up during the data backup process in the previous cycle includes: The first environmental parameter is generated based on the network latency and / or packet loss rate obtained from the heartbeat packet; The second environment parameter is generated based on the number of read / write operations per second (IOPS) and / or throughput of the client in the previous cycle.

7. The method according to claim 1, characterized in that, The determination of the current cycle's working parameters for the data backup process, based on the environmental parameters, working parameters, and a preset optimization model from the previous cycle, includes: The first environmental parameter is converted into a first input value based on a first function, and the second environmental parameter is converted into a second input value based on a second function; wherein, the first function represents the correlation between the first environmental parameter and the fragment size, and the second function represents the correlation between the second environmental parameter and the fragment size; The first input quantity, the second input quantity, the fragment size of the previous cycle, and the file size of the file to be backed up are input into the preset optimization model to obtain the working parameters of the current cycle of the data backup process.

8. The method according to claim 1, characterized in that, The process of driving the streaming backup tool to perform the data backup based on the current cycle's working parameters includes: The data in the data backup process is divided into multiple backup tasks based on the fragment size of the current period. The multiple backup tasks are added to the task queue in a specified order, and the thread pool is invoked to execute the backup tasks in the task queue concurrently based on the number of concurrent threads in the current period.

9. The method according to claim 8, characterized in that, The method further includes: If it is determined that the upload of any backup task's corresponding fragment fails, the failed backup task is added to the task queue based on the retry policy of the current period; wherein, the retry policy includes: the number of retries and / or the retry interval.

10. The method according to claim 9, characterized in that, If the backup type of the file to be backed up is a binary log backup type, the method further includes: Obtain the high-frequency data distribution rate corresponding to the shards of the binary log backup type during the data backup process of the previous cycle; The retry strategy for sharding the binary log backup type is determined based on the high-frequency data distribution rate.

11. A data backup device, characterized in that, The apparatus for performing fragmented data backup of at least one file to be backed up based on a streaming backup tool includes: The acquisition module is used to acquire the environmental parameters and working parameters of at least one file to be backed up in the previous cycle during the data backup process; An optimization module is used to determine the working parameters for the current cycle of the data backup process based on the environmental parameters, the working parameters, and a preset optimization algorithm from the previous cycle. The driver module is used to drive the streaming backup tool to perform the data backup based on the working parameters of the current cycle; The environmental parameters include one or more of the following: a first environmental parameter characterizing network performance and a second environmental parameter characterizing the input and output performance of the client to which the file to be backed up belongs; the working parameters include one or more of the following: the fragment size for partitioning, the number of concurrent threads for concurrent upload, and the retry strategy for fragment retransmission.

12. An electronic device, characterized in that, include: A processor and memory for storing computer programs that can run on the processor, wherein, The processor, when running a computer program, performs the steps of the method according to any one of claims 1 to 10.

13. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images