Checkpoint mode switching method of stream processing system and electronic device

Abstract

The application discloses a checkpoint mode switching method of a stream processing system and an electronic device, relates to the technical field of data writing, and comprises the following steps: when state data writing is performed based on a full-checkpoint mode, the duration of single data writing and the amount of single writing data are acquired, the data writing strategy of the full-checkpoint mode is evaluated in performance according to the duration of single data writing, the amount of single writing data, and preset maximum disk throughput and a preset data writing interval, a data writing decision score is determined, and when the data writing decision score meets a preset switching condition, the full-checkpoint mode is switched to an incremental checkpoint mode, so that the amount of data written each time and the writing duration are reduced, the influence of the data writing process on job performance is reduced, the technical problem that a state data writing strategy is difficult to adapt to a complex business scenario in the prior art is solved, and the technical effect that the checkpoint mode is automatically switched based on an actual job scenario and the flexibility of state data writing is improved is achieved.

Description

Technical Field

[0001] This application relates to the field of data writing technology, and in particular to a checkpoint mode switching method and electronic device for a stream processing system. Background Technology

[0002] During the Flink stream processing job, the state values ​​of the job are saved at preset time intervals based on Flink checkpoints. There are two types of Flink checkpoints: full checkpoints and incremental checkpoints.

[0003] In related technologies, the state data writing process during this job is performed using either full checkpoint or incremental checkpoint. However, this method has poor flexibility in state data writing and is difficult to adapt to complex business scenarios. Summary of the Invention

[0004] This application provides a checkpoint mode switching method and electronic device for a stream processing system, so as to at least solve the problem that state data writing strategies in related technologies are difficult to adapt to complex business scenarios.

[0005] This application provides a checkpoint mode switching method for a stream processing system. The method includes: acquiring the duration of a single data write and the amount of data written in a single write during the process of writing state data based on the full checkpoint mode; determining a data write decision score based on the duration of the single data write, the amount of data written in a single write, as well as a preset maximum disk throughput and a preset data write interval; and switching the full checkpoint mode to the incremental checkpoint mode when the data write decision score meets the preset switching conditions, so as to continue writing state data based on the incremental checkpoint mode.

[0006] This application also provides a checkpoint mode switching device for a stream processing system. The device includes: an acquisition module, used to acquire the duration of a single data write and the amount of data written in a single write during the process of writing status data based on the full checkpoint mode; a determination module, used to determine a data write decision score based on the duration of a single data write, the amount of data written in a single write, as well as a preset maximum disk throughput and a preset data write interval; and a control module, used to switch the full checkpoint mode to the incremental checkpoint mode when the data write decision score meets the preset switching conditions, so as to continue writing status data based on the incremental checkpoint mode.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the checkpoint mode switching method of any of the above-described stream processing systems when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the checkpoint mode switching method of any of the above-described stream processing systems.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the checkpoint mode switching method of any of the above-described stream processing systems.

[0010] This application addresses the issue of single data write duration and data volume during the process of writing status data to the target disk using a full checkpoint mode. It then evaluates the performance of the full checkpoint mode's data write strategy by combining this with preset maximum disk throughput and preset data write intervals, obtaining a data write decision score. When the data write decision score meets preset switching conditions, it is determined that the full checkpoint mode's data write strategy results in excessively large data volumes and long write times, consuming significant system resources and potentially hindering the performance of core business operations. Therefore, it switches from full checkpoint mode to incremental checkpoint mode to continue writing status data, reducing the data volume and write time per operation and minimizing the impact of the data write process on job performance. This solves the problem in related technologies where status data write strategies are difficult to adapt to complex business scenarios, achieving the technical effect of improving the flexibility of status data writing and adapting to complex business scenario requirements. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic flowchart illustrating a checkpoint mode switching method for a stream processing system provided in this application embodiment; Figure 2 This application provides a schematic diagram of a checkpoint storage order in an embodiment of the present application. Figure 3 A tree-like dependency diagram of a checkpoint is provided for an embodiment of this application; Figure 4 A flowchart illustrating the mode switching method of a checkpoint mode switching method for a stream processing system provided in this application embodiment; Figure 5 A schematic diagram of period control for asynchronous compression provided in an embodiment of this application; Figure 6This is a connection diagram of a checkpoint mode switching device for a stream processing system provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The specific application environment architecture or specific hardware architecture on which the execution of the checkpoint mode switching method of the stream processing system depends is described here.

[0017] In related technologies, Flink checkpoints have two types: full checkpoints and incremental checkpoints. A Flink operator refers to each processing node in a Flink stream processing job. A stream processing job can be viewed as a factory pipeline, with each operator being a processing step on that pipeline. A Flink checkpoint refers to a point in time where Flink checks for data flow. Flink stream processing jobs can be stateful (each operator may have its own state; for example, to count the data flow per minute, this operator needs to record how many data points have arrived). That is, during job execution, the state value needs to be saved periodically (at fixed time intervals), called a checkpoint. If the job fails and needs to be restarted, the most recently successfully saved checkpoint can be read, thus achieving stateful job recovery.

[0018] Full checkpointing refers to writing all Flink operator state data to disk each time it is triggered, regardless of whether this state data has changed. This implementation is relatively simple and suitable for job scenarios with a small amount of checkpoint data (e.g., jobs without long windows and infrequent changes). Furthermore, this method ensures that each checkpoint data is independent, with no dependencies between them, making it extremely convenient to clean up outdated checkpoint data. However, its performance is extremely poor in scenarios with high-frequency, large-volume checkpoint data writing, because the amount of checkpoint data that needs to be written each time is enormous.

[0019] Incremental checkpointing refers to writing only the changed data from each Flink operator on each trigger. In scenarios with high-frequency, large-volume checkpoint writes, the amount of data written each time is much smaller than that of a full checkpoint, resulting in better performance. However, there are some problems: incremental writes are based on the data of the previous checkpoint (and can continue recursively), meaning that checkpoints are dependent on each other. Historical checkpoint data cannot be easily cleaned up, wasting disk space. If the data of a dependent checkpoint is corrupted, the checkpoint cannot be recovered. Therefore, the more dependent checkpoints there are, the higher the probability of unrecoverable corruption, and the longer the job recovery time from checkpoints (i.e., recovery requires traversing all dependent checkpoints).

[0020] Based on the above analysis, each of the above checkpoint methods has its own considerations regarding disk space usage, amount of data written, and checkpoint speed, and cannot adapt to complex business scenarios.

[0021] To address the aforementioned technical issues, this application proposes a checkpoint mode switching method for a stream processing system. This method, during the process of writing state data to the target disk using full checkpoint mode, evaluates the performance of the full checkpoint mode's data writing strategy by considering factors such as single data write duration, single data write volume, preset maximum disk throughput, and preset data write interval. This yields a data write decision score. When the data write decision score meets preset switching conditions, the system switches from full checkpoint mode to incremental checkpoint mode, continuing to write state data. This reduces the amount of data written and the time consumed per write, minimizing the impact of the data write process on job performance. This balances the advantages and disadvantages of full checkpoint and incremental checkpoint, improves the flexibility of state data writing, and achieves the technical effect of adapting to complex business scenario requirements.

[0022] The following section describes the method in detail, taking into account the execution flow of the checkpoint mode switching method in a stream processing system.

[0023] Figure 1 This is a schematic flowchart illustrating a checkpoint mode switching method for a stream processing system provided in an embodiment of this application.

[0024] Reference Figure 1 As shown, the checkpoint mode switching method of the stream processing system in this application embodiment includes the following steps: S101, during the process of writing status data based on full checkpoint mode, obtain the duration of a single data write and the amount of data written in a single instance.

[0025] Specifically, incremental checkpoint mode refers to the incremental checkpoint mentioned above. When the job start command is received, the state data write operation is first performed in full checkpoint mode. That is, each time a state data write action is triggered, all state data of Flink operators are written to the target disk, and during each data write process, the duration of a single data write (i.e., checkpoint time) and the amount of data written in a single write (i.e., checkpoint write amount) are monitored.

[0026] S102, determine the data write decision score based on the single data write duration, the single data write volume, the preset maximum disk throughput, and the preset data write interval.

[0027] The preset maximum disk throughput and preset data write interval can be calibrated according to the actual situation. Specifically, the time consumption ratio can be identified based on the single data write duration and the preset data write interval to determine the workload of each status data write. The data write speed is calculated based on the single data write duration and the single data write volume to calculate the weight of the data write speed relative to the preset maximum disk throughput, so as to obtain a comprehensive evaluation and data write decision score. For example, the mapping relationship between the time consumption ratio and the evaluation score, as well as the mapping relationship between the weight of the data write speed relative to the preset maximum disk throughput and the evaluation score, can be pre-calibrated. Then, the corresponding evaluation score is obtained by looking up the table based on the actual calculated time consumption ratio and the weight of the data write speed relative to the preset maximum disk throughput. The sum of the evaluation scores is used as the decision score.

[0028] S103, when the data writing decision score meets the preset switching conditions, switch the full checkpoint mode to the incremental checkpoint mode to continue writing state data based on the incremental checkpoint mode.

[0029] Specifically, the data write decision score is used to evaluate the system resource consumption and performance impact on key business operations during the execution of the full checkpoint mode. Preset switching conditions are used to evaluate the data write strategy score. If the data write decision score meets the preset switching conditions, it is considered that the full checkpoint mode consumes too many system resources during execution, which will drag down the performance of key business operations, and a switch to incremental checkpoint mode is required. If the data write decision score does not meet the preset switching conditions, the full checkpoint mode execution state and data writing will continue.

[0030] Understandably, preset switching conditions can be constructed by setting preset configuration thresholds. For example, when the data write decision score is positively correlated with system resource usage and the performance impact on the main business, then the data write decision score is considered to meet the preset switching conditions if it is greater than the preset configuration threshold; otherwise, it is considered not to meet the conditions. When the data write decision score is negatively correlated with system resource usage and the performance impact on the main business, then the data write decision score is considered to meet the preset switching conditions if it is less than the preset configuration threshold; otherwise, it is considered not to meet the conditions. The specific settings can be configured according to the actual situation.

[0031] The incremental checkpoint mode corresponds to the incremental checkpoint mentioned above. When switching to incremental checkpoint mode, only the data that has changed for each Flink operator is written to the target disk each time a state data write is triggered, so as to reduce the amount of data written each time, reduce system resource consumption and the impact on the performance of the main business.

[0032] This embodiment assesses the data write decision score based on the real-time operation of the cluster, specifically by obtaining the duration and amount of data written per execution when using full checkpointing mode for state data writing. This score determines whether to continue using full checkpointing or incremental checkpointing, improving the flexibility of state data writing and adapting to complex business scenarios. Specifically, when the checkpoint data volume is too large or the time consumption is too long, the system switches from full checkpointing to incremental checkpointing to reduce the amount of data written and the time consumption per execution, minimizing the impact on job performance; conversely, it switches back from incremental checkpointing mode to full checkpointing mode.

[0033] In some embodiments of this application, determining a data write decision score based on a single data write duration, a single data write volume, a preset maximum disk throughput, and a preset data write interval includes: determining a first ratio based on a single data write duration and a preset data write interval; determining a data write speed based on a single data write volume and a single data write duration; determining a second ratio based on a data write speed and a preset maximum disk throughput; and determining a data write decision score based on the first ratio and the second ratio.

[0034] Specifically, the formula for calculating the decision score is as follows: , in, Write the data into a decision score. Duration of a single data write operation. This refers to the amount of data written in a single operation. To preset the maximum disk throughput, The preset data write interval, and The preset weights can be configured according to actual conditions. This embodiment uses a first ratio to represent the proportion of actual write time to a preset data write interval, and a second ratio to represent the proportion of actual write efficiency to the theoretical / total throughput capacity. Based on the expected influence of the first and second ratios on the data write decision score in practical applications, the preset weights are determined. and For example, when the expected first ratio and second ratio have the same impact on the data writing decision score, set... and Both are 0.5; when the expected influence of the first ratio on the data writing decision score is greater than that of the second ratio, the value is set to... It is 0.6. It is 0.4.

[0035] This embodiment constructs a formula for calculating the data write decision score by weighting and summing the proportion of a single data write duration to a preset data write interval and the weight of the data write speed to a preset maximum disk throughput. This improves the accuracy and efficiency of the data write decision score calculation. If the data write decision score is greater than a preset configuration threshold, it indicates that the full checkpoint is consuming too many system resources and will drag down the performance of the main business, meeting the preset switching conditions, and needs to be switched to an incremental checkpoint; otherwise, the full checkpoint continues to be used.

[0036] Furthermore, to prevent drastic fluctuations in measured values ​​from causing decision-making instability, EMA (Exponential Moving Average) is used for exponential smoothing when collecting the duration and amount of data written in a single transaction. Based on the smoothed duration and amount of data written in a single transaction, a data writing decision score is calculated.

[0037] For example, first, determine the calculation period N of the corresponding EMA based on the data type of the measured values. Then, calculate the smoothing coefficient α based on the selected period N, using the formula: α = 2 / (N+1). Next, initialize the EMA sequence and determine the smoothing value of the first EMA as the starting point of the sequence. For example, the first measured value can be used as the first EMA value, or the arithmetic mean of the previous N measured values ​​can be used as the first EMA value. Starting from the second data point, calculate the EMA value for each time step in chronological order. Each calculation uses the recursive formula: current EMA value = α × current measured value + (1-α) × previous EMA value. The calculated current EMA value is used for the calculation of the decision score in the above data writing.

[0038] In some embodiments of this application, during the process of writing state data based on incremental checkpoint mode, the method further includes: determining an asynchronous compression period and determining a target data writing time based on the asynchronous compression period; when the actual time reaches the target data writing time, identifying the actual checkpoint file generated by the corresponding data writing operation, performing an asynchronous compression operation on the actual checkpoint file to generate a target full checkpoint file corresponding to the actual checkpoint file, and replacing the actual checkpoint file with the target full checkpoint file to continue writing state data based on the target full checkpoint file.

[0039] Specifically, considering that incremental checkpointing only writes the data that has changed for each Flink operator during each data write, checkpoints have dependencies between them. Excessive checkpoint dependencies can easily lead to the following problems: 1. Resuming from a checkpoint requires traversing and merging all checkpoints in the dependency tree (i.e., dependency relationships), which is very time-consuming; 2. If a checkpoint file in the dependency tree is corrupted, data cannot be merged during recovery, and the checkpoint cannot be recovered; 3. The entire dependency tree occupies too much disk space, and there are too many checkpoints in the tree. It is difficult to clean up old orphaned checkpoints (that are not in any dependency tree), resulting in wasted disk space.

[0040] To avoid excessively long dependency trees in incremental data (too many previous checkpoints in hierarchical and cascading dependencies), an asynchronous compression step is used to balance the above situation and the time consumption of checkpointing, thus achieving a balance between the advantages and disadvantages of incremental checkpointing.

[0041] Specifically, the asynchronous compression period characterizes the triggering cycle of the asynchronous compression operation, that is, how many incremental checkpoints are intervals between asynchronous compression operations. This can be set according to the actual situation. For example, if it is desired to perform an asynchronous compression operation every 50 incremental checkpoints, then the asynchronous compression period is calculated based on the incremental checkpoint triggering cycle, and the asynchronous compression period = 50. The triggering period for incremental checkpoints.

[0042] During the incremental checkpointing process of writing state data, the checkpoint for asynchronous compression is determined based on the target data write time of the asynchronous compression cycle. When the actual time reaches the target data write time (i.e., the checkpoint for asynchronous compression), asynchronous compression is performed on the actual checkpoint file corresponding to that checkpoint, generating the target full checkpoint file. This target full checkpoint file contains all the data for the corresponding checkpoint. This target full checkpoint file replaces the original data file corresponding to that checkpoint (i.e., the actual checkpoint file). Therefore, except for checkpoints generated during asynchronous compression, all subsequent checkpoints will not depend on any checkpoints prior to this asynchronous compression, effectively truncating the dependency tree and limiting its maximum length. Previous checkpoint data files can be safely cleaned up.

[0043] Furthermore, the checkpoints generated during asynchronous compression refer to the new checkpoints generated simultaneously during the asynchronous compression process. For example, if the asynchronous compression checkpoint step is time-consuming and generates 5 checkpoints during this period, observing the dependency tree analysis results of these 5 checkpoints, they may depend on checkpoint files from before asynchronous compression (as case 1), or they may depend on data files within these 5 checkpoints (e.g., checkpoint 4 depends on checkpoint 1, as case 2). After successful asynchronous compression, the dependencies in case 1 need to be rewritten to point to the full checkpoint files generated by the asynchronous compression, while the dependencies in case 2 remain unchanged.

[0044] In some embodiments of this application, performing asynchronous compression on the actual checkpoint file includes: identifying at least one historical checkpoint file that has a data dependency relationship with the actual checkpoint file; and determining a target full checkpoint file based on the data dependency relationship and the at least one historical checkpoint file and the actual checkpoint file.

[0045] Specifically, when asynchronous compression is triggered, it ensures that the current incremental checkpoint is completed, identifies the current checkpoint file (the actual checkpoint file), and finds the previous checkpoint data files and historical checkpoint files that the current incremental part depends on. Similarly, the previously depended checkpoint data files may also depend on even earlier checkpoint files, requiring recursive searching. These data dependencies form a tree-like relationship. For example, the storage order of checkpoint files is as follows: Figure 2 As shown, the structure is sequential for each version, but the relationships between checkpoint files are actually tree-like dependencies, as follows: Figure 3 As shown.

[0046] After identifying all historical checkpoint files that the actual checkpoint file depends on, at least one historical checkpoint file and the actual checkpoint file are merged to form the target full checkpoint file. Then, the target full checkpoint file replaces the corresponding actual checkpoint file, thus truncating the dependency tree and ensuring the execution accuracy and efficiency of the asynchronous compression step.

[0047] In some embodiments of this application, identifying at least one historical checkpoint file that stores data dependencies with the actual checkpoint file includes: determining metadata information of the actual checkpoint file; determining data dependencies based on the metadata information; and determining at least one historical checkpoint file based on the data dependencies.

[0048] Specifically, the metadata information of the actual checkpoint file includes data dependencies. For example, the metadata information includes a file list and a reference list. The file list is a complete list or reference of all files included in the current checkpoint, and the reference list is a list of pointers to files that existed in one or more previous checkpoints. Then, based on each pointer in the reference list in the metadata information, the previous checkpoint it points to is found, and the metadata of that checkpoint is read. This process is repeated until a full checkpoint containing all the necessary basic states or a sufficiently early incremental checkpoint is found, thus constructing a complete data dependency relationship. Then, based on the checkpoint ID list in the data dependency relationship, all data files in the corresponding directory are read from the storage system, thus obtaining the historical checkpoint file corresponding to the actual checkpoint file.

[0049] This embodiment identifies data dependencies based on the metadata information of actual checkpoint files, thereby determining historical checkpoint files with data dependencies and improving the accuracy of historical checkpoint file identification.

[0050] In some embodiments of this application, determining a target full checkpoint file based on data dependencies and at least one historical checkpoint file and an actual checkpoint file includes: determining the file merging order based on data dependencies; and merging at least one historical checkpoint file and an actual checkpoint file based on the file merging order to obtain the target full checkpoint file.

[0051] In other words, after finding all historical checkpoint files based on data dependencies, these historical checkpoint files and the actual checkpoint files obtained this time are merged from front to back according to a tree structure. When different versions of the same data are encountered, the new data overwrites the old data, until a full data file is generated to replace the actual checkpoint files, thereby improving the generation accuracy of the target full checkpoint file.

[0052] In some embodiments of this application, during the asynchronous compression operation on the actual checkpoint file, the method further includes: obtaining the asynchronous compression data volume of the asynchronous compression operation; and switching the incremental checkpoint mode to the full checkpoint mode when the asynchronous compression data volume is less than a preset degradation threshold.

[0053] Specifically, continuing with the calculation formula for the data writing decision score as described above, we will take the calculation of the data writing decision score as an example, combined with... Figure 4 As shown, when the data writing decision score is greater than the preset configuration threshold, the system switches to incremental checkpoint mode.

[0054] After switching to incremental checkpoint mode, the calculation formula for the data write decision score is no longer applicable to the decision-making process of switching from incremental checkpoint mode to full checkpoint mode because the duration and amount of data written per transaction will be drastically reduced.

[0055] This embodiment, after switching to incremental checkpoint mode, identifies the amount of changed data at each checkpoint by monitoring the amount of asynchronous compressed data in each asynchronous compression operation. It then determines whether to switch from incremental checkpoint mode to full checkpoint mode, ensuring the rationality of the switching judgment and decision. Specifically, when the amount of asynchronous compressed data is less than a preset degradation threshold, it is considered that the amount of changed data at the checkpoint has significantly decreased, and a full checkpoint is initiated at the next checkpoint. The preset degradation threshold can be set according to actual conditions.

[0056] In some embodiments of this application, the method further includes: after switching from full checkpoint mode to incremental checkpoint mode, obtaining the asynchronous compressed data volume of multiple asynchronous compression operations; and determining a preset degradation threshold based on the asynchronous compressed data volume of multiple asynchronous compression operations.

[0057] In other words, after switching from full checkpoint mode to incremental checkpoint mode, the preset degradation threshold is determined based on the amount of asynchronous compressed data from the previous n asynchronous compression operations. For example, the preset degradation threshold is calculated by obtaining the amount of asynchronous compressed data from the previous 10 operations, so that the setting of the preset degradation threshold matches the actual business scenario. Understandably, during the execution of the first 10 asynchronous compression operations, the amount of asynchronous compressed data can be left unchecked, meaning the incremental checkpoint mode can be maintained. After completing 10 asynchronous compression operations, the amount of asynchronous compressed data for each subsequent asynchronous compression operation can be evaluated based on a predetermined preset degradation threshold to determine whether to switch back to full checkpoint mode. Alternatively, the amount of asynchronous compressed data for the first 10 asynchronous compression operations can be evaluated based on a pre-calibrated preset degradation threshold (e.g., a preset degradation threshold pre-calibrated based on the amount of asynchronous compressed data from historical asynchronous compression operations, or a preset degradation threshold calibrated by staff) to determine whether to switch back to full checkpoint mode. After completing 10 asynchronous compression operations, the amount of asynchronous compressed data for each subsequent asynchronous compression operation can be evaluated based on the preset degradation threshold calculated from the amount of asynchronous compressed data in the first 10 asynchronous compression operations to determine whether to switch back to full checkpoint mode.

[0058] In some embodiments of this application, determining a preset degradation threshold based on the asynchronous compressed data volume of multiple asynchronous compression operations includes: determining the average asynchronous compressed data volume based on the asynchronous compressed data volume of multiple asynchronous compression operations; and determining the preset degradation threshold based on the average asynchronous compressed data volume and a preset coefficient.

[0059] Specifically, the preset coefficient is used to characterize the level of data reduction allowed when switching from incremental checkpoint mode to full checkpoint mode. It can be set according to the actual situation, such as 10%, 20%, etc.

[0060] Continuing with the example of calculating the preset degradation threshold based on the asynchronous compressed data volume of the first 10 iterations after switching to incremental checkpoint mode, the preset coefficient is 10%. After obtaining the asynchronous compressed data volume of the first 10 iterations, the average of the asynchronous compressed data volume of the first 10 iterations is calculated as the baseline data volume. Then, 10% of the average asynchronous compressed data volume is used as the preset degradation threshold. That is to say, when the current asynchronous compressed data volume is less than 10% of the average asynchronous compressed data volume, it is considered that the checkpoint change data volume has significantly decreased, and the system switches to full checkpoint mode.

[0061] This embodiment uses the average value of multiple asynchronous compressed data volumes as the baseline data volume, and uses the baseline data volume with a preset coefficient as the preset degradation threshold, which improves the setting accuracy and efficiency of the preset degradation threshold and provides data support for the rationality of switching.

[0062] In some embodiments of this application, during the process of writing state data based on incremental checkpoint mode, the method further includes: obtaining the amount of asynchronous compressed data for each asynchronous compression operation; determining the incremental data change slope based on the amount of asynchronous compressed data for each asynchronous compression operation; and adjusting the asynchronous compression period based on the incremental data change slope.

[0063] Specifically, considering that the asynchronous compression operation generates full checkpoint data, the disk read / write bandwidth usage is higher than that of incremental checkpoints. Therefore, asynchronous compression needs to be performed every few incremental checkpoints to generate a full checkpoint, such as... Figure 5 As shown, in order to maximize the benefits of asynchronous compression while minimizing the impact on job performance, this embodiment adjusts the asynchronous compression cycle based on job monitoring metrics, namely the amount of asynchronously compressed data in each asynchronous compression operation.

[0064] To avoid drastic jumps in the calculated slope of incremental data change, after obtaining the incremental data volume (i.e., the asynchronous compressed data volume) during each asynchronous compression operation, an EMA (Exponential Moving Average) is used for smoothing. Then, based on the smoothed asynchronous compressed data volume, the slope of asynchronous compressed data change is monitored, thus obtaining the incremental data change slope. For example, after switching to incremental checkpoint, the incremental data change slope can be obtained by fitting the asynchronous compressed data volume within a preset sliding window, or by fitting the slope of all asynchronous compressed data volumes.

[0065] Then, the asynchronous compression period is adjusted based on the slope of the incremental data change. For example, when the slope of the incremental data change decreases, it is assumed that the amount of updated data is decreasing, so the asynchronous compression period is shortened; when the slope of the incremental data change increases, the asynchronous compression period is lengthened. Thus, the interval of asynchronous compression is dynamically adjusted based on the real-time operation of the job, achieving dynamic optimization.

[0066] In some embodiments of this application, determining the incremental data change slope based on the asynchronous compressed data volume of each asynchronous compression operation includes: determining the asynchronous compressed data volume of the current asynchronous compression operation and the asynchronous compressed data volume of the previous asynchronous compression operation based on the asynchronous compressed data volume of each asynchronous compression operation; determining the data change amount based on the asynchronous compressed data volume of the current asynchronous compression operation and the asynchronous compressed data volume of the previous asynchronous compression operation; and determining the incremental data change slope based on the data change amount and the asynchronous compression interval between the current asynchronous compression operation and the previous asynchronous compression operation.

[0067] In other words, the slope of incremental data change is calculated based on the asynchronous compressed data volume of the two most recent asynchronous compression operations, thereby improving the real-time representation of data changes.

[0068] The formula for calculating the slope of the incremental data change is: .

[0069] In some embodiments of this application, adjusting the asynchronous compression period based on the slope of incremental data change includes: determining the adjustment range based on the slope of incremental data change; and adjusting the asynchronous compression period based on the adjustment range.

[0070] Specifically, to avoid overly aggressive adjustments to the asynchronous compression cycle, the adjustable range of the asynchronous compression cycle can be limited, for example, by constructing an adjustable range of the asynchronous compression cycle from 25% to 400% of the baseline asynchronous compression cycle.

[0071] After obtaining the slope of the incremental data change, the adjustment range is first determined based on the slope. For example, a mapping table between the incremental data change slope and the adjustment range can be pre-defined. After calculating the incremental data change slope, the adjustment range can be determined by calling the table. Alternatively, a conversion formula between the incremental data change slope and the adjustment range can be pre-defined, as shown below: , in, To adjust the amplitude, This represents the maximum adjustment range in a single operation. To preset the sensitivity adjustment, The slope of the incremental data change. To Round to the nearest integer. The preset adjustment step.

[0072] Then, the asynchronous compression period is adjusted according to the adjustment range to ensure the adjustment accuracy and rationality of the asynchronous compression axis. For example, the adjustment range can be a period adjustment coefficient or an interval adjustment step size.

[0073] Continue to calculate the adjustment range using the formula above. For example, the adjustment range The formula for calculating the adjusted asynchronous compression interval is as follows: , in, This is the adjusted asynchronous compression interval. This is the interval between the last asynchronous compression. This is the upper limit of the compression interval. This is the lower limit of the compression interval. To adjust the amplitude.

[0074] After calculating the adjusted asynchronous compression interval, the adjusted asynchronous compression interval is used as the adjusted asynchronous compression period.

[0075] Furthermore, if the asynchronous compression interval remains constant for a long time... or This indicates that the dynamic adjustment range setting is unreasonable, and an alarm message needs to be sent to the administrator.

[0076] Additionally, if the current asynchronous compression operation is not completed before the next asynchronous compression operation is scheduled, the execution of the next asynchronous compression operation will be postponed and triggered after the current asynchronous compression operation is completed. In this case, the interval between the next asynchronous compression operation and the next operation should be postponed to the current asynchronous compression interval plus a step size, while simultaneously increasing the disk throughput percentage during asynchronous compression operations, for example, by 5%. If the disk throughput percentage and interval reach their limits, an alarm message will be sent to the cluster administrator.

[0077] In some embodiments of this application, the method is characterized in that, during the process of writing state data based on incremental checkpoint mode, the method further includes: in response to a job fault recovery instruction, after performing a preset number of data writing operations, performing the first asynchronous compression operation after fault recovery, wherein the time interval between the generation time of the job fault recovery instruction and the execution time of the first asynchronous compression operation is less than the asynchronous compression period.

[0078] Specifically, the preset number of times can be set according to the actual situation, for example, by using the formula: Baseline Asynchronous Compression Interval 0.1, determine the asynchronous compression interval of the first asynchronous compression operation after triggering fault recovery, and then round the ratio between the asynchronous compression interval and the triggering period of the incremental checkpoint to obtain the preset number of times.

[0079] In other words, if a job fails and needs to be recovered, an asynchronous compression operation will be performed immediately after the preset number of checkpoints following the recovery. This will serve as the first asynchronous compression operation after the initial recovery, and the starting point for the next asynchronous compression cycle will be calculated from this initial operation. The asynchronous compression disk throughput percentage will be adjusted to 5%. This configuration prevents jobs from failing frequently and repeatedly due to environmental issues, resulting in excessively long recovery times. It also prevents the need to process backlogged data immediately after job recovery, which would exacerbate cluster resource strain and job instability under conditions of surging resource consumption and simultaneous asynchronous compression.

[0080] In some embodiments of this application, during the process of writing state data based on incremental checkpoint mode, the method includes: obtaining the job throughput of each asynchronous compression operation; reducing the disk throughput ratio and job throughput of the asynchronous compression operation when the decrease ratio of job throughput is greater than a first preset ratio and the number of consecutive operations reaches a first preset number; reducing the disk throughput ratio of the asynchronous compression operation when the decrease ratio of job throughput is less than or equal to the first preset ratio and greater than or equal to a second preset ratio, and the number of consecutive operations reaches a first preset number, wherein the second preset ratio is less than the first preset ratio; and increasing the disk throughput ratio of the asynchronous compression operation when the decrease ratio of job throughput is less than the second preset ratio and the number of consecutive operations reaches a second preset number.

[0081] Specifically, the first preset ratio, the second preset ratio, the first preset number of times, and the second preset number of times can be set according to the actual situation.

[0082] Taking a first preset ratio of 5%, a second preset ratio of 1%, a first preset number of times of execution (3 times), and a second preset number of times of execution (10 times) as an example, during the data writing process based on incremental checkpoint mode, the throughput of the job during non-asynchronous compression and the throughput of the job during asynchronous compression (i.e., the throughput of the asynchronous compression operation) are monitored. The job throughput represents the processing speed of the job, such as the number of data entries processed per second, which can be obtained through data monitoring.

[0083] If three consecutive asynchronous compression processes show a decrease in job throughput exceeding 5%, the disk throughput percentage and job throughput for the next asynchronous compression operation will be reduced by the same percentage. If the job throughput for three consecutive asynchronous compression processes decreases between 1% and 5%, the asynchronous compression disk throughput percentage will not be adjusted. Conversely, if the job throughput for ten consecutive asynchronous compression processes decreases by less than 1%, the asynchronous compression disk throughput percentage will be increased by 5%, up to the baseline value. The asynchronous compression disk throughput percentage refers to the ratio of the asynchronous compression operation's disk throughput to the total disk throughput.

[0084] This embodiment automatically adjusts the disk throughput ratio of asynchronous compression based on the job throughput of each asynchronous compression operation, so as to speed up asynchronous compression while ensuring job performance.

[0085] As a specific embodiment of this application, the checkpoint mode switching method of the stream processing system proposes an automatic switching strategy between full checkpoint and incremental checkpoint. Specifically, considering that the amount of checkpoint data is small and the system resource consumption is not large, full checkpoint is used first. When the amount of checkpoint data is too large and the time consumption is too long, incremental checkpoint is switched to. When the job checkpoint increment drops, full checkpoint is switched to.

[0086] In addition, full checkpoints and incremental checkpoints are used in combination. Every certain number of incremental checkpoints, a full checkpoint is generated asynchronously through compression. This controls the maximum length of the checkpoint dependency tree, isolates old checkpoints, makes data cleanup easier, and frees up disk space. At the same time, it controls the checkpoint recovery time to prevent it from being too long. By limiting the number of files traversed during checkpoint recovery, the probability of being unable to recover due to file corruption is reduced.

[0087] In addition, considering that asynchronous compression consumes more system resources, this method designs an automatic adjustment strategy for the asynchronous compression cycle (interval) to balance the performance consumption of asynchronous compression with the beneficial effects of asynchronous compression.

[0088] Therefore, this method combines the advantages of full checkpointing and incremental checkpointing, solving the problems of checkpoints occupying too much disk space and being unable to be cleaned up, and checkpoints being prone to corruption and slow recovery due to excessively long dependency trees. At the same time, it can retain the advantages of incremental checkpointing, such as fast speed and small amount of data written.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0090] Embodiments of this application also provide a checkpoint mode switching device for a stream processing system. The device includes: an acquisition module, used to acquire the duration of a single data write and the amount of data written in a single write during the process of writing status data based on a full checkpoint mode; a determination module, used to determine a data write decision score based on the duration of the single data write, the amount of data written in a single write, a preset maximum disk throughput, and a preset data write interval; and a control module, used to switch from a full checkpoint mode to an incremental checkpoint mode when the data write decision score meets preset switching conditions, so as to continue writing status data based on the incremental checkpoint mode.

[0091] In some embodiments of this application, the determining module determines a data write decision score based on the single data write duration, the single data write volume, the preset maximum disk throughput, and the preset data write interval. Specifically, it is used to: determine a first ratio based on the single data write duration and the preset data write interval; determine a data write speed based on the single data write volume and the single data write duration; determine a second ratio based on the data write speed and the preset maximum disk throughput; and determine a data write decision score based on the first ratio and the second ratio.

[0092] In some embodiments of this application, during the process of writing state data based on incremental checkpoint mode, the control module is further configured to: determine the asynchronous compression period and determine the target data writing time according to the asynchronous compression period; when the actual time reaches the target data writing time, identify the actual checkpoint file generated by the corresponding data writing operation, perform asynchronous compression operation on the actual checkpoint file to generate the target full checkpoint file corresponding to the actual checkpoint file, and replace the actual checkpoint file with the target full checkpoint file so as to continue to perform state data writing based on the target full checkpoint file.

[0093] In some embodiments of this application, the control module performs asynchronous compression on the actual checkpoint file, specifically for: identifying at least one historical checkpoint file that has a data dependency relationship with the actual checkpoint file; and determining the target full checkpoint file based on the data dependency relationship and the at least one historical checkpoint file and the actual checkpoint file.

[0094] In some embodiments of this application, the control module identifies at least one historical checkpoint file that stores data dependencies with the actual checkpoint file, specifically for: determining metadata information of the actual checkpoint file; determining data dependencies based on the metadata information; and determining at least one historical checkpoint file based on the data dependencies.

[0095] In some embodiments of this application, the control module determines the target full checkpoint file based on data dependencies and at least one historical checkpoint file and an actual checkpoint file. Specifically, it is used to: determine the file merging order based on data dependencies; and merge at least one historical checkpoint file and an actual checkpoint file based on the file merging order to obtain the target full checkpoint file.

[0096] In some embodiments of this application, during the asynchronous compression operation on the actual checkpoint file, the control module is further configured to: obtain the asynchronous compression data volume of the asynchronous compression operation; and switch the incremental checkpoint mode to the full checkpoint mode when the asynchronous compression data volume is less than a preset degradation threshold.

[0097] In some embodiments of this application, the control module is further configured to: after switching from full checkpoint mode to incremental checkpoint mode, obtain the asynchronous compressed data volume of multiple asynchronous compression operations; and determine a preset degradation threshold based on the asynchronous compressed data volume of multiple asynchronous compression operations.

[0098] In some embodiments of this application, the control module determines a preset degradation threshold based on the asynchronous compressed data volume of multiple asynchronous compression operations. Specifically, it is used to: determine the average asynchronous compressed data volume based on the asynchronous compressed data volume of multiple asynchronous compression operations; and determine the preset degradation threshold based on the average asynchronous compressed data volume and a preset coefficient.

[0099] In some embodiments of this application, the control module is further configured to: obtain the amount of asynchronous compressed data for each asynchronous compression operation; determine the incremental data change slope based on the amount of asynchronous compressed data for each asynchronous compression operation; and adjust the asynchronous compression cycle based on the incremental data change slope.

[0100] In some embodiments of this application, the control module determines the incremental data change slope based on the asynchronous compressed data volume of each asynchronous compression operation. Specifically, it is used to: determine the asynchronous compressed data volume of the current asynchronous compression operation and the asynchronous compressed data volume of the previous asynchronous compression operation based on the asynchronous compressed data volume of each asynchronous compression operation; determine the data change amount based on the asynchronous compressed data volume of the current asynchronous compression operation and the asynchronous compression interval between the current asynchronous compression operation and the previous asynchronous compression operation; and determine the incremental data change slope based on the data change amount and the asynchronous compression interval between the current asynchronous compression operation and the previous asynchronous compression operation.

[0101] In some embodiments of this application, the control module adjusts the asynchronous compression period based on the slope of the incremental data change, specifically for: determining the adjustment range based on the slope of the incremental data change; and adjusting the asynchronous compression period based on the adjustment range.

[0102] In some embodiments of this application, during the process of writing status data based on incremental checkpoint mode, the control module is further configured to: in response to the job fault recovery instruction, obtain the actual number of data writes after the job fault recovery; when the actual number of data writes reaches a preset number, perform the first asynchronous compression operation after the fault recovery, wherein the time interval between the generation time of the job fault recovery instruction and the execution time of the first asynchronous compression operation is less than the asynchronous compression period.

[0103] In some embodiments of this application, during the process of writing status data based on incremental checkpoint mode, the control module is further configured to: obtain the job throughput of each asynchronous compression operation; reduce the disk throughput ratio and job throughput of the asynchronous compression operation when the decrease ratio of job throughput is greater than a first preset ratio and the number of occurrences reaches a first preset number; reduce the disk throughput ratio of the asynchronous compression operation when the decrease ratio of job throughput is less than or equal to the first preset ratio and greater than or equal to the second preset ratio, and the number of occurrences reaches a first preset number, wherein the second preset ratio is less than the first preset ratio; and increase the disk throughput ratio of the asynchronous compression operation when the decrease ratio of job throughput is less than the second preset ratio and the number of occurrences reaches a second preset number.

[0104] As a specific embodiment of this application, such as Figure 6 As shown, the checkpoint mode switching device of the stream processing system includes a checkpoint monitoring and recording module (i.e., acquisition module), a checkpoint type decision module (i.e., determination module and control module), an asynchronous compression module, and an asynchronous compression interval automatic adjustment module.

[0105] The checkpoint monitoring and recording module monitors the amount of data written for each checkpoint and the time consumed for each checkpoint. After enabling incremental checkpoint mode, it records the incremental data volume of each checkpoint, providing the subsequent checkpoint type decision module and asynchronous compression interval automatic adjustment module with a grasp of the cluster's operational status and factual support for their adjustment decisions.

[0106] The checkpoint type decision module is used to determine whether to use full checkpoint mode or incremental checkpoint mode based on the monitoring results, and can switch between the two.

[0107] The asynchronous compression module is used to periodically rewrite an existing incremental checkpoint into a full checkpoint. During the rewrite, all preceding checkpoints that the current checkpoint depends on are identified, merged to generate full data, and then used to replace the original incremental checkpoint. A full checkpoint effectively truncates the incremental checkpoint dependency tree, reducing its size. Furthermore, all checkpoints preceding the full checkpoint are no longer depended on by subsequent checkpoints, allowing for safe deletion and freeing up disk space.

[0108] The asynchronous compression interval automatic adjustment module is used to optimally balance the advantages and disadvantages of full checkpointing and incremental checkpointing. It can adjust the asynchronous compression interval in real time by monitoring the cluster's running status to find the best balance point and trigger asynchronous compression periodically.

[0109] For a description of the features of the checkpoint mode switching device in the embodiment of the stream processing system, please refer to the relevant description of the checkpoint mode switching method in the embodiment of the stream processing system, which will not be repeated here.

[0110] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described stream processing system checkpoint mode switching method embodiments.

[0111] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the checkpoint mode switching method embodiments of any of the above-described stream processing systems when running.

[0112] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0113] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described stream processing system checkpoint mode switching method embodiments.

[0114] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described stream processing system checkpoint mode switching method embodiments.

[0115] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0116] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] The above provides a detailed description of the checkpoint mode switching method and electronic device for a stream processing system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for switching checkpoint modes in a stream processing system, characterized in that, The method includes: During the process of writing status data based on the full checkpoint mode, the duration of a single data write and the amount of data written in a single instance are obtained. The data write decision score is determined based on the single data write duration, the single data write volume, the preset maximum disk throughput, and the preset data write interval. When the data writing decision score meets the preset switching conditions, the full checkpoint mode is switched to the incremental checkpoint mode so that the state data writing can continue to be performed based on the incremental checkpoint mode. Determine the asynchronous compression period, perform asynchronous compression operation based on the asynchronous compression period, and obtain the asynchronous compressed data volume of the asynchronous compression operation; When the amount of asynchronous compressed data is less than a preset degradation threshold, the incremental checkpoint mode is switched to the full checkpoint mode. The step of determining the data write decision score based on the single data write duration, the single data write volume, the preset maximum disk throughput, and the preset data write interval includes: The first ratio is determined based on the single data write duration and the preset data write interval; The data writing speed is determined based on the amount of data written in a single transaction and the duration of the single data write. The second ratio is determined based on the data write speed and the preset maximum disk throughput; The data is written into the decision score based on the first ratio and the second ratio.

2. The method according to claim 1, characterized in that, During the process of writing state data based on the incremental checkpoint mode, the method further includes: Determine the asynchronous compression period, and determine the target data write time based on the asynchronous compression period; When the actual time reaches the target data writing time, the actual checkpoint file generated by the corresponding data writing operation is identified, and an asynchronous compression operation is performed on the actual checkpoint file to generate the target full checkpoint file corresponding to the actual checkpoint file. The target full checkpoint file replaces the actual checkpoint file so that the state data writing can continue based on the target full checkpoint file.

3. The method according to claim 2, characterized in that, The asynchronous compression operation on the actual checkpoint file includes: Identify at least one historical checkpoint file that stores data dependencies with the actual checkpoint file; Based on data dependencies, the target full checkpoint file is determined according to the at least one historical checkpoint file and the actual checkpoint file.

4. The method according to claim 3, characterized in that, The at least one historical checkpoint file that identifies the data dependency relationship between the actual checkpoint file and the data storage relationship includes: Determine the metadata information of the actual checkpoint file; The data dependencies are determined based on the metadata information; The at least one historical checkpoint file is determined based on the data dependencies.

5. The method according to claim 4, characterized in that, The step of determining the target full checkpoint file based on data dependencies, according to the at least one historical checkpoint file and the actual checkpoint file, includes: The file merging order is determined based on the data dependencies; Based on the file merging order, the at least one historical checkpoint file and the actual checkpoint file are merged to obtain the target full checkpoint file.

6. The method according to claim 1, characterized in that, The method further includes: After switching from the full checkpoint mode to the incremental checkpoint mode, the asynchronous compressed data volume of multiple asynchronous compression operations is obtained. The preset degradation threshold is determined based on the amount of asynchronously compressed data from the multiple asynchronous compression operations.

7. The method according to claim 6, characterized in that, Determining the preset degradation threshold based on the asynchronous compressed data volume of the multiple asynchronous compression operations includes: The average asynchronous compressed data volume is determined based on the asynchronous compressed data volume of the multiple asynchronous compression operations. The preset degradation threshold is determined based on the average asynchronous compressed data volume and the preset coefficient.

8. The method according to claim 5, characterized in that, During the process of writing state data based on the incremental checkpoint mode, the method further includes: Get the amount of asynchronously compressed data for each asynchronous compression operation; The slope of incremental data change is determined based on the amount of asynchronous compressed data in each asynchronous compression operation; The asynchronous compression period is adjusted based on the slope of the incremental data change.

9. The method according to claim 8, characterized in that, The step of determining the incremental data change slope based on the asynchronous compressed data volume of each asynchronous compression operation includes: The asynchronous compression data volume of the current asynchronous compression operation and the asynchronous compression data volume of the previous asynchronous compression operation are determined based on the asynchronous compression data volume of each asynchronous compression operation. The amount of data change is determined based on the amount of asynchronously compressed data in the current asynchronous compression operation and the amount of asynchronously compressed data in the previous asynchronous compression operation; The incremental data change slope is determined based on the amount of data change and the asynchronous compression interval between the current asynchronous compression operation and the previous asynchronous compression operation.

10. The method according to claim 8, characterized in that, The step of adjusting the asynchronous compression period based on the slope of the incremental data change includes: The adjustment range is determined based on the slope of the incremental data change. The asynchronous compression period is adjusted according to the adjustment range.

11. The method according to claim 2, characterized in that, During the process of writing state data based on the incremental checkpoint mode, the method further includes: In response to a job failure recovery command, obtain the actual number of data writes after the job failure is recovered; When the actual data write count reaches a preset number, the first asynchronous compression operation after fault recovery is performed, wherein the time interval between the generation time of the job fault recovery instruction and the execution time of the first asynchronous compression operation is less than the asynchronous compression period.

12. The method according to claim 2, characterized in that, During the process of writing state data based on the incremental checkpoint mode, the method includes: Get the job throughput for each asynchronous compression operation; When the decrease in the job throughput is greater than a first preset percentage and the number of consecutive decreases reaches a first preset number, the disk throughput ratio and job throughput of the asynchronous compression operation are reduced. When the decrease rate of the job throughput is less than or equal to the first preset rate and greater than or equal to the second preset rate, and the number of consecutive decreases reaches the first preset number, the disk throughput ratio of the asynchronous compression operation is reduced, wherein the second preset rate is less than the first preset rate. When the decrease in the throughput of the operation is less than the second preset percentage and the number of consecutive decreases reaches the second preset number, the disk throughput ratio of the asynchronous compression operation is increased.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the checkpoint mode switching method of the stream processing system as described in any one of claims 1 to 12 when executing the computer program.

Images