Checkpoint triggering method and device, electronic equipment and storage medium

Abstract

The invention discloses a Checkpoint triggering method and device, electronic equipment and a storage medium, and relates to the technical field of big data processing. The method comprises the steps of calculating a task priority of a target task, wherein the task priority is used for indicating resource allocation of the target task and an adjustment strategy of a time interval triggered by Checkpoint; according to the operator delay parameter and the transmission delay parameter of the target task and the load parameter of the target task, predicting a time delay parameter for executing the target task; according to the task priority and the time delay parameter, the target time interval triggered by the Checkpoint of the target task is determined, and the time interval triggered by the current Checkpoint of the target task is adjusted according to the target time interval, so that the data processing efficiency and the system resource utilization rate are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, and particularly relates to a checkpoint triggering method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the field of big data stream processing, Flink has become a widely used open source computing framework due to its high throughput and low latency. The checkpoint designed by Flink guarantees consistency and system fault tolerance during data processing.

[0003] However, in practice, Flink defaults to triggering checkpoint at fixed time intervals, which leads to the increasing limitations of Flink in the face of high-throughput and high-flow real-time data stream processing scenarios. For example, the fixed time interval triggering method lacks the ability to perceive the current load and data processing rate of the system, and during data flooding, frequent checkpoint triggering will compete with normal data processing tasks for computing, storage and network bandwidth resources, which not only occupies resources for core computing, but also significantly increases record processing latency due to blocking and synchronous waiting of task execution threads, thereby seriously weakening the real-time response capability of the stream processing system. SUMMARY

[0004] The present application provides a checkpoint triggering method, device, electronic equipment and storage medium to dynamically adjust the time interval of checkpoint triggering.

[0005] In a first aspect, a checkpoint triggering method is provided, comprising: calculating a task priority of a target task, the task priority being used to indicate an adjustment strategy of resource allocation of the target task and a time interval of checkpoint triggering; predicting a time delay parameter of executing the target task according to an operator delay parameter, a transmission delay parameter of the target task, and a load parameter of the target task; determining a target time interval of checkpoint triggering of the target task according to the task priority and the time delay parameter, and adjusting a current time interval of checkpoint triggering of the target task according to the target time interval.

[0006] By the above method, the time interval of Checkpoint triggering of the target task can be dynamically adjusted based on the calculated business priority of the target task and the predicted time delay parameter of executing the target task. Compared with the fixed time interval triggering Checkpoint method of the prior art, the time interval of Checkpoint triggering can be flexibly adjusted, thereby effectively reducing the processing overhead of Checkpoint and improving the system resource utilization. Further, based on the operator delay parameter of the target task, the transmission delay parameter, and the load parameter of the target task, the time delay parameter of executing the target task is predicted, the delay change of the target task under different time intervals of Checkpoint triggering is comprehensively evaluated, a decision basis for dynamically adjusting the time interval of Checkpoint triggering of the target task is provided, and the real-time performance influence of Checkpoint triggering on the target task is minimized.

[0007] In a possible design, the calculation of the task priority of the target task comprises: determining a time-sensitive parameter of the target task according to the maximum time parameter allowed for task execution delay and the time requirement parameter for completing execution of the target task; wherein the time-sensitive parameter is used to indicate the importance of completing execution of the target task according to the time requirement parameter; determining a fault tolerance parameter of the target task according to the time recovery parameter of the target task and the data loss parameter; wherein the time recovery parameter is used to indicate the allowed time for the target task to recover from a fault, and the data loss parameter is used to indicate the allowed data loss amount of the target task; determining a load parameter of the target task according to the target resource occupied by the target task; calculating the task priority of the target task according to the time-sensitive parameter, the fault tolerance parameter, and the load parameter.

[0008] The importance and complexity of the task can be quantified based on the time-sensitive requirement, fault tolerance requirement, and load condition of the task (or business), the task priority of different tasks can be comprehensively evaluated, and the task priority can be used as a decision basis for subsequent resource allocation of the target task and adjustment strategy of the time interval of Checkpoint triggering, so that the time interval of Checkpoint triggering of the task with high task priority can be shortened, the time interval of Checkpoint triggering of the task with low task priority can be increased, the time interval of Checkpoint can be flexibly triggered and adjusted, and the system resource utilization and data processing efficiency can be improved.

[0009] In a possible design, the time delay parameter satisfies the following expression:

[0010] wherein the is a set time delay parameter of the target task, the is a set time interval of Checkpoint triggering, is a weight parameter of the set time interval, the is an operator delay parameter of the target task, the is a transmission delay parameter of the target task, the is a load parameter of the target task, the , the and the are weight parameters of the operator delay parameter, the transmission delay parameter and the load parameter respectively.

[0011] Through the design, the sensitivity of the task to the delay can be quantified by comprehensively considering the operator delay parameter, the transmission delay parameter and the load parameter of the target task, the delay change of the time interval of Checkpoint triggering of different tasks can be predicted, a decision basis for dynamically adjusting the time interval of Checkpoint triggering of the target task is provided, and the influence of Checkpoint triggering on real-time performance is ensured to be minimal.

[0012] In a possible design, the target time interval satisfies the following expression:

[0013] wherein the is an average parameter of historical service priorities of the target task, is a service priority of the target task, the is a time delay parameter of the target task, the is an average parameter of historical time delay parameters of the target task, is a current time interval of Checkpoint triggering of the target task.

[0014] In a possible design, after the time interval of Checkpoint triggering of the target task is determined according to the task priority and the delay parameter, the method further includes: in response to obtaining a time delay update parameter and / or a load update parameter of the target task, updating the time interval of Checkpoint triggering of the target task according to the time delay update parameter and / or the load update parameter.

[0015] Through the design, the time interval of Checkpoint triggering can be recalculated and adjusted based on the updated parameters of load and / or time delay, so as to realize adaptive feedback.

[0016] In a possible design, the method further includes: collecting training data of at least one task; wherein the training data includes an operator delay parameter, a transmission delay parameter, a load parameter, and a topology parameter of the at least one task; training the training data of the at least one task to obtain a delay-sensitive prediction model; wherein the delay-sensitive prediction model is configured to predict a time delay parameter of executing each task.

[0017] In a second aspect, a Checkpoint triggering apparatus is provided, including

[0018] a calculation module configured to calculate a task priority of a target task; wherein the task priority is configured to indicate an adjustment strategy of resource allocation of the target task and a time interval of Checkpoint triggering; a prediction module configured to predict a time delay parameter of executing the target task according to an operator delay parameter, a transmission delay parameter of the target task, and a load parameter of the target task; an adjustment module configured to determine a target time interval of Checkpoint triggering of the target task according to the task priority and the time delay parameter, and adjust a current time interval of Checkpoint triggering of the target task according to the target time interval.

[0019] In a possible design, the calculation module is specifically configured to: determine a time-sensitive parameter of the target task according to a maximum time parameter of allowing task execution delay and a time requirement parameter of completing execution of the target task; wherein the time-sensitive parameter is configured to indicate an importance degree of completing execution of the target task according to the time requirement parameter; determine a fault tolerance parameter of the target task according to a time recovery parameter of the target task and a data loss parameter; wherein the time recovery parameter is configured to indicate an allowed time of the target task from failure to recovery, and the data loss parameter is configured to indicate an allowed data loss amount of the target task; determine a load parameter of the target task according to a target resource occupied by the target task; calculate the task priority of the target task according to the time-sensitive parameter, the fault tolerance parameter, and the load parameter.

[0020] In a third aspect, an electronic device is provided, including: A memory for storing a computer program; and a processor for implementing the method steps of any one of the first aspect when executing the computer program stored on the memory.

[0021] A fourth aspect provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method steps of any one of the first aspect.

[0022] The above-mentioned second aspect to fourth aspect and the technical effects that can be achieved by each aspect are described above in the description of the first aspect or various possible solutions in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 An application scenario suitable for the embodiments of the present application is shown in the figure; Figure 2 A method flowchart of a Checkpoint triggering method provided by the embodiments of the present application is shown in the figure; Figure 3 A method flowchart of a complete Checkpoint triggering method provided by the embodiments of the present application is shown in the figure; Figure 4 A structure schematic diagram of a Checkpoint triggering device provided by the embodiments of the present application is shown in the figure; Figure 5 A structure schematic diagram of an electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other at will. And, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0025] The terms "first" and "second" in the specification and claims of the present application and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device. "Multiple" in the present application can mean at least two, for example, can be two, three or more, and the embodiments of the present application are not limited.

[0026] The exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that in the embodiments of the present application, some software, components, models and the like may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or must have used the scheme.

[0027] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are described first as follows.

[0028] (1) Flink is an open source distributed stream processing framework, mainly used for real-time data processing. It can efficiently process large-scale data streams and batch data, support event-driven computing, and provide powerful state management and fault tolerance mechanisms, and is widely used in real-time analysis, monitoring, machine learning and other scenarios.

[0029] (2) Checkpoint is a key mechanism for ensuring data consistency and fault tolerance in distributed systems, which realizes fault recovery by saving system state snapshots regularly. For example, when the system fails or is interrupted, it can ensure that the task can continue to execute from the fault point by restoring to the latest Checkpoint state, without the need to restart.

[0030] (3) Service Level Agreement (SLA) is a mutual agreement between service providers and users or between service providers to ensure the performance and reliability of services within a certain cost (usually this cost is the main factor driving the quality of service).

[0031] The following will introduce the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the following introduction of the application scenarios is only used to illustrate the embodiments of the present application but not to limit. In the implementation, the technical solutions provided by the embodiments of the present application can be applied flexibly according to actual needs.

[0032] Figure 1 The application scenarios to which the embodiments of the present application are applicable are shown in the figure. As shown in the figure, the application scenarios mainly include: a terminal 10 and a server 11. Among them, the terminal 10 and the server 11 can perform information interaction through a communication network, and the communication network can adopt a communication mode including a wireless communication mode and a wired communication mode.

[0033] Optionally, the terminal 10 can access the network and communicate with the server 11 through a cellular mobile communication technology, and the cellular mobile communication technology can include a 5th Generation Mobile Networks (5G) technology.

[0034] Optionally, the terminal 10 can access the network and communicate with the server 11 through a short-range wireless communication mode, and the short-range wireless communication mode can include a Wireless Fidelity (Wi-Fi) technology.

[0035] The embodiments of the present application do not make any limitation on the number of the above-mentioned devices, and as shown in the figure, only the terminal 10 and the server 11 are taken as examples for description, and the following will briefly introduce the above-mentioned devices and their respective functions. Figure 1

[0036] The terminal 10 is a device that can provide voice and / or data connectivity for a user, including a handheld terminal device with wireless connection function, a vehicle-mounted terminal device, etc. For example, the terminal 10 includes but is not limited to a mobile phone, a tablet computer, a notebook computer, a palm computer, a Mobile Internet Device (MID), a wearable device, a Virtual Reality (VR) device, an Augmented Reality (AR) device, a wireless terminal device in industrial control, a wireless terminal device in unmanned driving, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, or a wireless terminal device in smart home, etc.

[0037] ​In addition, the terminal 10 can be installed with a checkpoint trigger related client, which can be a software (for example, APP, browser, etc.), a webpage, an applet, etc. In the embodiments of the present application, the terminal 10 can use the above-mentioned checkpoint trigger related client, and can interact with the server 11 to realize dynamic adjustment of the time interval of checkpoint triggering of different tasks.

[0038] Further, the server 11 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0039] Further, in the embodiments of the present application, the server 11 described above can be equipped with a checkpoint trigger device corresponding to the above-mentioned client, which is used to calculate the task priority of the target task, which is used to indicate the adjustment strategy of the resource allocation and the time interval of the checkpoint triggering of the target task; according to the operator delay parameter, the transmission delay parameter of the target task, and the load parameter of the target task, the time delay parameter of executing the target task is predicted; according to the task priority and the time delay parameter, the target time interval of the checkpoint triggering of the target task is determined, and the current time interval of the checkpoint triggering of the target task is adjusted according to the target time interval, thereby improving the data processing efficiency and the system resource utilization.

[0040] Based on the background art, Flink uses fixed time interval triggering checkpoint by default, which leads to the fact that its inherent limitations are increasingly prominent in the face of high throughput and large flow real-time data stream processing scenarios. For example, in the financial transaction scenario (or communication service scenario, medical service scenario, intelligent transportation scenario, etc.), frequent triggering of checkpoint will occupy a large amount of system resources, leading to increased processing delay, which seriously affects the real-time performance of business processing. Further, different businesses have different requirements for data consistency and real-time performance of business processing, and this fixed time interval triggering checkpoint mechanism cannot meet the differentiated needs in diversified business scenarios.

[0041] In view of this, the embodiments of the present application provide a checkpoint triggering method and device, electronic equipment and storage medium, which dynamically adjust the time interval of checkpoint triggering, thereby improving the data processing efficiency and system resource utilization.

[0042] To further illustrate the technical solutions provided by the embodiments of the present application, the following will describe in detail in combination with the drawings and specific embodiments. Although the embodiments of the present application provide the following method operation steps as shown in the embodiments or drawings, more or less operation steps can be included in the method based on conventional or non-creative labor. The execution order of the steps is not limited to the execution order provided by the embodiments of the present application in the logical sense. The method can be executed in the order shown in the embodiments or drawings or in parallel during the actual processing or device execution.

[0043] Figure 2 A method flowchart of a Checkpoint triggering method provided by the embodiments of the present application. The flowchart can be executed by a Checkpoint triggering device (the Checkpoint triggering device can be integrated in a server) to dynamically adjust the time interval of Checkpoint triggering. As shown in the flowchart, Figure 2 the flowchart includes the following steps: S201: calculating the task priority of the target task, which is used to indicate the adjustment strategy of the time interval of Checkpoint triggering and the resource allocation of the target task.

[0044] Optionally, the calculation of the task priority of the target task can be performed by steps A1-A4: Step A1: determining the time-sensitive parameter of the target task according to the maximum time parameter allowed for task execution delay and the time requirement parameter for completing the execution of the target task, which is used to indicate the importance of completing the execution of the target task according to the time requirement parameter.

[0045] Optionally, the time requirement parameter can be defined based on a service level agreement to specify how long the target task needs to complete processing. Further, the time-sensitive parameter can satisfy the following expression: ……(1) wherein, is the time requirement parameter of the target task, is the maximum time parameter set for allowing the target task to execute delay. The The smaller the time-sensitive parameter is, the higher the timeliness requirement of the target task is, and thus the higher the importance of completing the execution of the target task according to the time requirement parameter is.

[0046] For example, assuming that the target task is a financial monitoring task, milliseconds, milliseconds, the time-sensitive parameter of the financial monitoring task is calculated by the above expression (1) , which indicates that the real-time requirement of the financial monitoring task accounts for 20% of the maximum capacity of the system.

[0047] Step A2: determining the fault tolerance parameter of the target task according to the time recovery parameter of the target task, which is used to indicate the allowed time for the target task to recover from a failure, and the data loss parameter of the target task, which is used to indicate the allowed amount of data loss of the target task.

[0048] Optionally, the fault tolerance parameter can satisfy the following expression: … (2) wherein, is the time recovery parameter of the target task, indicating the allowed time for the target task to recover from a failure, is the data loss parameter of the target task, indicating the amount of data loss that the target task can tolerate, and are respectively the maximum recovery time parameter and the maximum data loss parameter supported by the system. The The smaller the value is, the higher the fault tolerance requirement is, and thus the more important it is to ensure the fault tolerance of the target task.

[0049] For example, taking the above-mentioned target task as the financial monitoring task, seconds, seconds, and the maximum recovery time parameter and the maximum data loss parameter supported by the system are respectively 2 hours and 1 hour, the fault tolerance parameter of the financial monitoring task calculated by the above-mentioned expression (2) is .

[0050] Step A3: determining the load parameter of the target task according to the target resource occupied by the target task.

[0051] Optionally, the load parameter is calculated based on at least one of the resource of the central processing unit occupied by the target task, the resource of the memory occupied by the target task, and the resource of the disk occupied by the target task, for example, calculated based on the number of central processing units occupied by the target task, the memory configuration ratio, the disk read-write speed, and the data source size. The load parameter can satisfy the following expression: … (3) wherein, , , respectively represent the resource of the central processing unit occupied by the target task, the resource of the memory occupied by the target task, and the resource of the disk occupied by the target task. The The larger the value is, the larger the load ratio of the target task is, and thus the more important the target task is.

[0052] For example, taking the financial monitoring task as an example, the financial monitoring task occupies 30% of the central processor, 40% of the memory, and 50% of the disk. The load parameter of the financial monitoring task is calculated by the above expression (3) .

[0053] Step A4: calculating the task priority of the target task according to the time parameter, the fault tolerance parameter and the load parameter.

[0054] Optionally, the task priority satisfies the following expression: ……(4) wherein, are weight parameters of the time parameter, the fault tolerance parameter and the load parameter respectively, which can reflect the relative importance of different indexes (i.e. the time parameter, the fault tolerance parameter and the load parameter) to the task priority of the target task. It should be noted that, The specific value of each weight parameter can be adjusted according to actual business needs, which is not limited here.

[0055] For example, taking the financial monitoring task as an example, assuming that the , , , the weight parameter is set to , and the business priority of the financial monitoring task is calculated based on the above expression (4) : The task priority determines the adjustment strategy of the resource allocation and the time interval of the Checkpoint triggering of the financial monitoring task. For example, the task with higher task priority will be allocated more resources and shorter time interval to trigger the Checkpoint.

[0056] Through the above steps A1-A4, the importance and complexity of the task can be quantified based on the time requirement, fault tolerance requirement and load situation of the task (or business), and the task priority of different tasks can be comprehensively evaluated, so as to be used as a decision basis for the subsequent resource allocation of the target task and the adjustment strategy of the time interval of the Checkpoint triggering. The task with high task priority can shorten the time interval of the Checkpoint triggering, and the task with low task priority can increase the time interval of the Checkpoint triggering, so as to realize flexible triggering and adjustment of the time interval of the Checkpoint, thereby improving the system resource utilization and data processing efficiency.

[0057] S202: predicting the time delay parameter of executing the target task according to the operator delay parameter and the transmission delay parameter of the target task, and the load parameter of the target task.

[0058] Optionally, the time delay parameter can be predicted by a trained delay-sensitive prediction model. The training process of the delay-sensitive prediction model can include the following: collecting training data of at least one task, the training data including operator delay parameters, transmission delay parameters, load parameters, and topology parameters of the at least one task, training the training data of the at least one task to obtain the delay-sensitive prediction model, and using the delay-sensitive prediction model to predict the time delay parameter of executing each task.

[0059] Optionally, the operator delay parameter of each task can be collected by a built-in measurement system of Flink to indicate the calculation time consumed by each operator in the task when processing each piece of data; the transmission delay of the task can be collected by a network monitoring tool (such as NetFlow, network packet analysis software, etc.) to indicate the time consumed by the data transmission between different operators of the task or between different task nodes; and the load parameter can be collected by using a tool such as Prometheus to indicate the usage of the resources (such as central processing unit, memory, and disk) currently occupied by the task.

[0060] Further, when the operator delay parameter, the transmission delay parameter, and the load parameter of the target task collected subsequently are input into the trained delay-sensitive prediction model, the time delay parameter of executing the target task can be predicted, thereby providing a decision basis for adjusting the time interval of triggering the Checkpoint of the target task, and ensuring that the triggering of the Checkpoint has the least impact on real-time performance.

[0061] In some embodiments, the training data of the at least one task can be updated according to a set period, and the model parameters of the delay-sensitive prediction model can be updated according to the updated training data, so as to improve the accuracy of the prediction result of the delay-sensitive prediction model.

[0062] Optionally, the time delay parameter can satisfy the following expression: ……(5) wherein, to predict the time delay parameter of the target task delay, C is a set time delay parameter, is a set time interval of triggering the Checkpoint, is the operator delay parameter of the target task, is the transmission delay parameter of the target task, is the load parameter of the target task, , and are weight parameters of the operator delay parameter, the transmission delay parameter, and the load parameter, respectively. It should be noted that the weight parameters of the operator delay parameter, the transmission delay parameter, and the load parameter can be set according to the actual situation. , and The specific values ​​can be flexibly set according to actual needs, and no restrictions are imposed here.

[0063] S203: Based on the task priority and time delay parameters mentioned above, determine the target time interval for the checkpoint trigger of the target task, and adjust the current checkpoint trigger time interval of the target task according to the target time interval.

[0064] Optionally, the target time interval satisfies the following expression: …(6) in, This is the average parameter representing the business priority of the target task. Prioritize the target task's business objectives. To predict the time delay parameters of the target task, The average parameter of the historical time delay parameter of the target task. The time interval at which the current checkpoint of the target task is triggered.

[0065] Furthermore, after determining the checkpoint triggering interval for the target task, in response to obtaining the target task's time delay update parameters and / or load update parameters, the checkpoint triggering interval for the target task is updated based on these parameters. For example, the various resources (CPU, memory, disk, etc.) used by the target task can be monitored in real time through Flink and external monitoring systems (such as Prometheus, Grafana, etc.). When the load parameters calculated from the various resources of the task change, the checkpoint triggering interval can be recalculated and adjusted based on the changed load parameters to achieve adaptive feedback. As another example, when changes are detected in the various delay parameters of the target task (operator delay parameters, transmission delay parameters, etc.), the checkpoint triggering interval can be recalculated and adjusted based on the changed delay parameters to achieve adaptive feedback.

[0066] Based on the above Figure 2 The method shown, Figure 3 This document provides a complete flowchart of a Checkpoint triggering method for an embodiment of this application. Figure 3 As shown, the process includes the following steps: S301: Collect training data for at least one task, and train the training data for at least one task to obtain a latency-sensitive prediction model.

[0067] S302: According to the set period, the time delay parameter of executing the target task is predicted based on the delay-sensitive prediction model.

[0068] S303: The task priority of the target task is determined according to the failure parameter, the fault tolerance parameter and the load parameter of the target task.

[0069] In this step, the specific calculation process of the failure parameter, the fault tolerance parameter and the load parameter can refer to the description in the foregoing, which will not be repeated here.

[0070] It should be noted that the specific order of S302 and S303 in the above steps is not limited, S302 and S303 can be executed simultaneously, or S303 can be executed first.

[0071] S304: According to the task priority and the time delay parameter, it is determined whether the time interval of the current Checkpoint triggering of the target task needs to be adjusted; if yes, it is turned to S305, and if no, it is turned to S302.

[0072] S305: The time interval of the current Checkpoint triggering of the target task is adjusted according to the determined target time interval.

[0073] It should be noted that the method shown in the above Figure 2 or Figure 3 is also applicable to the adjustment of the time interval of the Checkpoint triggering of batch tasks. Assuming that the maximum time parameter of the system supporting the execution task delay is 1000ms when the system is initialized. Taking task A, task B and task C as examples, the time interval of the Checkpoint triggering of task A, task B and task C can be determined by steps B1-B6: Step B1: The time efficiency parameter, the fault tolerance parameter and the load parameter of task A, task B and task C are calculated.

[0074] The calculation process of this step is similar to the foregoing, which will not be repeated here.

[0075] Assuming that task A: , =1.0, =0.2; task B: , =0.1, =0.8; task C: , , =0.5.

[0076] Step B2: The business priority of each task is calculated according to the time efficiency parameter, the fault tolerance parameter and the load parameter of task A, task B and task C.

[0077] Assume the weight parameter of the aging parameter = 0.6, the weight parameter of the fault tolerance parameter = 0.3, the weight parameter of the load parameter = 0.1, the business priority of each of Task A, Task B and Task C can be calculated as: ; ; .

[0078] Step B3: According to the operator delay parameter, the transmission delay parameter and the load parameter of each of Task A, Task B and Task C, the time delay parameter of each can be predicted.

[0079] Assume the set time delay parameter of the system is 100ms, the set time interval of the Checkpoint trigger , the current time interval of the Checkpoint trigger of each is , the weight parameter of the operator delay parameter 0.4, the weight parameter of the transmission delay parameter , the weight parameter of the load parameter , the time delay parameter of each of Task A, Task B and Task C can be predicted as: ; ; .

[0080] Step B4: According to the task priority and the time delay parameter of each of Task A, Task B and Task C, the target time interval of the Checkpoint trigger of each can be determined.

[0081] Assume the calculated task priority of Task A, Task B and Task C is 0.5, 0.7 and 0.4 respectively, and the time delay parameter is 186.1ms, 303.5ms and 207.4ms respectively, the target time interval of the Checkpoint trigger of each of Task A, Task B and Task C can be determined as: The target time interval of the Checkpoint trigger of Task A ; The target time interval of the Checkpoint trigger of Task B ; The target time interval of the Checkpoint trigger of Task C .

[0082] Step B5: Adjust the time interval of the current Checkpoint trigger of Task A, Task B and Task C respectively according to the target time interval of Task A, Task B and Task C respectively.

[0083] Based on the example of Step B4, the time interval of the Checkpoint trigger of Task A is adjusted to 2.2 seconds, the time interval of the Checkpoint trigger of Task B is adjusted to 17 seconds, and the time interval of the Checkpoint trigger of Task C is adjusted to 0.5 seconds.

[0084] Step B6: Track the execution of Task A, Task B and Task C by using a monitoring tool, and when a change in the time delay parameter and / or the load parameter of a certain task is monitored, adjust the Checkpoint trigger of the task according to the changed time delay parameter and / or the load parameter.

[0085] Suppose the time delay parameter of Task A changes from 200 ms to 250 ms, the new time interval of the Checkpoint trigger of Task A can be determined according to the new time delay parameter 250 ms.

[0086] In summary, by using the Checkpoint trigger method, the time interval of the Checkpoint trigger of the target task can be dynamically adjusted based on the calculated business priority of the target task and the predicted time delay parameter of executing the target task. Compared with the fixed time interval trigger Checkpoint method of the prior art, the time interval of the Checkpoint trigger can be flexibly adjusted, thereby effectively reducing the processing overhead of Checkpoint and improving the utilization of system resources. Further, based on the operator delay parameter of the target task, the transmission delay parameter and the load parameter of the target task, the time delay parameter of executing the target task is predicted, the delay change of the target task under different time intervals of the Checkpoint trigger is comprehensively evaluated, a decision basis for dynamically adjusting the time interval of the Checkpoint trigger of the target task is provided, and the real-time performance impact of the Checkpoint trigger on the target task is minimized.

[0087] Based on the same technical concept, the present application also provides a Checkpoint trigger device, which can implement the Checkpoint trigger method flow in the embodiments of the present application.

[0088] Figure 4 A structural diagram of a Checkpoint trigger device provided in the embodiments of the present application. The device comprises a calculation module 401, a prediction module 402 and an adjustment module 403.

[0089] The computing module 401 is configured to calculate a task priority of a target task, wherein the task priority is used to indicate an adjustment strategy of resource allocation and a time interval of checkpoint triggering of the target task.

[0090] The predicting module 402 is configured to predict a time delay parameter of executing the target task according to an operator delay parameter, a transmission delay parameter of the target task, and a load parameter of the target task.

[0091] The adjusting module 403 is configured to determine a target time interval of checkpoint triggering of the target task according to the task priority and the time delay parameter, and adjust a current time interval of checkpoint triggering of the target task according to the target time interval.

[0092] Optionally, the computing module 401 is specifically configured to: determine a time-sensitive parameter of the target task according to a maximum time parameter of allowing task execution delay and a time requirement parameter of completing the target task, wherein the time-sensitive parameter is used to indicate an importance degree of completing the target task according to the time requirement parameter; determine a fault-tolerant parameter of the target task according to a time recovery parameter and a data loss parameter of the target task, wherein the time recovery parameter is used to indicate an allowed time of the target task from failure to recovery, and the data loss parameter is used to indicate an allowed data loss amount of the target task; determine a load parameter of the target task according to a target resource occupied by the target task; and calculate the task priority of the target task according to the time-sensitive parameter, the fault-tolerant parameter, and the load parameter.

[0093] It should be noted that the above device provided by the embodiments of the present application can realize all the method steps in the method embodiments and achieve the same technical effects, and thus the same parts and beneficial effects of the method embodiments will not be described in detail herein.

[0094] Based on the same technical concept, the embodiments of the present application further provide an electronic device which can realize the functions of the above checkpoint triggering device.

[0095] Figure 5 A structural schematic diagram of an electronic device provided by the embodiments of the present application.

[0096] The electronic device provided by the embodiments of the present application comprises at least one processor 501 and a memory 502 connected with the at least one processor 501, and the embodiments of the present application do not limit the specific connection medium between the processor 501 and the memory 502, Figure 5The connection between the processor 501 and the memory 502 through the bus 500 is taken as an example. The bus 500 is connected between the processor 501 and the memory 502. Figure 5 The connection between the processor 501 and the memory 502 through the bus 500 is taken as an example. The bus 500 is connected between the processor 501 and the memory 502. Figure 5 The connection between the processor 501 and the memory 502 through the bus 500 is taken as an example. The bus 500 is connected between the processor 501 and the memory 502.

[0097] In the embodiments of the present application, the memory 502 stores instructions executable by the at least one processor 501, and the at least one processor 501 can execute the foregoing Checkpoint triggering method by executing the instructions stored in the memory 502. The processor 501 can implement the functions of various modules in the apparatus shown in the foregoing embodiments. Figure 4

[0098] The processor 501 is the control center of the apparatus, and can connect various parts of the control device through various interfaces and lines, and monitor the whole apparatus by running or executing instructions stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the whole apparatus.

[0099] In the embodiments of the present application, the processor 501 can include one or more processing units, and the processor 501 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication. It can be understood that the foregoing modem processor can also not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 can be implemented on the same chip, and in some embodiments, they can also be implemented on independent chips respectively.

[0100] The processor 501 can be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the Checkpoint triggering method disclosed in conjunction with the embodiments of the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0101] ​The memory 502 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 502 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 502 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0102] By designing and programming the processor 501, the code corresponding to the Checkpoint triggering method introduced in the foregoing embodiments can be fixed in the chip, so that the chip can execute the Checkpoint triggering method of the embodiments shown in the running time. Figure 2 How to design and program the processor 501 is a technology known to those skilled in the art, which will not be described here.

[0103] It should be noted that the above electronic device provided by the embodiments of the present application can realize all the method steps realized by the method embodiments and achieve the same technical effects, and the same parts and beneficial effects in the embodiments will not be described in detail.

[0104] Based on the same technical concept, the embodiments of the present application provide a computer storage medium, which includes computer program code, when the computer program code runs on the computer, so that the computer executes any one of the Checkpoint triggering methods discussed above. Since the principle of solving the problem of the above computer storage medium is similar to that of the Checkpoint triggering method, the implementation of the above computer storage medium can be referred to the implementation of the method, and the repeated parts will not be described here.

[0105] In a specific implementation process, the computer storage medium can include a universal serial bus flash drive (USB), a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage media that can store program codes.

[0106] Based on the same technical concept, the embodiments of the present application also provide a computer program product, which comprises computer program codes, and when the computer program codes run on a computer, the computer program codes make the computer execute any one of the Checkpoint triggering methods discussed above. Since the principle of solving problems of the above computer program product is similar to that of the Checkpoint triggering method, the implementation of the above computer program product can be referred to the implementation of the method, and the repeated parts will not be described here.

[0107] The computer program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0108] The method in the present application can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When realized by software, it can be realized in the form of a computer program product in whole or in part. The computer program product comprises one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, the processes or functions described in the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, a core network device, an OAM or other programmable devices.

[0109] The computer program or instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, e.g., from a website, computer, server, or datacenter to another website, computer, server, or datacenter via a wired or wireless arrangement. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computer, or a data storage device integrated into a server, datacenter, or the like, that includes one or more of the available media. The medium can be a magnetic medium, e.g., a floppy diskette, a hard disk drive, a magnetic tape, or the like; an optical medium, e.g., a compact disc, a DVD, or the like; a semiconductor medium, e.g., a solid state hard drive, or the like. The computer readable storage medium can be a volatile or non-volatile medium, or can include both volatile and non-volatile media.

[0110] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, magnetic disks or optical storage) embodying computer program code.

[0111] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 The flowchart illustrations and / or block diagrams Figure 1 The flowchart illustrations and / or block diagrams

[0112] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 The flowchart illustrations and / or block diagrams Figure 1 The flowchart illustrations and / or block diagrams

[0113] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or steps of the functions specified in the flowchart

[0114] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A Checkpoint triggering method, characterized in that, The method comprises: calculating a task priority of a target task; wherein the task priority is used to indicate an adjustment strategy of a resource allocation and a time interval of a Checkpoint triggering of the target task; predicting a time delay parameter of executing the target task according to an operator delay parameter, a transmission delay parameter of the target task, and a load parameter of the target task; determining a target time interval of the Checkpoint triggering of the target task according to the task priority and the time delay parameter, and adjusting a current time interval of the Checkpoint triggering of the target task according to the target time interval.

2. The method of claim 1, wherein, The calculating a task priority of a target task comprises: determining a time-sensitive parameter of the target task according to a maximum time parameter of allowing a task execution delay and a time requirement parameter of stipulating a time of completing the target task; wherein the time-sensitive parameter is used to indicate an importance degree of completing the target task according to the time requirement parameter; determining a fault-tolerant parameter of the target task according to a time recovery parameter of the target task and a data loss parameter; wherein the time recovery parameter is used to indicate an allowed time of the target task from a failure to recovery, and the data loss parameter is used to indicate an allowed data loss amount of the target task; determining a load parameter of the target task according to a target resource occupied by the target task; calculating the task priority of the target task according to the time-sensitive parameter, the fault-tolerant parameter, and the load parameter.

3. The method of claim 1, wherein, The time delay parameter satisfies the following expression: wherein the is a set time delay parameter for predicting a time delay of a target task, the C is a set time delay parameter, the is a set time interval for Checkpoint triggering, is a weight parameter of the set time interval, the is an operator delay parameter of a target task, the is a transmission delay parameter of a target task, the is a load parameter of a target task, the , the and the are weight parameters of the operator delay parameter, the transmission delay parameter, the load parameter, respectively.

4. The method of claim 1, wherein, The target time interval satisfies the following expression: wherein the is an average parameter of the historical service priority of the target task, is the service priority of the target task, the is a time delay parameter predicted for the target task, the is an average parameter of the historical time delay parameter of the target task, is the time interval of the current Checkpoint trigger of the target task.

5. The method of claim 1, wherein, After the determining a time interval of a Checkpoint triggering of a target task according to a task priority and a delay parameter, the method further comprises: in response to obtaining a time delay update parameter and / or a load update parameter of the target task, updating the time interval of the Checkpoint triggering of the target task according to the time delay update parameter and / or the load update parameter.

6. The method according to any one of claims 1 to 5, wherein, The method further comprises: collecting training data of at least one task; wherein the training data comprises an operator delay parameter, a transmission delay parameter, a load parameter, and a topology parameter of the at least one task; training the training data of the at least one task to obtain a delay-sensitive prediction model; wherein the delay-sensitive prediction model is used to predict a time delay parameter of executing each task.

7. A checkpoint triggering device, characterized in that The system comprises: a calculating module, configured to calculate a task priority of a target task; wherein the task priority is used to indicate an adjustment strategy of a resource allocation and a time interval of a Checkpoint triggering of the target task; a predicting module, configured to predict a time delay parameter of executing the target task according to an operator delay parameter, a transmission delay parameter of the target task, and a load parameter of the target task; and An adjusting module is configured to determine a target time interval of a checkpoint trigger of the target task according to the task priority and the time delay parameter, and adjust a current time interval of the checkpoint trigger of the target task according to the target time interval.

8. The apparatus of claim 7, wherein, The computing module is specifically configured to: determine a time limit parameter of the target task according to a maximum time parameter of a delay allowed by a task and a time requirement parameter of completing the target task, wherein the time limit parameter is used to indicate an importance degree of completing the target task according to the time requirement parameter; determine a fault tolerance parameter of the target task according to a time recovery parameter of the target task and a data loss parameter, wherein the time recovery parameter is used to indicate a time allowed by the target task from a fault to recovery, and the data loss parameter is used to indicate an amount of data loss allowed by the target task; determine a load parameter of the target task according to a target resource occupied by the target task; calculate a task priority of the target task according to the time limit parameter, the fault tolerance parameter and the load parameter.

9. An electronic device, comprising: The computer program is stored in the memory and executed by the processor, and the computer program is configured to implement the method in any one of claims 1-6. The computer program is stored in the memory and executed by the processor, and the computer program is configured to implement the method in any one of claims 1-6. ​ 10. A computer-readable storage medium, characterized in that, ​

Images