Backup and recovery method and device based on computing power real-time task of intelligent computing center

Abstract

The invention provides a backup and recovery method and device based on a computing power real-time task of an intelligent computing center. The backup and recovery method comprises the steps of obtaining related parameters of a job snapshot configured by a user; obtaining configuration information configured for the created target data flow job or the created job snapshot by the user; calling a data stream processing engine according to the job snapshot configuration information or automatically recording a target job snapshot for the running target data stream job by the data stream processing engine, and storing the target job snapshot according to the storage path; acquiring a target job snapshot under the storage path according to a job identifier when the target data stream job runs and / or an acquired instruction for starting the target data stream job from the target job snapshot; and calling a data stream processing engine to start the target data stream job according to the target job snapshot. The operation can be quickly recovered from the newest operation snapshot, the recovery time after the system fails is remarkably shortened, the operation efficiency of the intelligent computing center can be improved, and therefore the overall computing power of the intelligent computing center is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computing power infrastructure, and in particular to a backup and recovery method and device based on real-time computing tasks of an intelligent computing center. Background Art

[0002] With the development of artificial intelligence technology and computing power technology, the concept of intelligent computing center has emerged. "Intelligent computing center" refers to the use of large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning and other scenarios) to provide the required computing power, data and algorithms. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0003] Flink (stream data processing platform) is a distributed processing engine that can perform stateful computing on bounded and unbounded data streams. Flink's fault tolerance mechanism can help jobs running on it recover and continue to execute after a failure. It saves the running job status in memory or external storage, so that these jobs can be quickly restored after being suspended or unexpectedly terminated.

[0004] Checkpoints and savepoints are job snapshots generated by flink for recovering from failures. Among them, checkpoints are job snapshots automatically generated by flink for recovering from failures. Their lifecycle is managed by flink. flink retains the most recent checkpoints while the job is running and deletes them when the job is canceled (checkpoints can also be configured to be retained and manually restored by the user). Savepoints are job snapshots manually triggered by the user to record operations such as redeployment / upgrade / rescaling of the state. In flink's native functions, if you want to use checkpoints, you must first enable the checkpoint function and configure the storage path and other parameters of the checkpoints, and then start the job on flink. flink will automatically generate multiple checkpoints during the job run, and the save path varies depending on the storage system. If you want to checkpoint and restore a job, you need to provide this path as a parameter to call the job's run command.

[0005] As for the use of the native Flink savepoint function, you also need to configure the savepoint storage path first. If the user needs to create a savepoint for a job running Flink, you need to provide the unique identifier of the job running on Flink as a parameter of the create savepoint command. If the creation is successful, Flink will return the storage path of the created savepoint. Depending on the storage file system, when the user wants to restore the job based on the savepoint, the path must be provided as a parameter of the restore job command; or when the user wants to delete the savepoint, the path must be provided as a parameter of the delete savepoint command.

[0006] However, for users running real-time jobs on Flink, they need to know the storage path of the job checkpoint and the unique identifier of the running instance in order to find the checkpoint or savepoint. These operations make the recovery mechanism of savepoints and checkpoints cumbersome, which limits the computing power improvement of the intelligent computing center. In the intelligent computing center, due to the importance of real-time data processing tasks, fast recovery capabilities are crucial. If the real-time job flow running on Flink is interrupted, the complex checkpoint and savepoint management may cause recovery delays, thereby affecting overall business efficiency. In addition, the time and energy required by users to configure and manage these checkpoints and savepoints may also distract them from other important data analysis and decision-making work, reducing the utilization rate of the computing resources of the intelligent computing center. Summary of the invention

[0007] The embodiments of the present invention provide a backup and recovery method and device based on the real-time tasks of the computing power of an intelligent computing center, so as to solve the technical problem in the related technology that the user needs to know the storage path of the save point or checkpoint or the unique identifier of the running instance in order to find the save point or checkpoint, which makes the backup and recovery mechanism of the checkpoint or save point more cumbersome and limits the improvement of the computing power of the intelligent computing center.

[0008] In a first aspect, an embodiment of the present invention provides a backup and recovery method based on a real-time task of computing power of an intelligent computing center, the method comprising:

[0009] Step S1: obtaining relevant parameters of the job snapshot configured by the user, wherein the relevant parameters at least include a storage path of the job snapshot;

[0010] Step S2: Obtain the target data flow job configuration or job snapshot configuration information created by the user;

[0011] Step S3: calling the data stream processing engine according to the job snapshot configuration information or the data stream processing engine automatically records the target job snapshot for the running target data stream job, and stores the target job snapshot according to the storage path;

[0012] Step S4: when the target data flow job stops unexpectedly or is stopped, after receiving an instruction from a user to start the target data flow job from the target job snapshot, obtaining the target job snapshot under the storage path according to the job identifier of the target data flow job when it is running and / or the instruction obtained to start the target data flow job from the target job snapshot;

[0013] Step S5: calling the data stream processing engine to start the target data stream job according to the target job snapshot, so that the target data stream job is started from the location where the target job snapshot is located.

[0014] Optionally, the target data flow job is an operator job.

[0015] Optionally, step S2 includes:

[0016] Step S21: Obtain the checkpoint parameters configured by the user for the target data flow job created.

[0017] Optionally, step S3 includes:

[0018] Step S31: the data stream processing engine automatically records a target job snapshot for the running target data stream job according to the job snapshot configuration information, and stores the target job snapshot according to the storage path.

[0019] Optionally, the step S31 includes:

[0020] Step S311: storing the checkpoint path of the target data flow job.

[0021] Optionally, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from a target data flow job checkpoint; the step S4 comprises:

[0022] Step S41: when the operator job stops unexpectedly or is stopped, after receiving a user instruction to start the target data flow job from the target data flow job checkpoint, obtain the target job snapshot in the storage path according to the job identifier of the target data flow job when it is running.

[0023] Optionally, the job identifier when the target data flow job is running is preset before the target data flow job is running.

[0024] Optionally, step S2 includes:

[0025] Step S22: obtaining a save point creation instruction created by the user for the target data flow job.

[0026] Optionally, step S3 includes:

[0027] Step S32: calling the data stream processing engine to record a target job snapshot for the running target data stream job according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

[0028] Optionally, the step S32 includes:

[0029] Step S321: storing the save point name, save point creation time, job identifier and save point path of the target data stream job.

[0030] Optionally, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from a save point of the target data flow job, and the instruction includes a save point name instruction; the step S4 includes:

[0031] Step S42: When the operator job stops unexpectedly or is stopped, after receiving an instruction from a user to start the target data flow job from the save point of the target data flow job, a snapshot of the target job is obtained under the storage path according to the job identifier of the target data flow job when it is running and the obtained save point name instruction.

[0032] Optionally, the target data flow job is an SQL job, the SQL job includes multiple SQL sections, and the SQL sections include multiple SQL statements.

[0033] Optionally, the step S2 includes:

[0034] Step S23: Obtain the checkpoint parameters configured by the user for the target data flow job created.

[0035] Optionally, step S3 includes:

[0036] Step S33: parsing the SQL statement, and calling the data stream processing engine in sequence to execute the SQL statement;

[0037] Step S34: the data stream processing engine automatically records a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and stores the target job snapshot according to the storage path;

[0038] If the SQL statement operates on the data flow, the SQL statement generates a job identifier for the target data flow job when it is executed; otherwise, the SQL statement does not generate a job identifier for the target data flow job when it is executed.

[0039] Optionally, the step S34 includes:

[0040] Step S341: storing the SQL statement checkpoint path of the target data flow job.

[0041] Optionally, step S4 includes:

[0042] Step S43: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the target data flow job from the target job snapshot, the target job snapshot is obtained under the storage path according to the job identifier of the SQL statement in the target data flow job.

[0043] Optionally, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from the SQL job checkpoint or an instruction for starting the target data flow job from the SQL paragraph checkpoint.

[0044] Optionally, before step S2, the method further includes:

[0045] Step S01: parsing the SQL statement, calling the data stream processing engine in sequence to execute the SQL statement, and making the SQL paragraph and SQL job to which the SQL statement belongs in a running state.

[0046] Optionally, step S2 includes:

[0047] Step S24: Obtain a savepoint creation instruction created by the user for the SQL job.

[0048] Optionally, step S3 includes:

[0049] Step S35: traverse all SQL sections in the SQL job, and traverse all SQL statements in the SQL sections;

[0050] Step S36: calling the data stream processing engine to record a target job snapshot for the running SQL statement according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

[0051] Optionally, the step S36 includes:

[0052] Step S361: storing the savepoint name, savepoint creation time, savepoint type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement savepoint path of the target data stream job.

[0053] Optionally, starting the target data stream job from the target job snapshot is starting the SQL job instruction from the SQL job savepoint, and the instruction includes a savepoint name and a savepoint type instruction; step S4 includes:

[0054] Step S44: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the SQL job from the SQL job save point, the target job snapshot is obtained under the storage path according to the obtained save point name and the save point type instruction.

[0055] Optionally, step S2 includes:

[0056] Step S25: Obtain a savepoint creation instruction created by the user for the SQL paragraph.

[0057] Optionally, step S3 includes:

[0058] Step S37: traverse all SQL statements in the SQL paragraph;

[0059] Step S38: calling the data stream processing engine to record a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

[0060] Optionally, the step S38 includes:

[0061] Step S381: storing the savepoint name, savepoint creation time, savepoint type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement savepoint path of the target data stream job.

[0062] Optionally, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the SQL paragraph from the SQL paragraph savepoint, and the instruction includes a savepoint name and a savepoint type instruction; step S4 includes:

[0063] Step S45: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the SQL segment from the SQL segment save point, the target job snapshot is obtained under the storage path according to the obtained save point name and the save point type instruction.

[0064] In a second aspect, an embodiment of the present invention provides a backup and recovery device based on real-time tasks of intelligent computing center computing power, the device comprising:

[0065] A parameter acquisition module, used to acquire relevant parameters of the job snapshot configured by the user, wherein the relevant parameters at least include a storage path of the job snapshot;

[0066] An information acquisition module is used to acquire the target data stream job configuration or job snapshot configuration information created by the user;

[0067] A backup module, configured to call the data stream processing engine according to the job snapshot configuration information or the data stream processing engine automatically records a target job snapshot for the running target data stream job, and stores the target job snapshot according to the storage path;

[0068] A query module, configured to obtain the target job snapshot under the storage path according to the job identifier of the target data flow job when it is running and / or the obtained instruction to start the target data flow job from the target job snapshot after receiving an instruction from a user to start the target data flow job from the target job snapshot under the condition that the target data flow job stops unexpectedly or is stopped;

[0069] The recovery module is used to call the data stream processing engine to start the target data stream job according to the target job snapshot, so that the target data stream job is started from the location of the target job snapshot.

[0070] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for backing up and restoring real-time tasks based on the computing power of an intelligent computing center as described in the first aspect.

[0071] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for backing up and restoring real-time tasks based on the computing power of an intelligent computing center as described in the first aspect are implemented.

[0072] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for backing up and restoring real-time tasks based on the computing power of an intelligent computing center as described in the first aspect.

[0073] According to the backup and recovery method based on the real-time task of the computing power of the intelligent computing center according to the embodiment of the present invention, the data stream processing engine is called based on the job snapshot configuration information or the data stream processing engine automatically records the job snapshot, and the data stream processing engine is called based on the job snapshot to start the job from the specified job snapshot, which can shield the technical processing details and create a job management mechanism for users that is easy to backup and restore. Users do not need to perform tedious manual operations, but can back up and restore in an intuitive way. In the intelligent computing center, data processing tasks often need to respond quickly to various real-time changes. Through the backup and recovery method of real-time tasks provided by the embodiment of the present invention, the job can be quickly restored from the latest job snapshot, significantly reducing the recovery time after the system failure, which can improve the operating efficiency of the intelligent computing center, and because the user's operation time in the checkpoint and savepoint management process is reduced, the computing power resources of the intelligent computing center can be more reasonably configured and utilized, thereby improving the overall computing power of the intelligent computing center. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0075] Figure 1 A flowchart of a backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention;

[0076] Figure 2 A flowchart of a backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention;

[0077] Figure 3 A flowchart of a backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention;

[0078] Figure 4 A flowchart of a backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention;

[0079] Figure 5 A flowchart of a backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention;

[0080] Figure 6 A flowchart of a backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention;

[0081] Figure 7A flowchart of a backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention;

[0082] Figure 8 A diagram showing an operator job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0083] Fig. 9 A diagram showing an operator job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0084] Fig.10 A diagram showing an operator job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0085] Fig.11 A diagram showing an operator job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0086] Fig.12 A diagram showing a method for backing up and restoring an operator job savepoint provided by an embodiment of the present invention;

[0087] Fig.13 A diagram showing a method for backing up and restoring an operator job savepoint provided by an embodiment of the present invention;

[0088] Fig.14 A diagram showing a SQL job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0089] Fig.15 A diagram showing a SQL job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0090] Fig.16 A diagram showing a SQL job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0091] Fig.17 A diagram showing a SQL job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0092] Fig.18 A diagram showing a SQL job checkpoint backup and recovery method provided by an embodiment of the present invention;

[0093] Fig.19 A diagram showing a SQL job savepoint backup and recovery method provided by an embodiment of the present invention;

[0094] Fig. 20 A diagram showing a SQL job savepoint backup and recovery method provided by an embodiment of the present invention;

[0095] Fig.21A diagram showing a SQL job savepoint backup and recovery method provided by an embodiment of the present invention;

[0096] Fig. 22 A structural block diagram of a backup and recovery method and device based on the real-time task of intelligent computing center computing power provided by an embodiment of the present invention;

[0097] Fig.23 A structural block diagram of a backup and recovery method and device based on the real-time task of intelligent computing center computing power provided by an embodiment of the present invention;

[0098] Fig.24 A structural block diagram of a backup and recovery method and device based on the real-time task of intelligent computing center computing power provided by an embodiment of the present invention;

[0099] Fig.25 A structural block diagram of a backup and recovery method and device based on the real-time task of intelligent computing center computing power provided by an embodiment of the present invention;

[0100] Fig.26 A structural block diagram of a backup and recovery method and device based on the real-time task of intelligent computing center computing power provided by an embodiment of the present invention;

[0101] Fig. 27 A structural block diagram of a backup and recovery method and device based on the real-time task of intelligent computing center computing power provided by an embodiment of the present invention;

[0102] Fig.28 A structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0103] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0104] The "computing power" mentioned in the present invention is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to execute certain computing requirements. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to the society through computing power infrastructure.

[0105] The "computing power (CP)" mentioned in the present invention is the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations per second (Floating Point Operations Per Second, FLOPS, where 1EFLOPS=10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs (Central Processing Unit) or 2 million mainstream notebooks. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 In the embodiment of the present application, the computing power adopts the half-precision floating point computing power (FP16) of the graphics card.

[0106] The "carrying capacity" (Network Power, NP) described in the present invention is a manifestation of the data transmission capability of computing facilities, including comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. Carrying capacity involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities. In the embodiment of the present application, carrying capacity uses video memory bandwidth. In the embodiment of the present application, storage capacity uses video memory bandwidth.

[0107] The "storage power" (SP) described in the present invention refers to the comprehensive capabilities of a data center in terms of data storage capacity, performance, safety and reliability, and green and low-carbon. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and server built-in storage devices. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB = 2^60bytes), and the commonly used unit of measurement for performance is the number of reads and writes per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB). The disaster recovery ratio is an important manifestation of safety and reliability. In the embodiment of the present application, the storage power adopts the video memory capacity.

[0108] The "computing power infrastructure" described in the present invention is a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity. It can realize centralized computing, storage, transmission and application of information, and presents characteristics such as diversity and ubiquity, intelligence and agility, safety and reliability, green and low-carbon. It is of great significance to promote industrial transformation and upgrading, enable my country's scientific and technological innovation, meet people's better life and realize high-efficiency social governance.

[0109] The “computing power” mentioned in the present invention includes general computing power CP 通用 、Intelligent computing power CP 智能 And super computing power CP 超级 .

[0110] The "general computing power" mentioned in the present invention refers to the computing power provided by servers based on CPU chips, which is used to support basic general computing such as cloud computing and edge computing.

[0111] The "intelligent computing power" mentioned in the present invention refers to the large-scale deployment of intelligent computing centers based on special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) for various innovative artificial intelligence applications, such as natural language processing, machine vision, etc.

[0112] The "super computing power" mentioned in the present invention is mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0113] The "intelligent computing center" mentioned in the present invention refers to a facility that provides the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning) by using large-scale heterogeneous computing resources, including general computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0114] The "computing resources" mentioned in the present invention refer to the technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guarantee resources such as wind, fire, water and electricity.

[0115] Figure 1 A backup and recovery method based on real-time tasks of intelligent computing center computing power provided by an embodiment of the present invention is shown, and the method includes:

[0116] Step S1: Obtain relevant parameters of a job snapshot configured by a user, wherein the relevant parameters at least include a storage path of the job snapshot.

[0117] Job snapshot related parameters need to be configured in advance, including the storage implementation of job snapshots (state.checkpoint-storage) and the storage path of job snapshots (state.checkpoints.dir).

[0118] Step S2: Obtain the target data stream job configuration or job snapshot configuration information created by the user.

[0119] After creating a target data flow job, the user configures or creates job snapshot configuration information for the target data flow job based on the required job snapshot properties.

[0120] Step S3: calling the data stream processing engine according to the job snapshot configuration information or the data stream processing engine automatically records the target job snapshot for the running target data stream job, and stores the target job snapshot according to the storage path.

[0121] The user clicks the button related to the target data flow job on the interface, passes the job snapshot configuration information to the data flow processing engine, and calls its native interface to start the target data flow job. The data flow processing engine records the job snapshot according to the job configuration information during the job running process. Among them, the data flow processing engine called is flink. If the target data flow job is terminated unexpectedly or the user clicks the cancel job button on the interface, step S4 is executed.

[0122] Step S4: when the target data flow job stops unexpectedly or is stopped, after receiving an instruction from a user to start the target data flow job from the target job snapshot, obtaining the target job snapshot under the storage path according to the job identifier of the target data flow job when it is running and / or the instruction obtained to start the target data flow job from the target job snapshot;

[0123] After receiving an instruction to start a target data flow job from a target job snapshot, the present invention obtains the target job snapshot under the storage path, without the need for the user to manually search for the job snapshot through the storage path.

[0124] Step S5: Call the data stream processing engine to start the target data stream job according to the target job snapshot, so that the target data stream job is started from the location where the target job snapshot is located.

[0125] That is to say, the Flink native interface is called with the job snapshot as the calling parameter to start the target data flow job from the specified job snapshot location.

[0126] Therefore, according to the backup and recovery method of real-time tasks in an embodiment of the present invention, the data stream processing engine is called based on the job snapshot configuration information, or the data stream processing engine automatically records the job snapshot, and the data stream processing engine is called based on the job snapshot to start the job from the specified job snapshot. This can shield the technical processing details and create a job management mechanism that is easy to backup and restore for users.

[0127] Users do not need to perform tedious manual operations, but can back up and restore in an intuitive way. In the intelligent computing center, data processing tasks often need to respond quickly to various real-time changes. Through the backup and recovery method of real-time tasks provided by the embodiment of the present invention, the job can be quickly restored from the latest job snapshot, significantly reducing the recovery time after a system failure, which can improve the operating efficiency of the intelligent computing center. In addition, since the user's operation time in the checkpoint and savepoint management process is reduced, the computing power resources of the intelligent computing center can be more reasonably configured and utilized, thereby improving the overall computing power of the intelligent computing center.

[0128] In one embodiment of the present invention, the target data flow job is an operator job.

[0129] Furthermore, if Figure 2 As shown, step S2 includes:

[0130] Step S21: Obtain the checkpoint parameters configured by the user for the target data stream job created.

[0131] The user creates an operator job and configures the current job parameters. For example, he selects options such as Enable checkpoints and Keep checkpoint parameters when canceling the job. After the user clicks the Start job button, the checkpoint-related parameters are passed to Flink.

[0132] Furthermore, the step S3 comprises:

[0133] Step S31: The data stream processing engine automatically records a target job snapshot for the running target data stream job according to the job snapshot configuration information, and stores the target job snapshot according to the storage path.

[0134] Preferably, step S31 includes: step S311: storing the checkpoint path of the target data flow job, that is, storing the target job snapshot at least includes storing the checkpoint path of the target data flow job.

[0135] That is, calling the data stream processing engine to start the target data stream job according to the target job snapshot is to call the flink native interface with the checkpoint path of the target data stream job as a parameter to start the target data stream job.

[0136] Further, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from a target data flow job checkpoint; the step S4 comprises:

[0137] Step S41: When the operator job stops unexpectedly or is stopped, after receiving a user instruction to start the target data flow job from the target data flow job checkpoint, obtain the target job snapshot in the storage path according to the job identifier when the target data flow job is running.

[0138] That is to say, the embodiment of the present invention automatically obtains the target job snapshot through the job identifier when the target data flow job is running, that is, it mainly finds the latest checkpoint path of the target data flow job, without the user having to search for the storage path or job identifier to obtain the target job snapshot.

[0139] Then, the flink native function is called with the latest checkpoint path as parameter to start the target dataflow job from the savepoint.

[0140] Preferably, the job identifier of the target data flow job when it is running is preset before the target data flow job is running.

[0141] Specifically, during the operation of the operator job, because the checkpoint storage path generated by flink automatically contains the job identifier (instance_id) of the instance currently running on flink, if it is not set in advance each time the job is started, flink will automatically regenerate this job identifier. When submitting the operator job, the operator job is converted into a JobGraph and then handed over to flink for execution. The operator job and the job running on flink are one-to-one corresponding. Therefore, the jobId (job ID) in the JobGraph can be set to the instance_id fixed value bound to the operator job in the present invention, ensuring that the same operator job has the same job identifier when running on flink, and the checkpoint can be found each time it runs according to this job identifier.

[0142] According to another embodiment of the present invention, Figure 3 As shown, step S2 includes:

[0143] Step S22: Obtain a save point creation instruction created by the user for the target data stream job.

[0144] Specifically, the user creates an operator job on the interface and clicks to start the job. During the job running, the user chooses to create a save point and enters the save point name.

[0145] Preferably, step S3 comprises:

[0146] Step S32: Calling the data stream processing engine to record a target job snapshot for the running target data stream job according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

[0147] Preferably, the step S32 includes: step S321: storing the save point name, save point creation time, job identifier and save point path of the target data flow job. That is, storing the target job snapshot at least includes storing the save point name, save point creation time, job identifier and save point path of the target data flow job.

[0148] That is to say, the embodiment of the present invention abstracts a savepoint information table, the main fields of which are information such as the savepoint name, savepoint creation time, job identifier and savepoint path. When the user creates a savepoint on the target data stream job, he only needs to provide the savepoint name to call flink to record the savepoint, and record the returned save path and savepoint name, creation time and other information into the savepoint information table.

[0149] Further, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from a save point of the target data flow job, and the instruction includes a save point name instruction; the step S4 includes:

[0150] Step S42: When the operator job stops unexpectedly or is stopped, after receiving an instruction from a user to start the target data flow job from the save point of the target data flow job, obtain a snapshot of the target job under the storage path according to the job identifier of the target data flow job when it is running and the obtained save point name instruction.

[0151] That is, after the job is terminated unexpectedly or the user clicks the cancel job button on the interface, the user selects the start job from save point button on the interface and selects a save point according to the job name. After obtaining the instruction to start the target data flow job from the save point of the target data flow job, the specific target job snapshot is found according to the job identifier in the save point information table and the save point name selected by the user, that is, the save point path of the target data flow job is mainly found.

[0152] Then, the flink native function is called with the savepoint path as parameter to start the target dataflow job from the savepoint.

[0153] In one embodiment of the present invention, the target data flow job is an SQL job, the SQL job includes multiple SQL sections, and the SQL sections include multiple SQL statements.

[0154] Furthermore, if Figure 4 As shown, step S2 includes:

[0155] Step S23: Obtain the checkpoint parameters configured by the user for the target data stream job created.

[0156] Specifically, the user configures the parameters of each SQL paragraph, for example, checking the checkpoint-enabling and checkpoint-keeping parameters when canceling a job, and other checkpoint parameters that need to be configured.

[0157] Preferably, step S3 comprises:

[0158] Step S33, parsing the SQL statement, and calling the data stream processing engine in sequence to execute the SQL statement;

[0159] Step S34: the data stream processing engine automatically records a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and stores the target job snapshot according to the storage path;

[0160] If the SQL statement operates on the data flow, the SQL statement generates a job identifier for the target data flow job when it is executed; otherwise, the SQL statement does not generate a job identifier for the target data flow job when it is executed.

[0161] Specifically, when the user clicks the start job button on the interface, the embodiment of the present invention performs the following operations according to the SQL paragraph loop: setting the checkpoint parameters and other environmental parameters, parsing multiple SQL statements in the current SQL paragraph, and calling the flink native interface in sequence to execute a single SQL statement. If it is an SQL statement that operates on a data stream, a running instance will be generated on flink, and the job identifier of the running instance will be returned. The embodiment of the present invention will record this job identifier, wherein flink automatically records checkpoints according to the paragraph configuration when running a job that operates an unbounded data stream, and automatically ends after running a job that operates a bounded data stream; if it is not an SQL statement that operates on a data stream, a running instance will not be generated on flink, and the job identifier of the running instance will not be returned.

[0162] Optionally, the step S34 includes: Step S341: storing the SQL statement checkpoint path of the target data flow job. That is, the storing of the target job snapshot at least includes storing the SQL statement checkpoint path of the target data flow job.

[0163] That is, calling the data stream processing engine according to the target job snapshot to start the target data stream job is to call the flink native interface with the SQL statement checkpoint path of the target data stream job as a parameter to start the target data stream job.

[0164] Furthermore, the step S4 comprises:

[0165] Step S43: When the SQL job stops unexpectedly or is stopped, after receiving an instruction from a user to start the target data flow job from the target job snapshot, obtain the target job snapshot under the storage path according to the job identifier of the SQL statement in the target data flow job.

[0166] Specifically, after the SQL statement job in a certain paragraph is terminated unexpectedly or the user clicks the Cancel paragraph or Cancel job button on the interface, when receiving the user's instruction to start the target data flow job from the target job snapshot, the embodiment of the present invention cyclically performs the following operations: according to the job identifier of each SQL statement in the SQL paragraph of the last run record, all checkpoint storage files of the last run are found in the currently configured checkpoint storage directory, and the latest file is obtained according to the modification time, and it is used as a parameter to call the flink native interface to start the statement.

[0167] Preferably, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from the SQL job checkpoint or an instruction for starting the target data flow job from the SQL paragraph checkpoint.

[0168] That is, the user can restore the job in two ways: directly click Start SQL job from latest checkpoint on the interface or select a SQL paragraph on the interface and click Start SQL paragraph from latest checkpoint.

[0169] According to one embodiment of the invention, Figure 5 As shown, before the step of obtaining the target data stream job configuration or job snapshot configuration information created by the user in step S2, the following step is also included:

[0170] Step S01, parse the SQL statement, call the data flow processing engine in sequence to execute the SQL statement, and put the SQL paragraph and SQL job to which the SQL statement belongs into a running state.

[0171] Specifically, according to an embodiment of the present invention, the following operations are performed in a loop: multiple SQL statements in the current paragraph are parsed, and the flink native interface is called in sequence to execute a single SQL statement. If there is an SQL statement for a popular operation on the data, a running instance will be generated on flink, and the paragraph and SQL job to which it belongs will be in a running state.

[0172] Furthermore, the step S2 comprises:

[0173] Step S24: Obtain a savepoint creation instruction created by the user for the SQL job.

[0174] Specifically, the user creates a SQL job on the interface and clicks Start Job. During the job running, the user chooses to create a job save point, enters a name and clicks OK.

[0175] Preferably, step S3 comprises:

[0176] Step S35, traverse all SQL sections in the SQL job, and traverse all SQL statements in the SQL sections;

[0177] Step S36: Call the data stream processing engine to record a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and store the target job snapshot according to the storage path.

[0178] That is, after receiving the user's instruction to create a job savepoint, it traverses all sections of the current SQL job and performs the following operations: it traverses all SQL statements in the current section, and for statements in the running stage, calls the flink native interface to store the target job snapshot for the current running job.

[0179] Further, step S36 includes: step S361: storing the save point name, save point creation time, save point type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement save point path of the target data flow job. That is, storing the target job snapshot at least includes storing the save point name, save point creation time, save point type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement save point path of the target data flow job.

[0180] Optionally, starting the target data stream job from the target job snapshot is starting the SQL job instruction from the SQL job savepoint, and the instruction includes a savepoint name and a savepoint type instruction; step S4 includes:

[0181] Step S44: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the SQL job from the SQL job save point, obtain the target job snapshot under the storage path according to the obtained save point name and the save point type instruction.

[0182] Specifically, when a job is terminated unexpectedly or the user clicks to cancel the job on the interface, the user chooses to start the job from a save point on the interface and selects a save point by name. The embodiment of the present invention finds the save point path of all statements of the corresponding job according to the save point name and save point type (job class) selected by the user, puts them into the startup parameters of each statement in turn, and calls the native interface to start all statements from the specified save point (if any).

[0183] In another embodiment of the present invention, Figure 6 As shown, before the step S2 of obtaining the target data stream job configuration or job snapshot configuration information created by the user, the following steps are included:

[0184] Step S01, parse the SQL statement, call the data flow processing engine in sequence to execute the SQL statement, and put the SQL paragraph and SQL job to which the SQL statement belongs into a running state.

[0185] Furthermore, the step S2 comprises:

[0186] Step S25: Obtain a savepoint creation instruction created by the user for the SQL paragraph.

[0187] Specifically, the user creates a SQL job on the interface and clicks Start Job. The user selects Create Segment Savepoint on a certain running segment of the job, enters a name and clicks OK.

[0188] Furthermore, the step S3 comprises:

[0189] Step S37, traverse all SQL statements in the SQL paragraph;

[0190] Step S38: Calling the data stream processing engine to record a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

[0191] Specifically, multiple SQL statements in the current paragraph are parsed, and the flink native interface is called in sequence to execute a single SQL statement. If there is an SQL statement that operates on the data stream, a running instance will be generated on flink, and the paragraph and SQL job to which it belongs will be in a running state.

[0192] After receiving a user's instruction to create a paragraph savepoint, the embodiment of the present invention traverses all SQL statements in the current paragraph, and for statements in the running stage, calls the flink native interface to store a target job snapshot for the current running job.

[0193] Preferably, the step S38 includes: step S381: storing the save point name, save point creation time, save point type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement save point path of the target data flow job. That is, storing the target job snapshot at least includes storing the save point name, save point creation time, save point type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement save point path of the target data flow job.

[0194] Further, the instruction for starting the target data stream job from the target job snapshot is an instruction for starting the SQL paragraph from the SQL paragraph savepoint, and the instruction includes a savepoint name and a savepoint type instruction; the step S4 includes:

[0195] Step S45: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the SQL segment from the SQL segment save point, obtain the target job snapshot under the storage path according to the obtained save point name and the save point type instruction.

[0196] When a paragraph ends unexpectedly or the user clicks Cancel paragraph on the interface of the present invention, the user chooses to start a SQL paragraph from a savepoint and selects a savepoint by name. The present invention finds the savepoint path of all statements of the corresponding paragraph according to the savepoint name and savepoint type (paragraph class) selected by the user, puts them into the startup parameters of each statement in turn, and calls the flink native interface to start all statements from the specified savepoint (if any).

[0197] It should be noted that for SQL jobs, users usually need to perform multiple steps to complete a requirement, including table creation, multi-table insertion, deletion, etc., which results in the SQL job consisting of multiple paragraphs and multiple statements. The present invention treats it as a whole to manage its save points and checkpoints, thereby avoiding users from getting caught up in the details of managing the save points and checkpoints of each SQL statement, which is more in line with the user's actual usage scenarios, allowing users to easily manage the checkpoints and savepoints of the entire job, and also realize differentiated management of the checkpoints and savepoints of multiple paragraphs in the same job. Figure 7 The process of saving and restoring different savepoint types (job type or paragraph type) of SQL jobs is shown.

[0198] In summary, the backup and recovery method of real-time tasks according to the embodiments of the present invention can automatically select different systems to generate and search for save points or check points according to a unified configuration. In addition, by maintaining the association between save points and check points and user jobs, users only need to focus on their business needs without having to understand technical details, which greatly simplifies the operating steps for users to use save points and check points.

[0199] Users do not need to perform tedious manual operations, but can back up and restore in an intuitive way. In the intelligent computing center, data processing tasks often need to respond quickly to various real-time changes. Through the backup and recovery method of real-time tasks provided by the embodiment of the present invention, the job can be quickly restored from the latest job snapshot, significantly reducing the recovery time after a system failure, which can improve the operating efficiency of the intelligent computing center. In addition, since the user's operation time in the checkpoint and savepoint management process is reduced, the computing power resources of the intelligent computing center can be more reasonably configured and utilized, thereby improving the overall computing power of the intelligent computing center.

[0200] The following describes the operation flow and basic principle of the real-time task backup and recovery method of the present invention according to a specific embodiment, wherein the storage of operator jobs using S3 (Simple Storage Service) and the storage of SQL jobs using HDFS (Hadoop Distributed File System) are taken as examples:

[0201] Example 1: Specific operation flow and basic principles of the operator job checkpoint backup and recovery method:

[0202] The user creates an operator job and configures the parameters of the current job: check Open checkpoint and Keep checkpoint parameters when canceling the job, and check other checkpoint parameters as needed, such as Figure 8 As shown;

[0203] Record the information about the job created by the user, number the current job, and assign it a job ID of a running instance on Flink: 3881ed1f89cbf64469dcadae7bd26906, such as Fig. 9 As shown;

[0204] The user clicks the start job button on the interface;

[0205] Pass the job ID of the pre-allocated Flink instance: 3881ed1f89cbf64469dcadae7bd26906 to the jobGraph before submission, pass the checkpoint related parameters to Flink and call its native start job interface;

[0206] Flink automatically records checkpoints according to the job configuration during the job running process, and the storage path of the generated checkpoints is in the preset checkpoint storage directory s3: / / / buckets / rdx799 / checkpoints plus the Flink instance job identifier: 3881ed1f89cbf64469dcadae7bd26906, such as Fig.10 As shown;

[0207] When the job is terminated unexpectedly or the user clicks the Cancel Job button on the interface, the user clicks the Start Job from Latest Checkpoint button again on the interface;

[0208] The present invention searches for all checkpoint storage files under the specified job identifier 3881ed1f89cbf64469dcadae7bd26906 in the currently configured checkpoint storage directory s3: / / / buckets / rdx799 / checkpoints, and obtains the latest file according to the modification time;

[0209] Use the latest checkpoint path as a parameter to call the Flink native interface to start the job. Fig.11 shown.

[0210] Embodiment 2: Specific operation flow and basic principle of operator job savepoint backup and recovery method:

[0211] The user creates an operator job on the interface and clicks to start the job;

[0212] The user selects Create Savepoint during the job run, enters a name, and clicks OK;

[0213] Call the flink native interface with the job ID of the current job running instance as a parameter to record the save point for the current job and return the save path;

[0214] The save point information, such as the job ID, save point name, the path returned by the previous step, the creation time, etc., is recorded in the save point information table, and the save is successfully returned to the front end. Fig.12 As shown;

[0215] When the job is terminated unexpectedly or the user clicks the cancel job button on the interface, the user selects to start the job from a save point on the interface of the present invention and clicks a save point by name;

[0216] Find the corresponding savepoint path s3: / buckets / rdx799 / savepoints / savepoint-9db08f-371d516e1058 according to the job ID of the current job and the savepoint name selected by the user;

[0217] Use the path found in the previous step as a parameter to call the Flink native interface to start the job from the specified save point. After starting, the job information is as shown in the figure below: Fig.13 shown.

[0218] Example 3: Specific operation flow and basic principle of SQL job checkpoint backup and recovery method:

[0219] The user creates a SQL job on the interface (this job contains multiple sections, each section contains multiple statements) and clicks to start the job;

[0220] The user configures the parameters of each section: check the checkpoint opening and keep the checkpoint parameters when canceling the job, and check other checkpoint parameters as needed, such as Fig.14 As shown;

[0221] The user clicks the start job button on the interface;

[0222] Perform the following operations according to the paragraph loop: set the checkpoint parameters and other environmental parameters, parse multiple SQL statements in the current paragraph, and call the Flink native interface in sequence to execute a single SQL statement;

[0223] If it is an SQL statement that operates on the data stream, a running instance will be generated on flink and the instance_id of the running instance will be returned. The present invention will record this instance_id; unbounded data streams will continue to run, and bounded data streams will automatically end after execution; SQL statements that do not operate on data streams will automatically end after flink is executed.

[0224] Flink automatically records checkpoints according to the paragraph configuration when running jobs that operate on unbounded data streams, such as Fig.15 and Fig.16 As shown;

[0225] When the SQL statements in one or more sections are terminated unexpectedly or the user clicks the Cancel Section or Cancel Job button on the interface, the user clicks the Start Job from Latest Checkpoint button on the interface;

[0226] Loop through all the sections of the current job and do the following: Based on the instance_id of each statement in the section recorded in the last run (the instance_id of the only statement in the select_countandsum section is 7d80f4b88916a85fe83c211847beb204 and the instance_id of the only statement in the insert_into_fromselect section is 47540d5e0cdbc8f3c35503fadd6a02df), find all the statement checkpoint storage files of the last run behind the currently configured checkpoint storage directory hdfs: / / nameservice1 / user / datacanvas / rdx7112 / flink-checkpoints, and get the latest file according to the modification time, and use it as a parameter to call the flink native interface to start the statement, such as Fig.17 and Fig.18 As shown, you can see that all statements operating unbounded data streams in the current job are started from the latest checkpoint.

[0227] Example 4: Specific operation flow and basic principle of SQL job savepoint backup and recovery method:

[0228] The user creates an SQL job on the interface. This job contains multiple paragraphs, and each paragraph contains multiple statements, which is the same as the previous embodiment. The invention performs the following operations according to the paragraph loop: parses multiple SQL statements in the current paragraph, and sequentially calls the flink native interface to execute a single SQL statement. If there is an SQL statement that operates on the data stream, a real-time job will be generated on flink, and the paragraph and SQL job to which it belongs will be in a running state;

[0229] During the job run, the user selects Create Job Savepoint, enters a name, and clicks OK;

[0230] Traverse all sections of the current job and perform the following operations: traverse all SQL statements in the current section, and for statements in the running stage, call the Flink native interface to record the save point for the current running job and return the save path, and record the save point information: current job id, section id, statement order, save point name, save point path, save point type, etc. to the save point information table;

[0231] like Fig.19As shown in the figure, the two savepoint save paths are: hdfs: / / nameservice1 / user / datacanvas / rdx7112 / flink-savepoints / savepoint-ce2d64-1079d1dfd19d and hdfs: / / nameservice1 / user / datacanvas / rdx7112 / flink-savepoints / savepoint-ffcd38-fc54ee615862;

[0232] After all the running statement save points (if any) of the current job are saved successfully, the save results are returned to the front end;

[0233] When the job is terminated unexpectedly or the user clicks the Cancel Job button on the interface of the present invention, the user selects Start Job from Save Point on the interface and selects a save point by name:

[0234] Find the savepoint paths of all statements of the corresponding job according to the savepoint name and savepoint type (job class) selected by the user, put them into the startup parameters of each statement in turn, and call the Flink native interface to start all statements from the specified savepoint (if any);

[0235] All statements in the current job run successfully. Statements that operate on data streams and have a previous save point are successfully started from the previous save point. For example, Fig. 20 and Fig.21 shown.

[0236] The present invention also provides a backup and recovery device 100 based on the real-time task of intelligent computing center computing power, such as Fig. 22 As shown, the device comprises:

[0237] A parameter acquisition module 10 is used to acquire relevant parameters of the job snapshot configured by the user, wherein the relevant parameters at least include a storage path of the job snapshot;

[0238] An information acquisition module 20 is used to acquire the target data stream job configuration or job snapshot configuration information created by the user;

[0239] A backup module 30, configured to call the data stream processing engine according to the job snapshot configuration information or the data stream processing engine automatically records a target job snapshot for the running target data stream job, and store the target job snapshot according to the storage path;

[0240] The query module 40 is used to obtain the target job snapshot under the storage path according to the job identifier of the target data flow job when it is running and / or the obtained instruction to start the target data flow job from the target job snapshot after receiving the instruction from the user to start the target data flow job from the target job snapshot under the condition that the target data flow job stops unexpectedly or is stopped;

[0241] The recovery module 50 is used to call the data stream processing engine to start the target data stream job according to the target job snapshot, so that the target data stream job is started from the location where the target job snapshot is located.

[0242] Optionally, the target data flow job is an operator job.

[0243] According to one embodiment of the present invention, Fig.23 As shown, the information acquisition module 20 also includes:

[0244] The first information acquisition module 21 is used to acquire the checkpoint parameters configured by the user for the target data stream job created.

[0245] Furthermore, the backup module 30 includes:

[0246] The first backup module 31 is used for the data stream processing engine to automatically record a target job snapshot for the running target data stream job according to the job snapshot configuration information, and store the target job snapshot according to the storage path.

[0247] Further, storing the target job snapshot at least includes storing a checkpoint path of the target data flow job.

[0248] Further, the instruction to start the target data flow job from the target job snapshot is an instruction to start the target data flow job from a target data flow job checkpoint; the query module 40 further includes:

[0249] The first query module 41 is used for obtaining a snapshot of the target job in the storage path according to the job identifier of the target data flow job when the target data flow job is running, after receiving an instruction from the user to start the target data flow job from the target data flow job checkpoint when the operator job stops unexpectedly or is stopped.

[0250] Preferably, the job identifier of the target data flow job when it is running is preset before the target data flow job is running.

[0251] In another embodiment of the present invention, Fig.24 As shown, the information acquisition module 20 includes:

[0252] The second information acquisition module 22 is used to acquire a save point creation instruction created by a user for the target data stream job.

[0253] Preferably, the backup module 30 further includes:

[0254] The second backup module 32 calls the data stream processing engine to record a target job snapshot for the running target data stream job according to the job snapshot configuration information, and stores the target job snapshot according to the storage path.

[0255] Further, storing the target job snapshot includes at least storing a save point name, a save point creation time, a job identifier, and a save point path of the target data flow job.

[0256] Optionally, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from a save point of the target data flow job, and the instruction includes a save point name instruction; the query module 40 further includes:

[0257] The second query module 42 is used to obtain a snapshot of the target job under the storage path according to the job identifier of the target data flow job when it is running and the obtained save point name instruction after receiving an instruction from the user to start the target data flow job from the save point of the target data flow job under the condition that the operator job stops unexpectedly or is stopped.

[0258] According to yet another embodiment of the present invention, the target data flow job is a SQL job, the SQL job includes a plurality of SQL sections, and the SQL sections include a plurality of SQL statements.

[0259] Preferably, if Fig.25 As shown, the information acquisition module 20 also includes:

[0260] The third information acquisition module 23 is used to acquire the checkpoint parameters configured by the user for the target data stream job created.

[0261] Furthermore, the backup module 30 also includes:

[0262] A first parsing module 331 is used to parse the SQL statement and sequentially call the data stream processing engine to execute the SQL statement;

[0263] A first storage module 332, which automatically records a target job snapshot for the SQL statement being run according to the job snapshot configuration information data stream processing engine, and stores the target job snapshot according to the storage path;

[0264] If the SQL statement operates on the data flow, the SQL statement generates a job identifier for the target data flow job when it is executed; otherwise, the SQL statement does not generate a job identifier for the target data flow job when it is executed.

[0265] In one embodiment of the present invention, storing the target job snapshot at least includes storing a SQL statement checkpoint path of the target data flow job.

[0266] Preferably, the query module 40 includes:

[0267] The third query module 43 is used to obtain the target job snapshot under the storage path according to the job identifier of the SQL statement in the target data flow job after receiving the user's instruction to start the target data flow job from the target job snapshot when the SQL job stops unexpectedly or is stopped.

[0268] Further, the instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from the SQL job checkpoint or an instruction for starting the target data flow job from the SQL paragraph checkpoint.

[0269] According to another embodiment of the present invention, Fig.26 As shown, the information acquisition module 20 also includes:

[0270] The fourth information acquisition module 24 is used to acquire a savepoint creation instruction created by the user for the SQL job.

[0271] Optionally, the backup module 30 includes:

[0272] The second parsing module 341 is used to parse the SQL statement, call the data flow processing engine in sequence to execute the SQL statement, and put the SQL paragraph and SQL job to which the SQL statement belongs into a running state.

[0273] A first traversal module 342 is used to traverse all SQL sections in the SQL job and traverse all SQL statements in the SQL sections;

[0274] The second storage module 343 is used to call the data stream processing engine to record the target job snapshot for the running SQL statement according to the job snapshot configuration information, and store the target job snapshot according to the storage path.

[0275] Further, storing the target job snapshot includes at least storing the save point name, save point creation time, save point type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement save point path of the target data flow job.

[0276] Preferably, the starting of the target data stream job from the target job snapshot is to start the SQL job instruction from the SQL job savepoint, and the instruction includes a savepoint name and a savepoint type instruction; the query module 40 includes:

[0277] The fourth query module 44 is used to obtain the target job snapshot under the storage path according to the obtained save point name and the save point type instruction after receiving the user's instruction to start the SQL job from the SQL job save point when the SQL job stops unexpectedly or is stopped.

[0278] In one embodiment of the present invention, Fig. 27 As shown, the information acquisition module 20 also includes:

[0279] The fifth information acquisition module 25 is used to acquire a savepoint creation instruction created by the user for the SQL paragraph.

[0280] Preferably, the backup module 30 further includes:

[0281] The second parsing module 341 is used to parse the SQL statement, call the data flow processing engine in sequence to execute the SQL statement, and make the SQL paragraph and SQL job to which the SQL statement belongs in a running state;

[0282] A second traversal module 352, used for traversing all SQL statements in the SQL paragraph;

[0283] The third storage module 353 is used to call the data stream processing engine to record the target job snapshot for the SQL statement being run according to the job snapshot configuration information, and store the target job snapshot according to the storage path.

[0284] Further, storing the target job snapshot includes at least storing the save point name, save point creation time, save point type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement save point path of the target data flow job.

[0285] Optionally, the instruction for starting the target data stream job from the target job snapshot is an instruction for starting the SQL paragraph from the SQL paragraph savepoint, and the instruction includes a savepoint name and a savepoint type instruction; the query module 40 further includes:

[0286] The fifth query module 45 is used to obtain the target job snapshot under the storage path according to the obtained save point name and the save point type instruction after receiving the user's instruction to start the SQL segment from the SQL segment save point when the SQL job stops unexpectedly or is stopped.

[0287] In summary, the backup and recovery device for real-time tasks according to the embodiment of the present invention can automatically select different systems to generate and search for save points or check points according to a unified configuration. In addition, by maintaining the association between save points and check points and user jobs, users only need to focus on their business needs without having to understand technical details, which greatly simplifies the operating steps for users to use save points and check points.

[0288] Users do not need to perform tedious manual operations, but can back up and restore in an intuitive way. In the intelligent computing center, data processing tasks often need to respond quickly to various real-time changes. Through the backup and recovery method of real-time tasks provided by the embodiment of the present invention, the job can be quickly restored from the latest job snapshot, significantly reducing the recovery time after a system failure, which can improve the operating efficiency of the intelligent computing center. In addition, since the user's operation time in the checkpoint and savepoint management process is reduced, the computing power resources of the intelligent computing center can be more reasonably configured and utilized, thereby improving the overall computing power of the intelligent computing center.

[0289] The embodiment of the present invention further provides an electronic device 200, such as Fig.28 As shown, it includes: a processor 210, a memory 220, and a program stored in the memory 220 and executable on the processor 210. When the program is executed by the processor, the steps of the backup and recovery method based on the real-time task of the intelligent computing center computing power as shown in the above embodiment are implemented.

[0290] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned backup and recovery method embodiment based on the real-time task of the intelligent computing center computing power is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0291] The embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the various processes of the above-mentioned backup and recovery method embodiment based on the real-time tasks of the intelligent computing center computing power, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0292] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0293] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0294] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A backup and recovery method based on real-time tasks of intelligent computing center computing power, characterized in that: The method comprises: Step S1: obtaining relevant parameters of the job snapshot configured by the user, wherein the relevant parameters at least include a storage path of the job snapshot; Step S2: Obtain the target data flow job configuration or job snapshot configuration information created by the user; Step S3: calling the data stream processing engine according to the job snapshot configuration information or the data stream processing engine automatically records the target job snapshot for the running target data stream job, and stores the target job snapshot according to the storage path; Step S4: when the target data flow job stops unexpectedly or is stopped, after receiving an instruction from a user to start the target data flow job from the target job snapshot, obtaining the target job snapshot under the storage path according to the job identifier of the target data flow job when it is running and / or the instruction obtained to start the target data flow job from the target job snapshot; Step S5: calling the data stream processing engine to start the target data stream job according to the target job snapshot, so that the target data stream job is started from the location where the target job snapshot is located.

2. The method according to claim 1, characterized in that The target data flow job is an operator job.

3. The method according to claim 2, characterized in that The step S2 comprises: Step S21: Obtain the checkpoint parameters configured by the user for the target data flow job created.

4. The method according to claim 3, characterized in that The step S3 comprises: Step S31: the data stream processing engine automatically records a target job snapshot for the running target data stream job according to the job snapshot configuration information, and stores the target job snapshot according to the storage path.

5. The method according to claim 4, characterized in that The step S31 comprises: Step S311: storing the checkpoint path of the target data flow job.

6. The method according to claim 5, characterized in that The instruction to start the target data flow job from the target job snapshot is an instruction to start the target data flow job from a target data flow job checkpoint; The step S4 comprises: Step S41: when the operator job stops unexpectedly or is stopped, after receiving a user instruction to start the target data flow job from the target data flow job checkpoint, obtain the target job snapshot in the storage path according to the job identifier of the target data flow job when it is running.

7. The method according to claim 6, characterized in that The job identifier when the target data flow job is running is preset before the target data flow job is running.

8. The method according to claim 2, characterized in that: The step S2 comprises: Step S22: obtaining a save point creation instruction created by the user for the target data flow job.

9. The method according to claim 8, characterized in that The step S3 comprises: Step S32: calling the data stream processing engine to record a target job snapshot for the running target data stream job according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

10. The method according to claim 9, characterized in that The step S32 comprises: Step S321: storing the save point name, save point creation time, job identifier and save point path of the target data stream job.

11. The method according to claim 10, characterized in that The instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from a save point of the target data flow job, and the instruction includes a save point name instruction; step S4 includes: Step S42: When the operator job stops unexpectedly or is stopped, after receiving an instruction from a user to start the target data flow job from the save point of the target data flow job, a snapshot of the target job is obtained under the storage path according to the job identifier of the target data flow job when it is running and the obtained save point name instruction.

12. The method according to claim 1, characterized in that The target data flow job is an SQL job, the SQL job includes multiple SQL sections, and the SQL sections include multiple SQL statements.

13. The method according to claim 12, characterized in that The step S2 comprises: Step S23: Obtain the checkpoint parameters configured by the user for the target data flow job created.

14. The method according to claim 13, characterized in that The step S3 comprises: Step S33: parsing the SQL statement, and calling the data stream processing engine in sequence to execute the SQL statement; Step S34: the data stream processing engine automatically records a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and stores the target job snapshot according to the storage path; If the SQL statement operates on the data flow, the SQL statement generates a job identifier for the target data flow job when it is executed; otherwise, the SQL statement does not generate a job identifier for the target data flow job when it is executed.

15. The method according to claim 14, characterized in that The step S34 comprises: Step S341: storing the SQL statement checkpoint path of the target data flow job.

16. The method according to claim 15, characterized in that The step S4 comprises: Step S43: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the target data flow job from the target job snapshot, the target job snapshot is obtained under the storage path according to the job identifier of the SQL statement in the target data flow job.

17. The method according to claim 15, characterized in that The instruction for starting the target data flow job from the target job snapshot is an instruction for starting the target data flow job from the SQL job checkpoint or an instruction for starting the target data flow job from the SQL paragraph checkpoint.

18. The method according to claim 12, characterized in that Before step S2, the method further includes: Step S01: parsing the SQL statement, calling the data stream processing engine in sequence to execute the SQL statement, and making the SQL paragraph and SQL job to which the SQL statement belongs in a running state.

19. The method according to claim 18, characterized in that The step S2 comprises: Step S24: Obtain a savepoint creation instruction created by the user for the SQL job.

20. The method according to claim 19, characterized in that The step S3 comprises: Step S35: traverse all SQL sections in the SQL job, and traverse all SQL statements in the SQL sections; Step S36: calling the data stream processing engine to record a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

21. The method according to claim 20, characterized in that The step S36 comprises: Step S361: storing the savepoint name, savepoint creation time, savepoint type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement savepoint path of the target data stream job.

22. The method according to claim 21, characterized in that The starting of the target data stream job from the target job snapshot is to start the SQL job instruction from the SQL job savepoint, and the instruction includes a savepoint name and a savepoint type instruction; the step S4 includes: Step S44: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the SQL job from the SQL job save point, the target job snapshot is obtained under the storage path according to the obtained save point name and the save point type instruction.

23. The method according to claim 18, characterized in that The step S2 comprises: Step S25: Obtain a savepoint creation instruction created by the user for the SQL paragraph.

24. The method according to claim 23, characterized in that The step S3 comprises: Step S37: traverse all SQL statements in the SQL paragraph; Step S38: calling the data stream processing engine to record a target job snapshot for the SQL statement being run according to the job snapshot configuration information, and storing the target job snapshot according to the storage path.

25. The method according to claim 24, characterized in that The step S38 comprises: Step S381: storing the savepoint name, savepoint creation time, savepoint type, SQL job identifier, SQL paragraph job identifier, SQL statement sequence and SQL statement savepoint path of the target data stream job.

26. The method according to claim 25, characterized in that The instruction for starting the target data stream job from the target job snapshot is an instruction for starting the SQL paragraph from the SQL paragraph savepoint, and the instruction includes a savepoint name and a savepoint type instruction; step S4 includes: Step S45: When the SQL job stops unexpectedly or is stopped, after receiving a user instruction to start the SQL segment from the SQL segment save point, the target job snapshot is obtained under the storage path according to the obtained save point name and the save point type instruction.

27. A backup and recovery device based on the real-time task of intelligent computing center computing power, characterized in that: The device comprises: A parameter acquisition module, used to acquire relevant parameters of the job snapshot configured by the user, wherein the relevant parameters at least include a storage path of the job snapshot; An information acquisition module is used to acquire the configuration information of a target data stream job created by a user or a job snapshot created; A backup module, configured to call the data stream processing engine according to the job snapshot configuration information or the data stream processing engine automatically records a target job snapshot for the running target data stream job, and stores the target job snapshot according to the storage path; A query module, configured to obtain the target job snapshot under the storage path according to the job identifier of the target data flow job when it is running and / or the obtained instruction to start the target data flow job from the target job snapshot after receiving an instruction from a user to start the target data flow job from the target job snapshot under the condition that the target data flow job stops unexpectedly or is stopped; The recovery module is used to call the data stream processing engine to start the target data stream job according to the target job snapshot, so that the target data stream job is started from the location of the target job snapshot.

28. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for backing up and restoring real-time tasks based on the computing power of an intelligent computing center as described in any one of claims 1 to 26.

29. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for backing up and restoring real-time tasks based on the computing power of an intelligent computing center as described in any one of claims 1 to 26.

30. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the method for backing up and restoring real-time tasks based on the computing power of an intelligent computing center as described in any one of claims 1 to 26.

Images