Job execution method and device

Abstract

The invention discloses a job execution method and device, and relates to the technical field of system operation and maintenance. A specific embodiment of the method comprises the steps of receiving an execution instruction for a target job; according to the execution sequence corresponding to the target job, determining and executing a current calculation process from a plurality of calculation processes of the target job; determining whether the current calculation process is a result storage process; responding to the current calculation process as a result storage process, storing an intermediate result of the current calculation process, and recording an execution state and execution starting time of the current calculation process; in response to an execution fault of the target job, determining a target calculation process with the latest execution starting time from calculation processes of which execution states represent normal execution; and executing job processing for the target job by using the intermediate result of the target calculation process. After a fault occurs, the implementation mode can reduce the time and adverse effects required by the system to carry out operation processing.

Description

Technical Field

[0001] The present invention relates to the technical field of system operation and maintenance, and in particular to a job execution method and device. Background Art

[0002] With the continuous development of various business systems, the amount of data accumulated by the system is getting larger and larger, and the logic of system processing is becoming more and more complex. The system usually completes various complex business logics through job processing. In the process of job processing, the system is bound to fail. Once a failure occurs, the system needs to re-process the job from the beginning. Re-processing the job takes a long time, affecting the normal progress of subsequent business. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and apparatus for executing a job, which can reduce the time required for the system to process a job after a failure occurs and the adverse effects caused by the system failure.

[0004] In a first aspect, an embodiment of the present invention provides a method for executing a job, including:

[0005] receiving an execution instruction for a target job;

[0006] Determine and execute a current computing process from among multiple computing processes of the target job according to the execution order corresponding to the target job;

[0007] Determine whether the current calculation process is a result saving process;

[0008] In response to the current computing process being a result saving process, saving an intermediate result of the current computing process, and recording an execution state and a start time of the current computing process;

[0009] In response to an execution failure of the target job, determining a target computing process having a latest start execution time from among the computing processes whose execution status represents normal execution;

[0010] Using the intermediate result of the target calculation process, job processing for the target job is performed.

[0011] Optionally, before receiving the execution instruction for the target job, the method further includes:

[0012] Acquiring efficiency data of the target operation;

[0013] In response to the efficiency data of the target job meeting the job optimization condition, determining a plurality of computing processes corresponding to the target job;

[0014] For each of the calculation processes: in response to the calculation process meeting the result saving condition, the calculation process is identified as a result saving process.

[0015] Optionally, the using the intermediate result of the target computing process to perform job processing for the target job includes:

[0016] determining at least one subsequent computing process of the target computing process;

[0017] According to the execution order corresponding to the target job, each subsequent calculation process is executed in sequence with the intermediate result of the target calculation process as input.

[0018] Optionally, in response to the target job having an execution failure, before determining the target computing process having the latest start execution time from the computing process in which the execution state characterizes the normal execution, the method further includes:

[0019] Determine the executing process corresponding to the target job;

[0020] In response to receiving error information for the executing process, or the execution time of the executing process being longer than a preset time, it is determined that an execution failure occurs in the target job.

[0021] Optionally, before using the intermediate result of the target computing process to perform job processing for the target job, the method further includes:

[0022] Determine a fault recovery strategy based on system resource information and / or error information of the job;

[0023] Performing fault recovery processing on the target job according to the fault recovery strategy.

[0024] Optionally, it also includes:

[0025] Get multiple business data;

[0026] According to the business attributes of each of the business data, the plurality of business data are divided into a plurality of data slices;

[0027] A data index for the business data on the data slice is created for each of the data slices.

[0028] Optionally, after creating a data index for the business data on each of the data slices, the method further includes:

[0029] Receive business inquiry requests;

[0030] Determine query information corresponding to the business query request;

[0031] Determine at least one target data slice matching the query information from the multiple data slices;

[0032] For each of the target data slices: using the data index of the target data slice, querying the response data on the target data slice;

[0033] The response data corresponding to each of the target data slices are combined to obtain the return data corresponding to the service request.

[0034] Optionally, it also includes:

[0035] Determine the target business identity;

[0036] Determining multiple associated systems of the target business identifier;

[0037] Querying the associated data corresponding to the target business identifier from each of the associated systems respectively;

[0038] The associated data corresponding to each of the associated systems are merged to generate a tracking record corresponding to the target business identifier.

[0039] In a second aspect, an embodiment of the present invention provides a job execution device, including:

[0040] An instruction receiving module, used for receiving an execution instruction for a target job;

[0041] A job execution module, configured to determine and execute a current computing process from among multiple computing processes of the target job according to an execution order corresponding to the target job;

[0042] A determination module, used to determine whether the current calculation process is a result saving process;

[0043] An information recording module, configured to save the intermediate results of the current computing process in response to the current computing process being a result saving process, and to record the execution state and start execution time of the current computing process;

[0044] a process determination module, configured to determine, in response to an execution failure of the target job, a target computing process having a latest start execution time from a computing process in which the execution state represents normal execution;

[0045] The job processing module is used to perform job processing for the target job by utilizing the intermediate result of the target computing process.

[0046] Optionally, it also includes:

[0047] A preprocessing module, used for obtaining efficiency data of the target operation;

[0048] In response to the efficiency data of the target job meeting the job optimization condition, determining a plurality of computing processes corresponding to the target job;

[0049] For each of the calculation processes: in response to the calculation process meeting the result saving condition, the calculation process is identified as a result saving process.

[0050] Optionally, the job processing module is specifically used to:

[0051] determining at least one subsequent computing process of the target computing process;

[0052] According to the execution order corresponding to the target job, each subsequent calculation process is executed in sequence with the intermediate result of the target calculation process as input.

[0053] Optionally, the process determination module is further used to:

[0054] Determine the executing process corresponding to the target job;

[0055] In response to receiving error information for the executing process, or the execution time of the executing process being longer than a preset time, it is determined that an execution failure occurs in the target job.

[0056] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0057] one or more processors;

[0058] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0059] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the above embodiments.

[0060] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described in any of the above embodiments when executed by a processor.

[0061] One embodiment of the above invention has the following advantages or beneficial effects: during the process of job processing of the target job, if the current calculation process is a result saving process, the intermediate result of the current calculation process is saved, and the execution status and start execution time of the current calculation process are recorded. When an execution failure occurs in the target job, the target calculation process with the latest start execution time is determined from the execution status characterization of the normal execution of the calculation process. Using the intermediate result of the target calculation process, the subsequent calculation process adjacent to the target calculation process is used as the starting point to complete the job processing of the target job. When an execution failure occurs in the job, there is no need to re-process the job from the beginning, which can reduce the time required for the system to process the job and the adverse effects caused by the system failure.

[0062] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0064] Figure 1 is a flowchart of a method for executing a job provided by the first embodiment of the present invention;

[0065] Figure 2 is a flowchart of a method for executing a job provided by a second embodiment of the present invention;

[0066] Figure 3 is a flowchart of a method for executing a job provided by a third embodiment of the present invention;

[0067] Figure 4 is a structural schematic diagram of a job execution device provided by an embodiment of the present invention;

[0068] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0070] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0071] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0072] Figure 1 is a flowchart of a method for executing a job provided by the first embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0073] Step 101: Receive an execution instruction for a target job.

[0074] Step 102: According to the execution order corresponding to the target job, determine and execute the current calculation process from multiple calculation processes of the target job.

[0075] The target job corresponds to multiple computing processes. According to the execution order corresponding to the target job, the computing processes corresponding to the target job are executed in sequence to complete the job processing of the target job.

[0076] Step 103: Determine whether the current calculation process is a result saving process.

[0077] Step 104: In response to the current computing process being a result saving process, save the intermediate result of the current computing process, and record the execution state and start execution time of the current computing process.

[0078] If the current computing process is a result saving process, save the intermediate results of the current computing process, and record the execution status and start time of the current computing process. The intermediate results are the processing results of the current computing process during execution. During the execution of the target job, the intermediate results may be deleted, modified, or discarded. The intermediate results can be saved in a cache, data file, or database. The execution status represents the execution status of the current computing process, and the execution status can be: normal execution, execution identification, executing, etc.

[0079] Step 105: In response to an execution failure of the target job, determine the target computing process having the latest start execution time from the computing processes that are normally executed by the execution status representation.

[0080] Step 106: Utilize the intermediate result of the target computing process to execute job processing for the target job.

[0081] At least one subsequent computing process of the target computing process is determined. Starting from the subsequent computing process immediately adjacent to the target computing process and taking the intermediate result of the target computing process as input, each subsequent computing process is executed in sequence to continue the job processing for the target job.

[0082] For example, the execution path corresponding to the target job is: A->B->C->D->E->F. A, B, C, D, E, and F are all computing processes, and A->B means that A must be executed before B, that is, A is a pre-dependent process of B.

[0083] If the target job has an execution failure, the target calculation process with the latest start execution time is determined to be C from the execution status representation of the normal execution calculation process, then the subsequent calculation process D next to C is taken as the starting point, and the intermediate result of C is used as input, and D, E, and F are executed in sequence to complete the job processing for the target job. In the scheme of the embodiment of the present invention, during the process of job processing of the target job, if the current calculation process is a result saving process, the intermediate result of the current calculation process is saved, and the execution status and start execution time of the current calculation process are recorded. When the target job has an execution failure, the target calculation process with the latest start execution time is determined from the execution status representation of the normal execution calculation process. The job processing of the target job is completed using the intermediate result of the target calculation process. Using the scheme of the embodiment of the present invention, when the job has an execution failure, there is no need to re-process the job from the beginning, which can reduce the time required for the system to process the job and the adverse effects caused by the system failure.

[0084] In one embodiment of the present invention, in response to an execution failure of a target job, multiple execution paths of the target job are determined; for each execution path, from the execution state representation of a normal computing process, the target computing process with the latest start execution time is determined from the multiple computing processes corresponding to the execution path.

[0085] Using the intermediate result of the target computing process to perform job processing for the target job includes: for each execution path, using the intermediate result of the target computing process in the execution path, performing execution processing for the execution path. According to the execution data of each execution path, the job processing for the target job is completed.

[0086] Each execution path consists of at least one calculation process. If the target job has multiple unfinished execution paths, determine the target calculation process corresponding to each execution path; for each execution path: use the intermediate result of the target calculation process corresponding to the execution path to perform the job processing for the execution path; use the processing results of each execution path to complete the job processing for the target job.

[0087] For example, the target job has an execution failure, and the target job has multiple unfinished pending execution paths: path 1 and path 2. Path 1 is: A->B->C, and path 2 is: A->D->E->F. Among them, A, B, C, D, E, and F are all calculation processes.

[0088] For path 1, determine A as the target execution calculation process, and use the intermediate result of A to execute the job processing for path 1. That is, take the subsequent calculation process B adjacent to A as the starting point, use the intermediate result of A as input, execute B and C in sequence, and complete the job processing of path 1.

[0089] For path 2, D is determined as the target execution calculation process, and the intermediate result of D is used to execute the job processing for path 2. That is, the subsequent calculation process E adjacent to D is used as the starting point, and the intermediate result of D is used as input, and E and F are executed in sequence to complete the job processing of path 2.

[0090] Finally, based on the job processing results of path 1 and path 2, the job processing for the target job is completed.

[0091] Furthermore, if there are multiple incomplete execution paths for the target job, at least one path to be executed is determined based on the dependency relationship of the execution paths; the target calculation process corresponding to each path to be executed is determined; for each path to be executed: the job processing for the path to be executed is performed using the intermediate result of the target calculation process corresponding to the path to be executed; and the job processing for the target job is completed using the processing results of each path to be executed.

[0092] At least one execution path to be executed is determined according to the dependency relationship of the execution paths. If all execution paths that the target execution path depends on have been executed, or the target execution path has no execution paths that it depends on, the target execution path is determined to be the execution path to be executed.

[0093] For example, if an execution failure occurs in the target job, the execution paths that have been completed for the target job are: path 1 and path 2. The execution paths that have not been completed for the target job are: path 3, path 4, and path 5. Among them, path 3 has no dependent execution paths, and the execution paths that path 4 depends on are path 1 and path 2, so path 3 and path 4 are determined as the paths to be executed. The execution paths that path 5 depends on are path 3 and path 4, so path 3 and path 4 need to be executed first before path 5 can be executed.

[0094] Path 3 and Path 4 are determined as the paths to be executed. For Path 3, since Path 3 has no dependent execution paths, Path 3 is directly executed.

[0095] For path 4, the target calculation process corresponding to path 4 is determined, and the job processing of path 4 is executed using the intermediate result of the target calculation process corresponding to path 4.

[0096] Finally, based on the job processing results of path 3 and path 4, the job processing for path 5 is executed to complete the job processing for the target job.

[0097] In one embodiment of the present invention, before receiving an execution instruction for a target job, it also includes: acquiring efficiency data of the target job; in response to the efficiency data of the target job meeting a job optimization condition, determining multiple computing processes corresponding to the target job; for each computing process: in response to the computing process meeting a result saving condition, identifying the computing process as a result saving process.

[0098] Efficiency data may include: job execution time, amount of data processed, probability of execution failure, etc. Job optimization conditions can be set according to needs, such as: job execution time is greater than the job time threshold, probability of execution failure is greater than the probability threshold, etc. If the efficiency data of the target job meets the job optimization conditions, the target job needs to be process-identified.

[0099] The result saving conditions can be set according to the needs, such as the result saving conditions can be: the process execution time is greater than the process time threshold, the data volume of the intermediate results is greater than the data volume threshold, etc. The target job that needs to be process-identified is determined through the job optimization conditions. The result saving process is determined from the multiple calculation processes of the target job through the result saving conditions, so that the calculation processes with long execution time or prone to errors can be marked as result saving processes. By saving the intermediate results of the result saving process during the execution process, the fault recovery efficiency of the target job is accelerated.

[0100] After identifying the calculation process as a result saving process, it also includes: determining the storage information and saving conditions corresponding to the calculation process. The storage information and saving conditions can be set according to needs. The storage information can be a storage medium, a storage path, etc. The saving condition can be that the storage duration is less than a duration threshold. According to the storage information, the intermediate results of the calculation process are saved in the storage medium and / or storage path corresponding to the storage information. If the intermediate results of the calculation process do not meet the saving conditions, the intermediate results of the calculation process are deleted to reduce the excessive storage space occupied.

[0101] In one embodiment of the present invention, it can be determined that an execution failure has occurred in a target job in the following manner: determining the executing process corresponding to the target job; in response to receiving an error message for the executing process, or the execution duration of the executing process being greater than a preset duration, determining that an execution failure has occurred in the target job.

[0102] If an error message is received for the executing process, or the execution time of the executing process is longer than the preset time, it is determined that the target job has an execution failure. It is possible to promptly determine whether the target job has an execution failure, so as to quickly complete the fault recovery of the target job based on the intermediate results of the target calculation process.

[0103] Figure 2is a flowchart of a job execution method provided by the second embodiment of the present invention, such as Figure 2 As shown, the method includes:

[0104] Step 201: Receive an execution instruction for a target job.

[0105] Step 202: According to the execution order corresponding to the target job, determine and execute the current calculation process from multiple calculation processes of the target job.

[0106] Step 203: Determine whether the current calculation process is a result saving process.

[0107] Step 204: In response to the current computing process being a result saving process, save the intermediate result of the current computing process, and record the execution state and start execution time of the current computing process.

[0108] Step 205: In response to an execution failure of the target job, determine the target computing process having the latest start execution time from the computing processes that are normally executed by the execution status representation.

[0109] Step 206: Determine at least one subsequent computing process of the target computing process.

[0110] Step 207: According to the execution order corresponding to the target job, each subsequent calculation process is executed in sequence with the intermediate result of the target calculation process as input.

[0111] In the solution of the embodiment of the present invention, if the target job has an execution failure, the target computing process with the latest start execution time is determined from the computing process in which the execution status characterizes the normal execution. By using the intermediate results of the target computing process, the subsequent computing processes of the target computing process are executed in sequence, and the execution process of the target job can be quickly restored, reducing unnecessary repeated operations.

[0112] Figure 3 is a flowchart of a method for executing a job provided by the third embodiment of the present invention, such as Figure 3 As shown, the method includes:

[0113] Step 301: Receive an execution instruction for a target job.

[0114] Step 302: According to the execution order corresponding to the target job, determine and execute the current computing process from multiple computing processes of the target job.

[0115] Step 303: Determine whether the current calculation process is a result saving process.

[0116] Step 304: In response to the current computing process being a result saving process, save the intermediate result of the current computing process, and record the execution state and start execution time of the current computing process.

[0117] Step 305: In response to an execution failure of the target job, a failure recovery strategy is determined according to system resource information and / or job error information.

[0118] Fault recovery strategies may include: troubleshooting network faults, checking for viruses, upgrading specific software, troubleshooting mount points and partitions, troubleshooting hardware faults, checking DNS (Domain Name System) resolution, etc. Build a mapping relationship between the system's resource information and / or the job's error information and the recovery strategy. Determine the fault recovery strategy based on the mapping relationship. The system's resource information and / or the job's error information may also be input into the fault analysis model, and the fault recovery strategy may be determined based on the input of the fault analysis model.

[0119] Step 306: Perform fault recovery processing on the target job according to the fault recovery strategy.

[0120] Step 307: Determine the target computing process with the latest start execution time from among the normal computing processes represented by the execution status.

[0121] Step 308: Utilize the intermediate results of the target computing process to perform job processing for the target job.

[0122] In the solution of the embodiment of the present invention, a fault recovery strategy is determined based on the resource information of the system and / or the error information of the job. After the target job is fault-recovered according to the fault recovery strategy, the job processing for the target job is continued to be completed to reduce the risk of the target job having another execution failure.

[0123] In one embodiment of the present invention, it also includes: acquiring multiple business data; dividing the multiple business data into multiple data slices according to the business attributes of each business data; and creating a data index for the business data on the data slice for each data slice.

[0124] After creating data indexes for the business data on each data slice respectively, the method also includes: receiving a business query request; determining query information corresponding to the business query request; determining at least one target data slice matching the query information from multiple data slices; for each target data slice: querying response data on the target data slice using the data index of the target data slice; combining the response data corresponding to each target data slice to obtain return data corresponding to the business request.

[0125] By sharding data and using a distributed computing framework for parallel processing, data is divided into multiple smaller pieces, each with independent storage and processing capabilities. Data sharding helps reduce the load on a single node and improve parallel processing capabilities. Distributed storage ensures high availability and reliability of data. By creating indexes for each piece of data, data can be quickly located and accessed, improving query efficiency.

[0126] In one embodiment of the present invention, it also includes: determining the target business identifier; determining multiple associated systems of the target business identifier; respectively querying the associated data corresponding to the target business identifier from each associated system; merging the associated data corresponding to each associated system to generate a tracking record corresponding to the target business identifier. According to the business identifier, a tracking record corresponding to each target business identifier is generated. The tracking record saves the complete path from the source to the final processing result. The tracking record can ensure the traceability and compliance of the data, and can further improve the feedback efficiency of the system.

[0127] The embodiment of the present invention further provides a job execution method based on distributed storage, and the specific processing method includes the following steps:

[0128] Step 1: Split the business data into multiple smaller pieces to improve the parallelism of data processing. Use a distributed storage system to store data pieces on multiple nodes in the network. The distributed storage system has high availability and scalability.

[0129] Step 2: Create an index for each data piece to quickly locate and access the data. The data index is built based on one or more attributes of the data. Attributes may include: transaction time, transaction type, etc.

[0130] Step 3: Parallel processing of jobs. Using a distributed computing framework to process business data stored in a distributed storage system in parallel can significantly improve the speed and efficiency of data processing. Among them, the distributed computing framework can use Hadoop. The steps to build the Hadoop distributed computing framework are as follows:

[0131] S1. Environment setup. Install the Hadoop cluster, including NameNode (which manages file system metadata) and DataNode (which stores actual data). Configure the Hadoop cluster's network, file system, and other related settings.

[0132] S2. Data upload: Use Hadoop's file system commands to upload business data to distributed storage.

[0133] S3. Write a MapReduce program to implement the data processing logic. The Map phase is responsible for data slicing and mapping, and the Reduce phase is responsible for data aggregation and reduction.

[0134] S4. Submit the MapReduce job to the cluster for execution.

[0135] S5. Monitoring and debugging. Use the UI or command line tools to monitor the execution status and resource usage of the job, and debug and optimize the performance of the job as needed.

[0136] S6. Obtain results. After the job is completed, obtain the processing results from the distributed storage.

[0137] Step 4: Introduce stream processing during job processing. Use stream processing frameworks such as Apache Kafka and Apache Flink to process business data in real time. Analyze data as it is generated, rather than waiting for batch processing to complete. Build an event-based system that triggers real-time alerts or downstream processing when specific events occur. Specific events may include: large transactions, abnormal transactions, etc.

[0138] Step 5: Provide external services through the data lake and data warehouse.

[0139] The data lake is used to store raw, unprocessed transaction data, retaining the original format and details of the data. Based on the data lake, the data warehouse builds a cleaned, integrated and optimized data warehouse for high-performance analytical queries. Data federation is used to achieve the collaborative work of the data lake and the data warehouse, so that data in different formats and structures can be accessed and analyzed under a unified query interface.

[0140] Step 6: Introduce graph database to complete business query. Use graph database to store and query complex relationships in transaction data, such as many-to-many relationships between customers, accounts, and transactions. Use graph database and graph algorithms to identify and track abnormal behaviors and events.

[0141] Step 7: Establish a distributed cache mechanism. Cache hot data, use distributed cache systems such as Redis, cache frequently accessed transaction data or calculation results, and reduce the access pressure on backend storage. For complex and time-consuming calculations, cache intermediate results to avoid repeated calculations.

[0142] Step 8: Generate tracking records. Tracking records are used to save the complete path of data from the source to the final processing result to ensure data traceability and compliance.

[0143] Step 9: Implement automated operation and maintenance and elastic expansion. Use containerization technology and automation tools (such as Kubernetes, Docker, etc.) to achieve automated deployment and monitoring of data processing processes. Use containerization technology to dynamically adjust the allocation of computing resources and storage resources based on the processing requirements and load of transaction data.

[0144] Step 10: Data aggregation. After processing the jobs of each data slice, aggregate the processing results to obtain the final job processing results. Data aggregation can be achieved through the aggregation operation provided by the distributed computing framework.

[0145] Step 11: Data backup and recovery. To ensure data security and reliability, it is necessary to regularly back up business data and develop corresponding recovery strategies. In a distributed storage system, data redundancy and fault tolerance mechanisms can be used to achieve data backup and recovery.

[0146] In the solution of the embodiment of the present invention, by introducing a stream processing framework, business data can be processed and analyzed in real time without waiting for all data batches to be processed, which can respond to market changes and customer needs more quickly. The steps of automated operation and maintenance and elastic expansion ensure that the data processing process can be flexibly adjusted and expanded according to demand, and the use of containerization technology and automation tools simplifies the deployment and monitoring process. The elastic scaling mechanism allows dynamic adjustment of computing and storage resources according to load conditions.

[0147] The consistency and integrity of business data are ensured through data indexing, data aggregation and data tracking technologies. Data indexing helps to quickly locate and access data. Data aggregation provides a consistent view of the final processing results. Data backup and recovery strategies ensure the security and reliability of business data. Data is redundantly stored on multiple nodes with fault-tolerant mechanisms to prevent data loss or damage.

[0148] Figure 4 is a schematic diagram of a structure of a job execution device provided by an embodiment of the present invention, such as Figure 4 As shown, the device comprises:

[0149] The instruction receiving module 401 is used to receive an execution instruction for a target job;

[0150] The job execution module 402 is used to determine and execute the current computing process from multiple computing processes of the target job according to the execution order corresponding to the target job;

[0151] Determination module 403, used to determine whether the current calculation process is a result saving process;

[0152] The information recording module 404 is used to save the intermediate results of the current calculation process in response to the current calculation process being a result saving process, and to record the execution state and start execution time of the current calculation process;

[0153] A process determination module 405 is used to determine the target computing process with the latest start execution time from the computing processes whose execution status represents the normal execution in response to the execution failure of the target job;

[0154] The job processing module 406 is used to perform job processing for a target job using the intermediate results of the target computing process.

[0155] Optionally, it also includes:

[0156] A preprocessing module, used to obtain efficiency data of the target job;

[0157] In response to the efficiency data of the target job meeting the job optimization condition, determining a plurality of computing processes corresponding to the target job;

[0158] For each computing process: in response to the computing process meeting the result saving condition, the computing process is identified as a result saving process.

[0159] Optionally, the job processing module 406 is specifically used for:

[0160] determining at least one subsequent computing process of the target computing process;

[0161] According to the execution order corresponding to the target job, each subsequent calculation process is executed in sequence with the intermediate result of the target calculation process as input.

[0162] Optionally, the process determination module 405 is further configured to:

[0163] Determine the executing process corresponding to the target job;

[0164] In response to receiving error information for the executing process, or the execution time of the executing process being longer than a preset time, it is determined that an execution failure occurs in the target job.

[0165] Optionally, it also includes:

[0166] A fault recovery module is used to determine a fault recovery strategy based on system resource information and / or job error information;

[0167] Perform failure recovery on the target job according to the failure recovery policy.

[0168] Optionally, it also includes:

[0169] Sharding module, used to obtain multiple business data;

[0170] According to the business attributes of each business data, multiple business data are divided into multiple data slices;

[0171] A data index for the business data on each data slice is created for each data slice.

[0172] Optionally, it also includes:

[0173] Request response module, used to receive business query requests;

[0174] Determine query information corresponding to the business query request;

[0175] Determine at least one target data slice matching the query information from the multiple data slices;

[0176] For each target data slice: using the data index of the target data slice, query the response data on the target data slice;

[0177] The response data corresponding to each target data piece is combined to obtain the return data corresponding to the business request.

[0178] Optionally, it also includes:

[0179] A tracking module for determining the target business identifier;

[0180] Determine multiple associated systems of target business identification;

[0181] Query the associated data corresponding to the target business identifier from each associated system respectively;

[0182] The associated data corresponding to each associated system is merged to generate a tracking record corresponding to the target business identifier.

[0183] An embodiment of the present invention provides an electronic device, including:

[0184] one or more processors;

[0185] The storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0186] An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the job execution method in the embodiment of the present invention.

[0187] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0188] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0189] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0190] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0191] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0192] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, they may be described as: an instruction receiving module, a job execution module, a determination module, an information recording module, a process determination module, and a job processing module. The names of these modules do not constitute limitations on the modules themselves in certain circumstances, for example, the instruction receiving module may also be described as a "module for receiving execution instructions for a target job".

[0194] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0195] receiving an execution instruction for a target job;

[0196] Determine and execute a current computing process from among multiple computing processes of the target job according to the execution order corresponding to the target job;

[0197] Determine whether the current calculation process is a result saving process;

[0198] In response to the current computing process being a result saving process, saving an intermediate result of the current computing process, and recording an execution state and a start time of the current computing process;

[0199] In response to an execution failure of the target job, determining a target computing process having a latest start execution time from among the computing processes whose execution status represents normal execution;

[0200] Using the intermediate result of the target calculation process, job processing for the target job is performed.

[0201] According to the technical solution of the embodiment of the present invention, during the process of job processing of the target job, if the current calculation process is a result saving process, the intermediate result of the current calculation process is saved, and the execution status and start execution time of the current calculation process are recorded. When an execution failure occurs in the target job, the target calculation process with the latest start execution time is determined from the execution status characterization of the normal execution calculation process. The job processing of the target job is completed using the intermediate result of the target calculation process. Using the solution of the embodiment of the present invention, when an execution failure occurs in the job, there is no need to re-process the job from the beginning, which can reduce the time required for the system to process the job and the adverse effects caused by the system failure.

[0202] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for executing a job, characterized in that: include: receiving an execution instruction for a target job; Determine and execute a current computing process from among multiple computing processes of the target job according to the execution order corresponding to the target job; Determine whether the current calculation process is a result saving process; In response to the current computing process being a result saving process, saving an intermediate result of the current computing process, and recording an execution state and a start time of the current computing process; In response to an execution failure of the target job, determining a target computing process having a latest start execution time from among the computing processes whose execution status represents normal execution; Using the intermediate result of the target calculation process, job processing for the target job is performed.

2. The method according to claim 1, characterized in that Before receiving the execution instruction for the target job, the method further includes: Acquiring efficiency data of the target operation; In response to the efficiency data of the target job meeting the job optimization condition, determining a plurality of computing processes corresponding to the target job; For each of the calculation processes: in response to the calculation process meeting the result saving condition, the calculation process is identified as a result saving process.

3. The method according to claim 1, characterized in that: The using the intermediate result of the target computing process to perform job processing for the target job includes: determining at least one subsequent computing process of the target computing process; According to the execution order corresponding to the target job, each subsequent calculation process is executed in sequence with the intermediate result of the target calculation process as input.

4. The method according to claim 1, characterized in that: In response to the target job having an execution failure, before determining the target computing process with the latest start execution time from the computing process in which the execution state characterizes the normal execution, the method further includes: Determine the executing process corresponding to the target job; In response to receiving error information for the executing process, or the execution time of the executing process being longer than a preset time, it is determined that an execution failure occurs in the target job.

5. The method according to claim 1, characterized in that Before executing the job processing for the target job by using the intermediate result of the target computing process, the method further includes: Determine a fault recovery strategy based on system resource information and / or error information of the job; Performing fault recovery processing on the target job according to the fault recovery strategy.

6. The method according to claim 1, characterized in that Also includes: Get multiple business data; According to the business attributes of each of the business data, the plurality of business data are divided into a plurality of data slices; A data index for the business data on the data slice is created for each of the data slices.

7. The method according to claim 6, characterized in that After the data index for the business data on the data slice is created for each data slice respectively, the method further includes: Receive business inquiry requests; Determine query information corresponding to the business query request; Determine at least one target data slice matching the query information from the multiple data slices; For each of the target data slices: using the data index of the target data slice, querying the response data on the target data slice; The response data corresponding to each of the target data slices are combined to obtain the return data corresponding to the service request.

8. The method according to claim 1, characterized in that Also includes: Determine the target business identity; Determining multiple associated systems of the target business identifier; Querying the associated data corresponding to the target business identifier from each of the associated systems respectively; The associated data corresponding to each of the associated systems are merged to generate a tracking record corresponding to the target business identifier.

9. A work execution device, characterized in that: include: An instruction receiving module, used for receiving an execution instruction for a target job; A job execution module, configured to determine and execute a current computing process from among multiple computing processes of the target job according to an execution order corresponding to the target job; A determination module, used to determine whether the current calculation process is a result saving process; An information recording module, configured to save the intermediate results of the current computing process in response to the current computing process being a result saving process, and to record the execution state and start execution time of the current computing process; a process determination module, configured to determine, in response to an execution failure of the target job, a target computing process having a latest start execution time from a computing process in which the execution state represents normal execution; The job processing module is used to perform job processing for the target job by utilizing the intermediate result of the target computing process.

10. The device according to claim 9, characterized in that Also includes: A preprocessing module, used for obtaining efficiency data of the target operation; In response to the efficiency data of the target job meeting the job optimization condition, determining a plurality of computing processes corresponding to the target job; For each of the calculation processes: in response to the calculation process meeting the result saving condition, the calculation process is identified as a result saving process.

11. The device according to claim 9, characterized in that The job processing module is specifically used for: determining at least one subsequent computing process of the target computing process; According to the execution order corresponding to the target job, each subsequent calculation process is executed in sequence with the intermediate result of the target calculation process as input.

12. The device according to claim 9, characterized in that The process determination module is also used for: Determine the executing process corresponding to the target job; In response to receiving error information for the executing process, or the execution time of the executing process being longer than a preset time, it is determined that an execution failure occurs in the target job.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

14. A computer readable medium having a computer program stored thereon, characterized in that When the program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images