A data processing method and related equipment

Abstract

The present invention discloses a data processing method and related equipment, which can determine the surplus time that matches the allowed response time of the target node to the target job, determine the output interval time of the checkpoint based on the surplus time and the maximum throughput of the target node in the process of executing the target job, and control the target node to output the checkpoint in the process of executing the target job according to the output interval time. The present invention can effectively avoid the job recovery time being too long and not meeting the allowed response time requirement, and can minimize the resource consumption required for writing the checkpoint and performing fault recovery.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method and related equipment. Background Art

[0002] With the development of computer technology, data processing technology continues to improve.

[0003] Currently, nodes can provide fault tolerance by outputting checkpoints during job processing. Specifically, when a node fails during job processing, it can be restored to the state corresponding to the checkpoint without having to restart the job processing, which can effectively improve job processing efficiency.

[0004] However, the existing technology cannot determine the effective output interval length of the checkpoint. Summary of the invention

[0005] In view of the above problems, the present invention provides a data processing method and related devices that overcome the above problems or at least partially solve the above problems. The technical solutions are as follows:

[0006] A data processing method, the method comprising:

[0007] Determine a surplus time that matches an allowed response time of the target node to the target job;

[0008] Determine the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job;

[0009] According to the output interval duration, the target node is controlled to output a checkpoint during execution of the target job.

[0010] Optionally, determining the surplus time that matches the allowed response time of the target node to the target job includes:

[0011] Determining an allowable response time of the target node to the target job;

[0012] In the historical execution duration data of the target node for the target job, determining the execution duration ranked at a preset percentile as the target execution duration;

[0013] The surplus time is determined based on the allowed response time and the target execution time.

[0014] Optionally, determining the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job includes:

[0015] The product of the surplus time and the maximum throughput is determined as the output interval time.

[0016] Optionally, determining the surplus time based on the allowed response time and the target execution time includes:

[0017] The difference between the allowed response time and the target execution time is determined as the surplus time.

[0018] A data processing device, comprising: a first determining unit, a second determining unit and a control unit; wherein:

[0019] The first determining unit is used to determine a surplus time length that matches an allowed response time length of the target node to the target job;

[0020] The second determining unit is used to determine the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job;

[0021] The control unit is used to control the target node to output a checkpoint during execution of the target job according to the output interval duration.

[0022] Optionally, the first determining unit includes: a third determining unit, a fourth determining unit and a fifth determining unit;

[0023] The third determining unit is used to determine the allowed response time of the target node to the target job;

[0024] The fourth determining unit is configured to determine, in the historical execution duration data of the target node for the target job, an execution duration ranked at a preset percentile as a target execution duration;

[0025] The fifth determining unit is used to determine the surplus time based on the allowed response time and the target execution time.

[0026] Optionally, the second determination unit is used to determine the output interval duration as the product of the surplus time duration and the maximum throughput.

[0027] Optionally, the fifth determination unit is used to determine the difference between the allowed response time and the target execution time as the surplus time.

[0028] An electronic device comprises at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute the above-mentioned data processing method.

[0029] A computer-readable storage medium stores a program, which implements the above data processing method when executed by a processor.

[0030] The data processing method and related equipment proposed in the present invention can determine the surplus time that matches the allowed response time of the target node to the target job, determine the output interval time of the checkpoint based on the surplus time and the maximum throughput of the target node in the process of executing the target job, and control the target node to output the checkpoint in the process of executing the target job according to the output interval time. The present invention can effectively avoid the job recovery time being too long and failing to meet the allowed response time requirement, and can minimize the resource consumption required for writing the checkpoint and performing fault recovery.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0033] Figure 1 A flow chart of a first data processing method provided by an embodiment of the present invention is shown;

[0034] Figure 2 A flow chart of a second data processing method provided by an embodiment of the present invention is shown;

[0035] Figure 3 A schematic diagram showing the structure of a first data processing device provided by an embodiment of the present invention is shown;

[0036] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0038] like Figure 1 As shown, this embodiment proposes a first data processing method. The method may include the following steps:

[0039] S101, determining a surplus time that matches the allowed response time of the target node to the target job;

[0040] The target node may be an electronic device used to process the target job, such as a mobile phone or a tablet computer.

[0041] The target job may be a batch job or a non-batch job, which is not limited in the present invention.

[0042] The response time may be the time required for the target node to complete the process of executing the target job from receiving the processing instruction of the target job to completing the process of executing the target job.

[0043] The allowed response time may be the allowed response time when the target node processes the target job.

[0044] Optionally, the allowed response time may be the maximum allowed response time, or may be a value smaller than the maximum allowed response time.

[0045] Optional, such as Figure 2 As shown, in the second data processing method proposed in this embodiment, step S101 may include:

[0046] S201, determining the allowed response time of the target node to the target job;

[0047] S202: In the historical execution time data of the target node for the target job, determine the execution time ranked at a preset percentile as the target execution time;

[0048] S203: Determine the surplus time based on the allowed response time and the target execution time.

[0049] Specifically, the allowed response time can be set by the technician according to the target operation and actual situation, and the present invention does not limit this.

[0050] The historical execution time data may include the execution time required for the target node to process the target job in history.

[0051] Among them, the preset percentile can be set by technical personnel according to actual conditions, and the present invention is not limited to this.

[0052] Specifically, the present invention can arrange the execution times required for all historical target nodes to process target jobs in order from small to large, and then determine the execution times that are arranged in the order of the preset percentile of the total as the target execution time. For example, when the preset percentile is the 95th percentile, the present invention can arrange the execution times required for all historical target nodes to process target jobs, and then determine the execution times that are arranged in the order of the 95th percentile of the total as the target execution time.

[0053] Specifically, the present invention can calculate the surplus time according to the allowed response time and the target execution time.

[0054] Optionally, the above-mentioned determination of the surplus time based on the allowed response time and the target execution time includes:

[0055] The difference between the allowed response time and the target execution time is determined as the surplus time.

[0056] Specifically, after obtaining the allowed response time and the target execution time, the present invention can subtract the allowed response time from the target execution time to obtain the surplus time.

[0057] S102, determining the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job;

[0058] Specifically, the present invention can determine the maximum throughput of the target node when processing the target job based on the current idle resources and specific execution performance of the target node.

[0059] Specifically, after obtaining the surplus time and the maximum throughput, the present invention can calculate the output interval time of the checkpoint based on the surplus time and the maximum throughput.

[0060] Optionally, step S102 may include:

[0061] The product of the surplus time and the maximum throughput is determined as the output interval time.

[0062] Specifically, after determining the surplus time and the maximum throughput, the present invention can multiply the surplus time and the maximum throughput, and the value obtained by the multiplication can be determined as the output interval time of the checkpoint.

[0063] S103 . According to the output interval duration, control the target node to output a checkpoint during the execution of the target job.

[0064] Specifically, the present invention can output and save the checkpoints at corresponding moments according to the determined output interval of the checkpoints during the process of the target node processing the target job.

[0065] It should be noted that when the target job is a batch job, the present invention can first determine the directed acyclic graph of the target job execution based on the log of the historical job, and determine each node in the directed acyclic graph as the target node in turn. Specifically, the present invention can prioritize the upstream node as the target node in the order from upstream to downstream, determine the output interval duration of the checkpoint in the process of the target node processing the target job, until the output interval duration of the checkpoint in the process of processing the target job of all nodes in the directed acyclic graph is determined.

[0066] Specifically, in the process of determining the directed acyclic graph of the target job execution, the present invention can obtain the input and output file lists by collecting the configuration files of all jobs in the cluster, establish an inverted index between files and jobs through the file directory, and then determine the directed acyclic graph based on the data obtained from the inverted index.

[0067] Specifically, if an obstacle occurs when the target node is processing the target job, fault recovery can be performed according to the currently saved checkpoint, so that the job processing can continue from the state corresponding to the checkpoint of the target job without restarting the processing of the target job, which can effectively improve the job processing efficiency and fault tolerance.

[0068] It should be noted that the present invention first estimates the longest recovery time that the target node can tolerate, and then regularly outputs the data that exceeds the recovery time in the form of checkpoints, thereby achieving effective control over the target job failure recovery time, thereby ensuring the output interval duration of the determined checkpoints, and effectively avoiding the job recovery time being too long and failing to meet the allowed response time requirements.

[0069] It should also be noted that when a node is processing a job, writing a checkpoint and recovering from a fault both consume certain resources. If checkpoints are frequently written, although the job recovery speed can be accelerated, a large amount of I / O resources will be consumed. It can be understood that the checkpoint output interval determined by the present invention can minimize the resource consumption required for writing a checkpoint and performing fault recovery.

[0070] The data processing method proposed in this embodiment can determine the surplus time that matches the allowed response time of the target node to the target job, determine the output interval time of the checkpoint based on the surplus time and the maximum throughput of the target node in the process of executing the target job, and control the target node to output the checkpoint in the process of executing the target job according to the output interval time. The present invention can effectively avoid the job recovery time being too long and failing to meet the allowed response time requirement, and can minimize the resource consumption required for writing the checkpoint and performing fault recovery.

[0071] and Figure 1The method shown corresponds to Figure 3 As shown, this embodiment proposes a first data processing device. The device may include: a first determination unit 101, a second determination unit 102 and a control unit 103; wherein:

[0072] The first determining unit 101 is used to determine a surplus time length that matches the allowed response time length of the target node to the target job;

[0073] The target node may be an electronic device used to process the target job, such as a mobile phone or a tablet computer.

[0074] The target job may be a batch job or a non-batch job, which is not limited in the present invention.

[0075] The response time may be the time required for the target node to complete the process of executing the target job from receiving the processing instruction of the target job to completing the process of executing the target job.

[0076] The allowed response time may be the allowed response time when the target node processes the target job.

[0077] Optionally, the allowed response time may be the maximum allowed response time, or may be a value smaller than the maximum allowed response time.

[0078] Optionally, in the second data processing device proposed in this embodiment, the first determining unit 101 may include: a third determining unit, a fourth determining unit and a fifth determining unit;

[0079] A third determining unit, used to determine the allowed response time of the target node to the target job;

[0080] A fourth determining unit is used to determine the execution time ranked at a preset percentile in the historical execution time data of the target node for the target job as the target execution time;

[0081] The fifth determining unit is used to determine the surplus time based on the allowed response time and the target execution time.

[0082] Specifically, the allowed response time can be set by the technician according to the target operation and actual situation, and the present invention does not limit this.

[0083] The historical execution time data may include the execution time required for the target node to process the target job in history.

[0084] Among them, the preset percentile can be set by technical personnel according to actual conditions, and the present invention is not limited to this.

[0085] Specifically, the present invention can arrange the execution times required for all historical target nodes to process target jobs in order from small to large, and then determine the execution times that are arranged in the order of the preset percentile of the total as the target execution time. For example, when the preset percentile is the 95th percentile, the present invention can arrange the execution times required for all historical target nodes to process target jobs, and then determine the execution times that are arranged in the order of the 95th percentile of the total as the target execution time.

[0086] Specifically, the present invention can calculate the surplus time according to the allowed response time and the target execution time.

[0087] Optionally, the fifth determination unit is used to determine the difference between the allowed response time and the target execution time as the surplus time.

[0088] Specifically, after obtaining the allowed response time and the target execution time, the present invention can subtract the allowed response time from the target execution time to obtain the surplus time.

[0089] A second determining unit 102 is used to determine the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job;

[0090] Specifically, the present invention can determine the maximum throughput of the target node when processing the target job based on the current idle resources and specific execution performance of the target node.

[0091] Specifically, after obtaining the surplus time and the maximum throughput, the present invention can calculate the output interval time of the checkpoint based on the surplus time and the maximum throughput.

[0092] Optionally, the second determining unit 102 is configured to determine the product of the surplus time and the maximum throughput as the output interval time.

[0093] Specifically, after determining the surplus time and the maximum throughput, the present invention can multiply the surplus time and the maximum throughput, and the value obtained by the multiplication can be determined as the output interval time of the checkpoint.

[0094] The control unit 103 is used to control the target node to output a checkpoint in the process of executing the target job according to the output interval duration.

[0095] Specifically, the present invention can output and save the checkpoints at corresponding moments according to the determined output interval of the checkpoints during the process of the target node processing the target job.

[0096] It should be noted that when the target job is a batch job, the present invention can first determine the directed acyclic graph of the target job execution based on the log of the historical job, and determine each node in the directed acyclic graph as the target node in turn. Specifically, the present invention can prioritize the upstream node as the target node in the order from upstream to downstream, determine the output interval duration of the checkpoint in the process of the target node processing the target job, until the output interval duration of the checkpoint in the process of processing the target job of all nodes in the directed acyclic graph is determined.

[0097] Specifically, in the process of determining the directed acyclic graph of the target job execution, the present invention can obtain the input and output file lists by collecting the configuration files of all jobs in the cluster, establish an inverted index between files and jobs through the file directory, and then determine the directed acyclic graph based on the data obtained from the inverted index.

[0098] Specifically, if an obstacle occurs when the target node is processing the target job, fault recovery can be performed according to the currently saved checkpoint, so that the job processing can continue from the state corresponding to the checkpoint of the target job without restarting the processing of the target job, which can effectively improve the job processing efficiency and fault tolerance.

[0099] It should be noted that the present invention first estimates the longest recovery time that the target node can tolerate, and then regularly outputs the data that exceeds the recovery time in the form of checkpoints, thereby achieving effective control over the target job failure recovery time, thereby ensuring the output interval duration of the determined checkpoints, and effectively avoiding the job recovery time being too long and failing to meet the allowed response time requirements.

[0100] It should also be noted that when a node is processing a job, writing a checkpoint and recovering from a fault both consume certain resources. If checkpoints are frequently written, although the job recovery speed can be accelerated, a large amount of I / O resources will be consumed. It can be understood that the checkpoint output interval determined by the present invention can minimize the resource consumption required for writing a checkpoint and performing fault recovery.

[0101] The data processing device proposed in this embodiment can determine the surplus time that matches the allowed response time of the target node to the target job, determine the output interval time of the checkpoint based on the surplus time and the maximum throughput of the target node in the process of executing the target job, and control the target node to output the checkpoint in the process of executing the target job according to the output interval time. The present invention can effectively avoid the job recovery time being too long and failing to meet the allowed response time requirement, and can minimize the resource consumption required for writing the checkpoint and performing fault recovery.

[0102] The above-mentioned data processing device may include a processor and a memory. The above-mentioned first determination unit, second determination unit and control unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0103] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and data processing is achieved by adjusting kernel parameters.

[0104] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned data processing method is implemented.

[0105] An embodiment of the present invention provides a processor, which is used to run a program, wherein the above data processing method is executed when the program is running.

[0106] like Figure 4 As shown, an embodiment of the present invention provides an electronic device 100, which includes at least one processor 200, at least one memory 300 connected to the processor 200, and a bus 400; wherein the processor 200 and the memory 300 communicate with each other through the bus 400; the processor 200 is used to call the program instructions in the memory 300 to execute the above data processing method. The electronic device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0107] The present invention also provides a computer program product, which, when executed on an electronic device, is suitable for executing a program for initializing the following method steps:

[0108] Determine a surplus time that matches the allowed response time of the target node to the target job;

[0109] Determine the checkpoint output interval based on the surplus time and the maximum throughput of the target node in the process of executing the target job;

[0110] According to the output interval, the target node is controlled to output checkpoints during the execution of the target job.

[0111] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] In a typical configuration, an electronic device includes one or more processors (CPU), a memory, and a bus. The electronic device may also include an input / output interface, a network interface, and the like.

[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0114] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0116] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0117] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the allowed response time of the target node to the target job; In the historical execution duration data of the target node for the target job, determining the execution duration ranked at a preset percentile as the target execution duration; Determining a surplus time based on the allowed response time and the target execution time; Determine the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job; According to the output interval duration, the target node is controlled to output a checkpoint during execution of the target job.

2. The data processing method according to claim 1, characterized in that: The determining the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job includes: The product of the surplus time and the maximum throughput is determined as the output interval time.

3. The data processing method according to claim 1, characterized in that: The determining the surplus time based on the allowed response time and the target execution time includes: The difference between the allowed response time and the target execution time is determined as the surplus time.

4. A data processing device, characterized in that: The device comprises: a first determining unit, a second determining unit and a control unit; wherein: The first determining unit is used to determine a surplus time that matches an allowed response time of the target node to the target job; The second determining unit is used to determine the output interval duration of the checkpoint based on the surplus duration and the maximum throughput of the target node in the process of executing the target job; The control unit is used to control the target node to output a checkpoint in the process of executing the target job according to the output interval duration; The first determining unit includes: a third determining unit, a fourth determining unit and a fifth determining unit; The third determining unit is used to determine the allowed response time of the target node to the target job; The fourth determining unit is configured to determine, in the historical execution duration data of the target node for the target job, an execution duration ranked at a preset percentile as a target execution duration; The fifth determining unit is used to determine the surplus time based on the allowed response time and the target execution time.

5. The data processing device according to claim 4, characterized in that: The second determining unit is used to determine the output interval duration by multiplying the surplus duration by the maximum throughput.

6. The data processing device according to claim 4, characterized in that: The fifth determining unit is used to determine the difference between the allowed response time and the target execution time as the surplus time.

7. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute the data processing method as described in any one of claims 1 to 3.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 3 is implemented.

Images