An optimization method for improving task stability and a terminal

Abstract

The application discloses an optimization method and a terminal for improving task stability, presetting a distributed system storage file, storing result data of executed corresponding subtasks in a local disk corresponding to a root node and the distributed system storage file in sequence when the corresponding subtasks are executed in each root node, reading the result data from each root node in sequence, judging whether reading abnormality occurs or not, and executing step S3 if the reading abnormality occurs, and reading the result data of the corresponding root node from the distributed system storage file. The application not only stores the result data in the local disk when the corresponding subtasks are executed in each root node, but also stores the result data in the distributed system storage file, preferentially reads the data in the local disk when the data is read, and then reads the data in the distributed system storage file when the reading abnormality occurs, so that the task time consumption caused by disk node failure and the like is effectively reduced, and the stability of task execution is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an optimization method for improving task stability and a terminal. BACKGROUND

[0002] Many current systems have mass data task execution functions. In a big data scenario, the system mainly improves the execution efficiency of distributed tasks by the mapperreduce mode, that is, the mr mode. The main principle is that, in the map task, the map task is split into various distributed nodes, each node reads data by itself, processes the data, and saves the processed result data in the node itself. Then, each reduce node reads data from the hard disk of the map sub-task node to perform subsequent processing. If a map task running on a node fails before transmitting the map intermediate result to the reduce task, the map task will be re-run on another node to build the map intermediate result again.

[0003] In this way, the result data stored by each sub-task in the map is only one copy in the local disk. When a node fails or the disk is abnormal, all tasks on the disk will be affected and cannot read data and continue the task. Re-running the task will greatly affect the task execution time. SUMMARY

[0004] The technical problem to be solved by the present application is to provide an optimization method for improving task stability and a terminal to reduce the task time consumption caused by disk node failure and other conditions and improve the stability of task execution.

[0005] To solve the above technical problems, the technical solution adopted by the present application is:

[0006] An optimization method for improving task stability, comprising the following steps:

[0007] S1, presetting a distributed system storage file, when each root node executes a corresponding sub-task, storing the result data of the execution in the local disk corresponding to the root node and the distributed system storage file in sequence;

[0008] S2, reading the result data from the local disk corresponding to each root node in sequence, and judging whether a reading exception occurs, if so, executing step S3;

[0009] S3, reading the result data of the corresponding root node from the distributed system storage file.

[0010] To solve the above technical problems, another technical solution adopted by the present application is:

[0011] An optimization terminal for improving task stability, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0012] S1, presetting a distributed system storage file, and storing result data in the distributed system storage file and a local disk corresponding to each root node in sequence when executing a corresponding subtask of each root node;

[0013] S2, reading the result data from the local disk corresponding to each root node in sequence, and judging whether a reading exception occurs, and if yes, executing step S3;

[0014] S3, reading the result data of the corresponding root node from the distributed system storage file.

[0015] The present application has the beneficial effect of providing an optimization method and terminal for improving task stability, presetting a distributed system storage file for a distributed task, so that when each root node executes a corresponding subtask, the result data is not only stored in a local disk, but also stored in a distributed system storage file, and when reading data, the data in the local disk is read first, and when a reading exception occurs, the data on the distributed system storage file is read, so that the local disk is used as the main part, and the distributed system storage file is used as the auxiliary part, effectively reducing the task time consumption caused by disk node failure and the like, and improving the stability of task execution. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a step schematic diagram of an optimization method for improving task stability of the present application.

[0017] Figure 2 It is a flowchart of an optimization method for improving task stability of the present application from a single subnode to read result data.

[0018] Figure 3 It is a system block diagram of an optimization terminal for improving task stability of the present application.

[0019] Label explanation:

[0020] 1. An optimization terminal for improving task stability; 2. a memory; 3. a processor. DETAILED DESCRIPTION

[0021] To explain the technical content, purposes and effects of the present application in detail, the following will be described in combination with the embodiments and the accompanying drawings.

[0022] Please refer to Figure 1 andFigure 2 An optimization method for improving task stability, comprising the following steps:

[0023] S1, presetting a distributed system storage file, and storing result data in the local disk corresponding to each root node and the distributed system storage file in sequence when executing the corresponding subtask of each root node;

[0024] S2, reading the result data from the local disk corresponding to each root node in sequence, and judging whether a reading abnormality occurs, if yes, executing step S3;

[0025] S3, reading the result data of the corresponding root node from the distributed system storage file.

[0026] From the above description, the beneficial effects of the present application are that, for a distributed task, a distributed system storage file is preset, so that when each root node executes the corresponding subtask, not only the result data is stored in the local disk, but also the result data is stored in the distributed system storage file. When reading data, the data in the local disk is read first, and then the data in the distributed system storage file is read when a reading abnormality occurs. The local disk is used as the main part, and the distributed system storage file is used as the auxiliary part. The task time consumption caused by disk node failure and the like is effectively reduced, and the stability of task execution is improved.

[0027] Further, the reading abnormality includes network failure or local disk failure.

[0028] From the above description, when judging the reading abnormality, not only the case that the network fails, the local disk fails and the like cannot be read is considered, but also the case that the reading rate is limited is considered. In this way, the storage function of the preset distributed system storage file is fully played, and the reading rate is improved as much as possible.

[0029] Further, the step S2 comprises:

[0030] setting a reading rate lower limit value;

[0031] judging whether the reading rate of reading the result data from the root node is equal to or lower than the reading rate lower limit value, if yes, executing the step S3.

[0032] From the above description, it can be seen that the read rate lower limit value is set, and when the read rate of the read process object to the local disk reaches or is lower than the read rate lower limit value, it indicates that the read process has seriously affected the efficiency of the overall task execution, and then the distributed system storage file is read for timely remediation to reduce the time consumption; by setting a reasonable read rate lower limit value, the distributed system storage file can also be avoided from being frequently called by the read process, the system pressure is reduced, and the overall distributed system cluster stability is avoided from being affected.

[0033] Further, the step S3 comprises:

[0034] S31, when the read exception occurs, record the read progress of the result data of the root node read from the local disk;

[0035] S32, from the read progress, continue to read the remaining data of the result data from the distributed system storage file.

[0036] From the above description, when the data of the distributed system storage file is read, the amount of read data on the local disk can be recorded first, i.e. the read progress, so as to avoid the need to read again on the distributed system storage file, reduce the amount of data to be read, and improve the read speed.

[0037] Further, the result data is stored in the local disk corresponding to the root node and the distributed system storage file in sequence, comprising:

[0038] The result data is written into the local cache of the root node, and then the result data is flushed to the local disk by means of asynchronous disk flushing;

[0039] After the flushing is completed, an asynchronous message containing the result data is sent to a preset asynchronous consumer, and the result data is written into the distributed system storage file by the asynchronous consumer.

[0040] From the above description, when the data is stored, the result data of the local disk is flushed to the distributed system storage file by asynchronous processing, which can effectively reduce the response time of system execution and improve the running efficiency.

[0041] Please refer to Figure 3 An optimization terminal 1 for improving task stability, comprising a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor, wherein the processor 3 implements the following steps when executing the computer program:

[0042] S1, a preset distributed system storage file is stored, when each root node executes a corresponding subtask, the executed result data is sequentially stored in the local disk corresponding to the root node and the distributed system storage file;

[0043] S2, the result data is sequentially read from the local disk corresponding to each root node, and whether a reading exception occurs is judged, if yes, step S3 is executed;

[0044] S3, the result data of the corresponding root node is read from the distributed system storage file.

[0045] From the above description, the beneficial effects of the present application are that, for a distributed task, a preset distributed system storage file is stored, so that when each root node executes a corresponding subtask, not only the result data is stored in the local disk, but also the result data is stored in the distributed system storage file, when reading data, the data of the local disk is read first, and when a reading exception occurs, the data on the distributed system storage file is read, the local disk is used as the main part, and the distributed system storage file is used as the auxiliary linkage, the task time consumption caused by disk node failure and the like is effectively reduced, and the stability of task execution is improved.

[0046] Further, the reading exception includes network failure or local disk failure.

[0047] From the above description, when judging the reading exception, not only the case that the network failure, the local disk failure and the like cannot be read is considered, but also the case that the reading rate is limited is considered, so that the storage function of the preset distributed system storage file is fully played, and the reading rate is improved as much as possible.

[0048] Further, the step S2 includes:

[0049] Setting a reading rate lower limit value;

[0050] Judging whether the reading rate of reading the result data from the root node is equal to or lower than the reading rate lower limit value, if yes, the step S3 is executed.

[0051] From the above description, the reading rate lower limit value is set, and when the reading rate of the reading process with the local disk as the object reaches or is lower than the reading rate lower limit value, it indicates that the reading process has seriously affected the efficiency of the whole task execution, and then the distributed system storage file is read, so that timely remediation is carried out, and the time consumption is reduced; by setting a reasonable reading rate lower limit value, the distributed system storage file can also be called frequently in the reading process, the system pressure is reduced, and the stability of the whole distributed system cluster is avoided.

[0052] Further, the step S3 includes:

[0053] S31, record a reading progress of the result data of the root node read from the local disk at present when the reading exception occurs;

[0054] S32, continue to read the remaining data of the result data from the distributed system storage file from the reading progress.

[0055] From the above description, when the data of the distributed system storage file is read, the amount of data read on the local disk can be recorded first, that is, the reading progress, so that the reading on the distributed system storage file needs to be re-performed, the amount of data to be read is reduced, and the reading speed is improved.

[0056] Further, the result data of the execution is stored in the local disk corresponding to the root node and the distributed system storage file in sequence, and the method comprises the following steps:

[0057] The result data is written into the local cache of the root node, and then the result data is refreshed to the local disk by means of asynchronous disk flushing;

[0058] After the refreshing is completed, an asynchronous message containing the result data is sent to a preset asynchronous consumer, and the asynchronous consumer writes the result data into the distributed system storage file.

[0059] From the above description, when the data is stored, the result data of the local disk is refreshed to the distributed system storage file by means of asynchronous processing, which can effectively reduce the response time of system execution and improve the running efficiency.

[0060] Please refer to Figure 1 and Figure 2 , the embodiment one of the present application is:

[0061] An optimization method for improving task stability, comprising the following steps:

[0062] S1, preset a distributed system storage file, when each root node executes a corresponding subtask, the result data of the execution is stored in the local disk corresponding to the root node and the distributed system storage file in sequence;

[0063] In the embodiment, the hdfs underlying the hadoop is used as an example for temporarily storing data in the distributed system. In the MR architecture, when the mapper finishes executing the task, each node still writes data to the local cache, but records the result data to the hdfs in an asynchronous manner. Specifically, the result data is written to the local cache of the root node, and then the result data is flushed to the local disk in an asynchronous manner. When each local disk flushing action is completed, an asynchronous message is sent, the message content is the content of the flushing, a special asynchronous consumer consumes the message, and writes the message to the hdfs file in the hadoop cluster. The process needs to ensure that the format and data stored in the local disk and the hdfs are consistent.

[0064] S2, read the result data from the local disk corresponding to each root node in sequence, and determine whether a reading exception occurs, if yes, execute step S3;

[0065] In the embodiment, when the reduce needs to read data from each mapper node, the data is preferentially read from the local disk where the mapper node is located. Specifically, if a network failure or a mapper disk node failure occurs, or the reading data rate is reduced to the lower limit of the reading rate, step S3 is executed.

[0066] S3, read the result data of the corresponding root node from the distributed system storage file.

[0067] In the process, step S3 includes:

[0068] S31, when a reading exception occurs, record the reading progress of the result data of the root node read from the local disk;

[0069] S32, continue to read the remaining data of the result data from the distributed system storage file from the reading progress.

[0070] That is, the reduce node takes the amount of data read from the mapper local disk as a calculation point, that is, the reading progress, and when reading data from the hdfs, only data after the calculation point is read, so that the amount of data read can be effectively reduced, and when the mapper node fails, the data can still be normally read, without affecting the overall task.

[0071] Please refer to Figure 3 The second embodiment of the present application is:

[0072] An optimization terminal 1 for improving task stability comprises a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3, and the processor 3 implements an optimization method for improving task stability as described in Embodiment I when executing the computer program.

[0073] In summary, the present application provides a method and terminal for improving task stability, for distributed tasks, a distributed system storage file is preset, so that when each root node executes the corresponding subtask, not only the result data is stored in the local disk, but also the result data is stored in the distributed system storage file, when reading data, the data in the local disk is read first, and when reading abnormity occurs, the data in the distributed system storage file is read, the local disk is used as the main part, and the distributed system storage file is used as the auxiliary part, effectively reducing the task time consumption caused by disk node failure, and improving the stability of task execution.

[0074] The above is only an embodiment of the present application, and does not limit the patent range of the present application, any equivalent transformation, direct or indirect application in related technical fields by using the content of the specification and drawings of the present application are also included in the patent protection range of the present application.

Claims

1. An optimization method for improving task stability, characterized by, The method comprises the following steps: S1, presetting a distributed system storage file, and storing result data of execution in a local disk corresponding to each root node and the distributed system storage file in sequence when each root node executes a corresponding subtask; S2, reading the result data from the local disk corresponding to each root node in sequence, and judging whether a reading exception occurs, and if yes, executing step S3; S3, reading the result data of the corresponding root node from the distributed system storage file; The step S3 comprises: S31, recording a reading progress of the result data of the root node read from the local disk when the reading exception occurs; S32, continuing to read remaining data of the result data from the distributed system storage file from the reading progress; The storing of the result data of execution in the local disk corresponding to the root node and the distributed system storage file in sequence comprises: writing the result data into a local cache of the root node, and then refreshing the result data to the local disk through an asynchronous disk brushing mode; after refreshing is completed, sending an asynchronous message containing the result data to a preset asynchronous consumer, and writing the result data into the distributed system storage file by the asynchronous consumer.

2. The optimization method of claim 1, wherein, The reading exception comprises a network failure or a local disk failure.

3. The optimization method of claim 2, wherein, The step S2 comprises: setting a reading rate lower limit value; judging whether a reading rate of reading the result data from the root node is equal to or lower than the reading rate lower limit value, and if yes, executing the step S3.

4. An optimization terminal for improving task stability, characterized by, The computer program product comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the following steps when executing the computer program: S1, presetting a distributed system storage file, and storing result data of execution in a local disk corresponding to each root node and the distributed system storage file in sequence when each root node executes a corresponding subtask; S2, reading the result data from the local disk corresponding to each root node in sequence, and judging whether a reading exception occurs, and if yes, executing step S3; S3, reading the result data of the corresponding root node from the distributed system storage file; The step S3 comprises: S31, recording a reading progress of the result data of the root node read from the local disk when the reading exception occurs; S32, continuing to read remaining data of the result data from the distributed system storage file from the reading progress; The storing of the result data of execution in the local disk corresponding to the root node and the distributed system storage file in sequence comprises: writing the result data into a local cache of the root node, and then refreshing the result data to the local disk through an asynchronous disk brushing mode; after refreshing is completed, sending an asynchronous message containing the result data to a preset asynchronous consumer, and writing the result data into the distributed system storage file by the asynchronous consumer.

5. The optimization terminal for improving task stability according to claim 4, characterized by, The reading exception includes a network failure or a local disk failure.

6. The optimization terminal for improving task stability according to claim 5, wherein The step S2 includes: Setting a reading rate lower limit value; Determining whether a reading rate of currently reading the result data from the root node is equal to or lower than the reading rate lower limit value, and if so, executing the step S3.

Images