A task execution method, device and equipment

Abstract

The embodiments of this specification provide a task execution method, device and equipment, which can be applied to the field of big data technology. The method includes: splitting the target task to obtain at least two execution instructions; the at least two execution instructions are used to constitute the execution process of the target task; setting a reading thread for each execution instruction; the reading thread is used to export the execution log of the corresponding execution instruction respectively; in the process of executing the target task, obtaining the execution log exported by each reading thread; based on the execution log, displaying the execution progress of the target task. The above method can grasp the execution status of each step and obtain a clearer perception of the process execution process. When a task execution error occurs, the key part information can be directly displayed through screening, which not only reduces the workload of the operation and maintenance personnel, but also makes it easier to implement breakpoint re-calling, error analysis and other tasks when an execution error occurs, which is conducive to improving the execution efficiency of the work.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of big data technology, and in particular to a task execution method, device and equipment. Background Art

[0002] With the development of computer technology, in more and more technical fields, tasks to be processed can be converted into programmed jobs, and then the corresponding jobs can be processed using pre-set computer program instructions. In order to effectively execute the jobs, it is often necessary to monitor the execution results of the jobs. At present, the execution results of the jobs are generally obtained by exporting the execution logs of the jobs and viewing the execution logs.

[0003] However, the above method of obtaining the execution result of a job can only view the overall execution effect of the job after the job is completed. A job is generally composed of multiple execution steps, so it is impossible to obtain the real-time execution process of the job by directly viewing the execution log of the job. In addition, in the case of a job execution error, directly analyzing the cause of the error from the execution log of the job not only has high requirements for the experience of the tester, but also increases the time required for error correction. Therefore, there is an urgent need for a technical solution that can grasp the execution progress of the task in detail and accurately. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a task execution method, device and equipment to solve the problem of how to grasp the execution progress of the task in a detailed and accurate manner.

[0005] To solve the above technical problems, an embodiment of the present specification provides a task execution method, including: splitting a target task to obtain at least two execution instructions; the at least two execution instructions are used to constitute the execution process of the target task; setting a reading thread for each execution instruction; the reading thread is used to export the execution logs of the corresponding execution instructions respectively; in the process of executing the target task, obtaining the execution logs exported by each reading thread; and displaying the execution progress of the target task based on the execution logs.

[0006] The embodiment of this specification also proposes a task execution device, including: a target task splitting module, used to split the target task into at least two execution instructions; the at least two execution instructions are used to constitute the execution process of the target task; a reading thread setting module, used to set a reading thread for each execution instruction; the reading thread is used to export the execution log of the corresponding execution instruction respectively; an execution log acquisition module, used to obtain the execution log exported by each reading thread during the execution of the target task; an execution progress display module, used to display the execution progress of the target task based on the execution log.

[0007] The embodiment of this specification also proposes a task execution device, including a memory and a processor; the memory is used to store computer program instructions; the processor is used to execute the computer program instructions to implement the following steps: splitting the target task into at least two execution instructions; the at least two execution instructions are used to constitute the execution process of the target task; setting a reading thread for each execution instruction; the reading thread is used to export the execution log of the corresponding execution instruction respectively; in the process of executing the target task, obtaining the execution log exported by each reading thread; and displaying the execution progress of the target task based on the execution log.

[0008] It can be seen from the technical solution provided by the above embodiments of this specification that after obtaining the target task, the embodiments of this specification can split the target task, and set the corresponding reading thread for the execution instruction obtained by the splitting, so that in the execution process of the target task, based on the execution order of the execution instruction in the target task, the execution log of each execution instruction can be obtained in turn by using the reading thread, and finally the execution progress of the target task can be displayed through the execution log. Through the above method, in the execution process of the target task, the execution status of each step can be grasped, and a clearer perception can be obtained for the execution process of the process. When the task has execution errors and the like, the information of the key parts can also be directly displayed by screening, which not only reduces the workload of the operation and maintenance personnel, but also makes it easier to realize breakpoint re-reference, error analysis and other tasks when the task execution error occurs, which is conducive to improving the execution efficiency of the work. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 A flowchart of a task execution method according to an embodiment of this specification;

[0011] Figure 2 This is a schematic diagram of obtaining an execution log according to an embodiment of this specification;

[0012] Figure 3 This is a schematic diagram of exporting an execution log according to an embodiment of this specification;

[0013] Figure 4 A schematic diagram of a breakpoint continuation method in an embodiment of this specification

[0014] Figure 5This is a schematic diagram of an execution log collection embodiment of this specification;

[0015] Figure 6 A schematic diagram of a task execution method flow in an embodiment of this specification;

[0016] Figure 7 This is a module diagram of a task execution device according to an embodiment of this specification;

[0017] Figure 8 This is a structural diagram of a task execution device according to an embodiment of this specification. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0019] In order to solve the above technical problems, a task execution method according to an embodiment of this specification is first introduced. The execution subject of the task execution method can be a task execution device, which includes but is not limited to a server, an industrial computer, a PC, etc. Figure 1 As shown, the task execution method may include the following specific implementation steps.

[0020] S110: Split the target task into at least two execution instructions; the at least two execution instructions are used to constitute the execution process of the target task.

[0021] The target task can be a task to be performed, for example, it can be a job based on a big data platform, or it can be other types of jobs. The target task generally has a long execution process. If the execution progress of the task is obtained by reading the task log corresponding to the target task, the specific steps cannot be directly obtained through the log, so that the execution process of the target task cannot be more carefully mastered. In addition, the execution of each specific step in the target task may not be described in the task log, which makes it inconvenient to obtain the execution status of the target task and to restart the breakpoint.

[0022] The target task may be composed of multiple execution steps. Specifically, these execution steps may correspond to each line of code in the target task, or may correspond to an inseparable minimum execution unit. These execution steps may be executed sequentially, i.e., the next step is executed after one execution step is processed, or they may be executed concurrently, i.e., multiple steps may be executed at the same time.

[0023] Therefore, the target task can be split to obtain at least two execution instructions. The execution instruction is used to represent the minimum division step in the execution of the target task, that is, it can be used to represent the execution steps introduced above. These execution instructions together constitute the execution process of the target task. After determining the execution instruction, if the execution results of each execution instruction in the target task can be obtained in real time during the execution process of the target task, the execution progress of the target task can be grasped in more detail.

[0024] In some implementations, the target task may be a .hql script. When the execution instructions are split, the target task may be split into at least two Hive SQL commands based on a split symbol in the target task. The split symbol has the effect of splitting the execution instructions in the target task. Specifically, the split symbol may be, for example, a semicolon to separate statements.

[0025] The corresponding Hive SQL data set can be obtained after splitting based on the segmentation symbol, and the data in the Hive SQL data set can be used as an execution instruction.

[0026] S120: Setting a reading thread for each execution instruction; the reading thread is used to export the execution log of the corresponding execution instruction respectively.

[0027] After obtaining the execution instruction in the target task, a reading thread can be set for the execution instruction. The reading thread is used to export the execution log of the corresponding execution instruction. The execution log can be used to record the execution status of the execution instruction, that is, to represent the specific information of the target task when it is executed, such as the start time, end time, execution result and other information. In the embodiment of this specification, when the reading thread corresponds to each execution instruction, the execution log read by each reading thread is the log of the specific execution process corresponding to each execution instruction. Therefore, the information in each execution log can be obtained respectively, so as to describe the execution status of each step in the target task respectively, so that the execution status of the target task can be grasped in more detail.

[0028] Specifically, the reading thread can locate the corresponding execution instruction by setting at least one parameter among the target task identifier, the execution instruction identifier, and the execution instruction code. The target task identifier can be used to identify the target task, for example, it can be the serial number, hash value and other information of the target task. The execution instruction identifier can be an identifier corresponding to the execution instruction, such as the hash value of the execution instruction, or the specific location of the execution instruction in the target task. The execution instruction code can be all or part of the actual code of the execution instruction. The location of the execution instruction can be determined by the above three identifiers, so that the corresponding execution log can be derived when the execution instruction is executed. In actual applications, other information can also be used for identification as needed, which is not limited to the above examples and will not be repeated here.

[0029] In some implementations, when the split execution instructions are a Hive SQL data set, a record of the Running status can be first inserted into the Oracle table corresponding to the Hive SQL data set to record information such as the task identifier, Hive SQL, Hive SQL sequence number, and start time, so that in subsequent steps, the SQL progress within the task can be displayed in real time on the front end based on the record.

[0030] After that, you can connect to the Hive database through JDBC to get a HiveStatement object. Before executing each Hive SQL statement, use the reading thread GetLogThread to export the above HiveStatement object and other information, such as task ID, Hive SQL, and Hive SQL sequence number, to obtain the execution log.

[0031] The above implementation can effectively obtain the information of each Hive SQL execution process by setting a corresponding reading thread for each Hive SQL, and can obtain the specific execution status of each Hive SQL and detailed parameter information during the execution process by parsing the execution log. It should be noted that the above implementation is only a specific example proposed in combination with the actual application environment. In actual applications, the implementation can be adjusted according to needs or other implementations can be adopted to adapt to specific scenarios, which will not be repeated here.

[0032] In the embodiment of the present specification, when the reading thread corresponds to each execution instruction, the specific information described for the execution instruction in the execution log can be obtained according to the execution log obtained by the reading thread, so that the execution status of each execution instruction in the target task can be obtained, and the implementation status of each specific process in the execution process of the target task can be grasped in detail.

[0033] S130: In the process of executing the target task, the execution logs exported by each reading thread are obtained.

[0034] In the case where each execution instruction is respectively provided with a corresponding reading thread, the execution log of the corresponding execution instruction can be obtained by using the set reading thread. The specific acquisition process can refer to the description in the above step S120, which will not be repeated here.

[0035] Combination Figure 2 , using a specific example to illustrate the process of reading the execution log, based on the implementation in step S120, after obtaining the HiveStatement object corresponding to each Hive SQL, the Statement.execute(String sql) method can be called to execute the Hive SQL in the Hive SQL data set in sequence, thereby realizing the execution of the target task.

[0036] Correspondingly, such as Figure 3 As shown, in the subsequent steps, the reading thread GetLogThread can call the HiveStatement.getQueryLog(boolean incremental, int fetchSize) method to obtain each execution instruction, that is, the execution log of the Hive SQL statement in real time and accurately, and output it to the job scheduling log, so that the execution progress of each Hive SQL in the job can be viewed in real time. In addition, the reading thread GetLogThread can also parse the execution log to obtain the HiveServer node IP, Hive QueryID, YarnApplicationID, error information, etc. from the execution log.

[0037] In some implementations, the specific step of obtaining the execution log may be to locate the candidate execution log based on the HiveServer node IP information, and then filter out the execution log corresponding to the Hive QueryID information from the candidate execution log.

[0038] S140: Displaying the execution progress of the target task based on the execution log.

[0039] The execution progress can be used to indicate the real-time execution progress of the current target task, for example, the currently executed execution instruction and the corresponding execution result, or it can be used to indicate the execution status of the target task itself. The execution status of the target task can be grasped through the execution progress, so that the execution details of the target task can be effectively collected to analyze the target task.

[0040] In some implementations, the execution progress of the target task may include the execution status of the target task, and the execution status includes an execution success status and an execution failure status.

[0041] The execution success state can be used to indicate that the target task is executed successfully, and all execution instructions therein can be executed smoothly. When the target task is in the execution success state, subsequent tasks can be executed without taking corresponding error correction operations for the target task.

[0042] The execution failure state is used to indicate that an error occurs during the execution of the target task, which may be an error in one or more execution instructions. In order to ensure the correct implementation of the task, it is often necessary to re-execute the target task to obtain a better execution effect.

[0043] In a specific example, determining the execution status can be to obtain the return result of Statement.execute(Stringsql) executing the execution instruction, that is, Hive SQL. If true is returned and no exception is thrown, it means that the Hive SQL is executed successfully, and its execution status is marked as a successful execution status. If true is not returned or an exception is directly thrown, the record status corresponding to the execution instruction in the Oracle table is changed to failed, that is, it is marked as an execution failure status.

[0044] However, in the case of execution failure, if the target task is executed from the beginning, it may consume a lot of time. In addition, if the cause of the error cannot be determined, directly re-executing the target task may also cause the target task to fail again.

[0045] Therefore, in some implementations, if the execution status of the target task is an execution failure status, an error log can be obtained from the execution log. The error log can be used to indicate a log output when an execution instruction fails. Specifically, the log information in the error log can be used to indicate whether the corresponding execution instruction fails, and the error log can be obtained by filtering it out by analyzing the error log.

[0046] After the error log is obtained, the execution instruction corresponding to the error log can be used as the target execution instruction. When the execution is interrupted and re-raised, the target task can be directly executed from the target execution instruction to reduce the time of re-executing the task.

[0047] In practical applications, the cause of the target task execution error can also be analyzed based on the error information in the error log, so that the target task or the target execution instruction can be debugged based on the above information to reduce the time consumed in finding the cause of the error and improve the success rate of re-execution of the target task.

[0048] like Figure 4 As shown, when there is a command execution failure, you can first determine whether a breakpoint resumption is required. If necessary, the specific method of interrupting and resuming based on the target execution instruction in the task can be to first scan the corresponding target execution instruction in the Oracle table, obtain the corresponding execution statement, such as a Hive SQL statement, start running the job based on the identifier corresponding to the target execution instruction, without repeating the execution instruction that was successfully executed, and update the records in the Oracle table in real time based on the execution process.

[0049] Based on the above implementation, when a task execution error occurs, a breakpoint re-call method is used to perform error analysis and task re-execution, thereby avoiding the process of re-executing the task and reducing the time and resource consumption of task execution.

[0050] Of course, in actual applications, when an error occurs during the execution process, instead of using the breakpoint resumption method, the entire job can be re-proposed to re-run all execution instructions in the task in sequence, and all records in the data table can be updated accordingly.

[0051] In some embodiments, when determining the error log, multiple error logs may be found. The execution instructions corresponding to these error logs may be caused by multiple errors respectively, but it is more likely that after an execution instruction fails, other execution instructions implemented based on the execution instruction also fail, thereby generating multiple error logs. In this case, in order to reduce the time consumed by error correction, the log output time corresponding to these error logs can be obtained, and then the target error log can be filtered based on the log output time. The log output time corresponding to the target error log can be the earliest time, that is, the target error log is the first error log to appear.

[0052] After the target error log is determined, the execution instruction corresponding to the target error log may be used as the target execution instruction, and the target task may be re-executed from the target execution instruction.

[0053] When there are multiple error logs, by analyzing the time corresponding to these error logs, the execution instruction in which the error occurs earliest can be effectively determined, so that the interrupt and re-call operation can be performed directly from the execution instruction, saving the time consumed in error correction.

[0054] In some implementations, after the execution progress is obtained, the execution progress can be displayed in a front-end interface. For example, when the method is applied to a big data operation scenario, the front-end interface can be a terminal device interface of an operation and maintenance personnel; when the method is applied to a job test scenario, the front-end interface can be a terminal device interface of a tester. In actual applications, the execution progress can be displayed based on demand, and there is no limitation to this.

[0055] In some implementations, the execution progress can be displayed based on the execution logs, or the execution logs can be aggregated and the aggregated execution logs can be displayed as the execution progress. The purpose of the aggregation is to analyze the execution logs corresponding to each execution instruction, group the related execution logs into the same set, and display the corresponding logs based on the corresponding set.

[0056] Combination Figure 5 , using a specific example of the execution process to illustrate, you can first call the cluster log download rest interface and download the execution log corresponding to the HiveServer node where each Hive SQL is located. After that, according to the Hive QueryID in the Oracle table, you can filter out the logs related to the QueryID in the execution log, and save them as filtered HiveServer logs. After that, connect to the corresponding cluster, obtain the corresponding execution instructions on HDFS according to the ApplicationID, that is, the Yarn task logs run by Hive SQL, and download the job scheduling logs, filtered HiveServer logs, and Yarn task logs after aggregation, and display them as the execution progress.

[0057] By aggregating the execution logs before displaying them, it is possible to avoid viewing only a single execution instruction or a single execution task when displaying the execution progress, thereby reducing the difficulty of analyzing the logs and enabling more efficient task processing.

[0058] Combine the following Figure 6 , the above process is summarized using a specific scenario example. In this scenario example, the .hql script is first split to obtain a Hive SQL data set, and the Hive SQL data set is executed. The specific execution process includes a running module and a log module. The running module is responsible for running each Hive SQL in the Hive SQL data set in sequence, and the log module is responsible for outputting the execution log corresponding to each Hive SQL. Based on the execution log, in the event of a job execution failure, the job can be interrupted and re-proposed based on the information contained in the execution log to improve the efficiency of job execution.

[0059] Based on the introduction of the above embodiments and scenario examples, it can be seen that after the method obtains the target task, the target task can be split, and the corresponding reading thread can be set for the execution instruction obtained by the splitting, so that in the execution process of the target task, based on the execution order of the execution instruction in the target task, the execution log of each execution instruction can be obtained in sequence by using the reading thread, and finally the execution progress of the target task can be displayed through the execution log. Through the above method, in the execution process of the target task, the execution status of each step can be grasped, and a clearer perception can be obtained for the execution process of the process. When the task has execution errors and the like, the information of the key parts can also be directly displayed by screening, which not only reduces the workload of the operation and maintenance personnel, but also makes it easier to realize breakpoint re-reference, error analysis and other tasks when the task execution error occurs, which is conducive to improving the execution efficiency of the work.

[0060] based on Figure 1 The corresponding task execution method introduces a task execution device in the embodiment of this specification. The task execution device can be set in the task execution device. Figure 7 As shown, the task execution device may include the following modules.

[0061] The target task splitting module 710 is used to split the target task into at least two execution instructions; the at least two execution instructions are used to constitute the execution process of the target task.

[0062] The reading thread setting module 720 is used to set a reading thread for each execution instruction; the reading thread is used to export the execution log of the corresponding execution instruction respectively.

[0063] The execution log acquisition module 730 is used to acquire the execution logs exported by each reading thread during the execution of the target task.

[0064] The execution progress display module 740 is used to display the execution progress of the target task based on the execution log.

[0065] based on Figure 1 Corresponding to the task execution method, this specification embodiment provides a task execution device. Figure 8 As shown, the task execution device may include a memory and a processor.

[0066] In this embodiment, the memory may be implemented in any appropriate manner. For example, the memory may be a read-only memory, a mechanical hard disk, a solid-state hard disk, or a USB flash drive, etc. The memory may be used to store computer program instructions.

[0067] In this embodiment, the processor can be implemented in any appropriate manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. The processor can execute the computer program instructions to implement the following steps: split the target task to obtain at least two execution instructions; the at least two execution instructions are used to constitute the execution process of the target task; a reading thread is set for each execution instruction; the reading thread is used to export the execution log of the corresponding execution instruction respectively; in the process of executing the target task, the execution log exported by each reading thread is obtained; based on the execution log, the execution progress of the target task is displayed.

[0068] It should be noted that the task execution methods, devices and equipment described in the embodiments of this specification can be applied to the field of big data technology, and can also be applied to other technical fields without limitation.

[0069] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0070] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0071] It can be known from the above description of the implementation mode that the technicians in this field can clearly understand that the present specification can be implemented by means of software plus the necessary first hardware platform. Based on such an understanding, the technical solution of the present specification can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0072] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0073] This specification can be used in many first or special computer system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0074] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0075] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A task execution method, characterized in that: include: Split the target task into at least two execution instructions; The at least two execution instructions are used to constitute the execution process of the target task; A reading thread is set for each execution instruction; the reading thread is used to export the execution log of the node where the corresponding execution instruction is located, and is also used to parse the execution log of the corresponding execution instruction to obtain log information; wherein the execution instruction includes a Hive SQL command, the node includes a HiveServer node, and the log information includes at least one of obtaining HiveServer node IP information, Hive QueryID information, Yarn ApplicationID information, and error information; the execution log is used to record the execution status of the execution instruction; In the process of executing the target task, the execution logs exported by each reading thread are obtained; wherein the execution logs exported by each reading thread are obtained, including: locating candidate execution logs based on HiveServer node IP information; filtering out execution logs corresponding to Hive QueryID information from the candidate execution logs, and saving them as filtered HiveServer logs; The execution progress of the target task is displayed based on the execution log; wherein, the execution progress of the target task is displayed based on the execution log, including: obtaining the Yarn task log of Hive SQL run according to Yarn ApplicationID information, and downloading the job scheduling log, filtered HiveServer log and Yarn task log after aggregation, and displaying them as the execution progress.

2. The method according to claim 1, characterized in that The target task includes a .hql script; the target task is split to obtain at least two execution instructions, including: The target task is split into at least two Hive SQL commands based on a split symbol in the target task.

3. The method according to claim 1, characterized in that The displaying the execution progress of the target task based on the execution log further includes: Determining the execution progress of the target task according to the execution log; The execution progress is displayed.

4. The method according to claim 1, characterized in that The reading thread locates the corresponding execution instruction by setting at least one parameter among the target task identifier, the execution instruction identifier, and the execution instruction code.

5. The method according to claim 1, characterized in that The displaying of the execution progress of the target task based on the execution log further includes: The execution status of the target task is determined based on the execution log corresponding to each execution instruction; the execution status includes an execution success status and an execution failure status.

6. The method according to claim 5, characterized in that After displaying the execution progress of the target task based on the execution log, the method further includes: When the target task is in an execution failure state, an error log is obtained from the execution log; the error log represents a log output when an execution instruction fails; The target execution instruction is located based on the error log to re-execute the target task from the target execution instruction.

7. The method according to claim 6, characterized in that The locating the target execution instruction based on the error log to re-execute the target task from the target execution instruction includes: In the case where there are at least two error logs, filtering the target error log based on the log output time corresponding to the error logs; Determine the execution instruction corresponding to the target error log as the target execution instruction, so as to re-execute the target task from the target execution instruction.

8. A task execution device, characterized in that: include: A target task splitting module is used to split the target task into at least two execution instructions; The at least two execution instructions are used to constitute the execution process of the target task; A reading thread setting module is used to set a reading thread for each execution instruction; the reading thread is used to export the execution log of the node where the corresponding execution instruction is located, and is also used to parse the execution log of the corresponding execution instruction to obtain log information; wherein, the execution instruction includes a Hive SQL command, the node includes a HiveServer node, and the log information includes obtaining at least one of HiveServer node IP information, Hive QueryID information, Yarn ApplicationID information, and error information; the execution log is used to record the execution status of the execution instruction; An execution log acquisition module is used to acquire the execution logs derived by each reading thread during the execution of the target task; wherein acquiring the execution logs derived by each reading thread includes: locating candidate execution logs based on HiveServer node IP information; filtering out the execution log corresponding to Hive QueryID information from the candidate execution logs, and storing them as filtered HiveServer logs; An execution progress display module is used to display the execution progress of the target task based on the execution log; wherein, displaying the execution progress of the target task based on the execution log includes: obtaining the Yarn task log run by Hive SQL according to the Yarn ApplicationID information, and collecting and downloading the job scheduling log, the filtered HiveServer log and the Yarn task log to display the execution progress.

9. A task execution device comprising a memory and a processor; The memory is used to store computer program instructions; The processor is used to execute the computer program instructions to implement the following steps: split the target task to obtain at least two execution instructions; the at least two execution instructions are used to form the execution process of the target task; a reading thread is set for each execution instruction; the reading thread is used to respectively export the execution log of the node where the corresponding execution instruction is located, and is also used to parse the execution log of the corresponding execution instruction to obtain log information; wherein, The execution instruction includes a HiveSQL command, the node includes a HiveServer node, and the log information includes at least one of obtaining HiveServer node IP information, Hive QueryID information, Yarn ApplicationID information, and error information; the execution log is used to record the execution status of the execution instruction; In the process of executing the target task, the execution logs exported by each reading thread are obtained; wherein, the execution logs exported by each reading thread are obtained, including: locating candidate execution logs based on HiveServer node IP information; filtering out execution logs corresponding to Hive QueryID information from the candidate execution logs, and saving them as filtered HiveServer logs; displaying the execution progress of the target task based on the execution logs; wherein, displaying the execution progress of the target task based on the execution logs includes: obtaining the Yarn task log run by HiveSQL according to Yarn ApplicationID information, and downloading the job scheduling logs, the filtered HiveServer logs and the Yarn task logs after aggregation, and displaying them as the execution progress.

Images