Task log analysis method, system, computer equipment and storage medium

Abstract

This application discloses a task log analysis method, system, computer device, and storage medium. The method includes: an input step: inputting any task into the Agent of the NodeManager node; a collection step: the Agent collects container information in real time and sends it to Kafka; a processing step: Spark Streaming processes the information in Kafka to obtain information processing results; and a storage step: saving the information processing results in MySQL. This invention reduces manual labor, automatically analyzes logs related to distributed computing tasks, and displays the results.

Description

Technical Field

[0001] The present invention belongs to the field of distributed computing task log analysis methods, and specifically relates to a task log analysis method, system, computer equipment and storage medium. Background Art

[0002] With the advancement of technology, human society has entered the era of big data. Big data refers to data that cannot be processed within a reasonable timeframe using traditional methods. To address this problem, numerous distributed computing frameworks, such as MapReduce, Spark, and Flink, have emerged. However, while these distributed computing frameworks offer advantages in computing throughput and low latency, they also present numerous challenges. For example, in daily operations and maintenance, common problems often arise: task execution failures, slow task computations, and cluster crashes caused by abnormal tasks. Without comprehensive information about the cluster and tasks, it can be difficult to quickly pinpoint the source of task problems.

[0003] Currently, task log analysis is performed manually. When a task ends, YARN's built-in task log aggregation function will upload the logs generated by each task to a designated directory on HDFS. Operations and maintenance personnel can download task logs locally for viewing and analysis. However, this can lead to the following problems: directly viewing task logs is not very user-friendly; large task log files will take more time to locate the problem logs; locating task problems sometimes requires combining the machine's hardware logs, but the machine's hardware logs are not collected, and for cluster security reasons, not everyone can log in to the cluster node; machine hardware failures can also cause the loss of logs on that machine. Therefore, a distributed task log analysis system has been designed to help developers and operations and maintenance personnel quickly locate task problems.

[0004] Poor readability. If a task takes a long time to run or has many subtasks, the log file generated by the task will be very large, sometimes even reaching GB levels. In addition, the log will contain a lot of unimportant exception information. Therefore, it is very difficult and time-consuming for operation and maintenance personnel to locate the cause of the task problem in such a large amount of log information.

[0005] Inefficiency. For cluster security reasons, cluster login permissions are restricted, so only cluster administrators can log in to cluster nodes. This prevents developers from observing the running status of a container in real time. To obtain the container status, developers must contact the cluster administrator, which is inefficient.

[0006] Information is incomplete. When troubleshooting task problems, you often need to combine it with hardware information about the node where the container is located, but this information is not recorded in the container log.

[0007] Data is easily lost. Since the time granularity of aggregated logs is relatively coarse, if the node where the container is located fails before the task log aggregation is completed, the container log cannot be aggregated to HDFS, resulting in the loss of this part of the task log. Summary of the Invention

[0008] The embodiments of the present application provide a task log analysis method, system, and computer storage device to at least solve the problem of subjective factors in related technologies.

[0009] The present invention provides a task log analysis method, which includes:

[0010] Input step: Input any task to the Agent of the NodeManager node;

[0011] Collection step: The Agent collects Container information in real time and sends it to Kafka;

[0012] Processing step: Spark Streaming processes the information in the Kafka to obtain an information processing result.

[0013] Saving step: Saving the information processing results in MySQL.

[0014] In the above-mentioned distributed computing task log analysis method, the container information includes:

[0015] Container operation logs, Container CPU usage, Container memory usage, Container JVM information, current node load information, network card traffic information, and disk information.

[0016] In the above-mentioned distributed computing task log analysis method, the processing steps include:

[0017] The Spark Streaming consumes the information in the Kafka in real time and performs a comprehensive analysis of the task in combination with multiple logs.

[0018] The above-mentioned distributed computing task log analysis method further includes a query step:

[0019] The processing results stored in the MySQL can be queried and displayed.

[0020] The present invention also includes a distributed computing task log analysis system, which includes:

[0021] An input module, which inputs any task to the Agent of the NodeManager node;

[0022] A collection module collects container information in real time and sends it to Kafka;

[0023] The processing module processes the information in the Kafka to obtain an information processing result.

[0024] A saving module saves the information processing result to MySQL.

[0025] In the above-mentioned distributed computing task log analysis system, the information of the Container includes:

[0026] Container operation logs, Container CPU usage, Container memory usage, Container JVM information, current node load information, network card traffic information, and disk information.

[0027] In the above-mentioned distributed computing task log analysis system, the processing module consumes the information in Kafka in real time and performs a comprehensive analysis of the task in combination with multiple logs.

[0028] The above-mentioned distributed computing task log analysis system further includes a query module, which queries the processing results stored in the MySQL.

[0029] The present invention also includes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the distributed computing task log analysis method when executing the computer program.

[0030] The present invention also includes a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the distributed computing task log analysis method as described above is implemented.

[0031] The beneficial effects of the present invention are:

[0032] 1. Reduces manual labor, automatically analyzes logs related to distributed computing tasks, and displays the results.

[0033] 2. Efficient and real-time, it can perform real-time analysis on the running status of distributed computing tasks in the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0035] In the attached figure:

[0036] Figure 1 It is a flow chart of the distributed computing task log analysis method;

[0037] Figure 2 This is a flowchart of the specific steps of the distributed computing task log analysis method;

[0038] Figure 3 It is a structural diagram of the distributed computing task log analysis system of the present invention;

[0039] Figure 4 is a framework diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0041] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0042] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0043] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0044] The present invention is described in detail below with reference to the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in this field based on these embodiments are all within the scope of protection of the present invention.

[0045] Before elaborating on various embodiments of the present invention, the core inventive concept of the present invention is summarized and elaborated in detail through the following several embodiments.

[0046] Before elaborating on various embodiments of the present invention, the core inventive concept of the present invention is summarized and elaborated in detail through the following several embodiments.

[0047] Please refer to Figure 1 , Figure 1 This is a flow chart of the distributed computing task log analysis method. Figure 1 As shown, the distributed computing task log analysis method of the present invention includes:

[0048] Input step S1: Input any task to the Agent of the NodeManager node;

[0049] Collection step S2: The Agent collects the container information in real time and sends it to Kafka;

[0050] Processing step S3: Spark Streaming processes the information in the Kafka to obtain an information processing result;

[0051] Saving step S4: Saving the information processing result in MySQL.

[0052] The container information includes:

[0053] Container operation logs, Container CPU usage, Container memory usage, Container JVM information, current node load information, network card traffic information, and disk information.

[0054] The processing steps include:

[0055] The Spark Streaming consumes the information in the Kafka in real time and performs a comprehensive analysis of the task in combination with multiple logs.

[0056] It also includes the query step:

[0057] The processing results stored in the MySQL can be queried and displayed.

[0058] The distributed computing task log analysis method of the present invention is described in detail below with reference to examples.

[0059] Example 1:

[0060] This example discloses a specific implementation method of a statistics-based distributed computing task log analysis method (hereinafter referred to as the "method").

[0061] The present invention solves the problem of data loss and incomplete information. An Agent program will be deployed on each NodeManager node, which is responsible for collecting the running logs of all containers on the node, the container running status (including CPU usage, memory usage), and the machine hardware status (including machine load, network card load, disk status, etc.) in real time.

[0062] The present invention also solves the problems of readability and low efficiency by filtering out useless logs in the task log, displaying only the key information that may affect the task operation, and converting the operation log into a visual chart.

[0063] NodeManager node: responsible for computing node resource management, managing the container lifecycle, monitoring the CPU and memory usage of each container, and monitoring the health status of computing nodes.

[0064] Container: Distributed computing tasks are split into multiple subtasks. These subtasks are run in the container on the NodeManager node, and the container logs are written to the local disk.

[0065] Agent: Responsible for collecting relevant log information in real time for subsequent analysis and calculation.

[0066] Kafka: Used to temporarily store data produced by the Agent and provide it to the Spark Streaming program for consumption and calculation.

[0067] Spark Streaming: A streaming computing framework that analyzes the data sent by the Agent to Kafka and saves the analysis results to the database.

[0068] MySQL: used to store data generated by Spark Streaming analysis

[0069] The specific implementation steps of the present invention are as follows Figure 2 As shown:

[0070] Each NodeManager node deploys an agent for real-time collection of task log information and machine hardware information.

[0071] The Agent collects the Container's operation logs, the Container's CPU usage, the Container's memory usage, the Container's JVM information, the current node's load information, network card traffic information, disk information, etc. in real time and sends them to Kafka.

[0072] Spark Streaming consumes information from Kafka in real time, performs comprehensive analysis on tasks based on multiple logs, and then saves the analysis results to MySQL.

[0073] Provide visual query and display.

[0074] Example 2:

[0075] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of the distributed computing task log analysis system of the present invention. Figure 3 A distributed computing task log analysis system of the present invention includes:

[0076] An input module, which inputs any task to the Agent of the NodeManager node;

[0077] A collection module collects container information in real time and sends it to Kafka;

[0078] The processing module processes the information in the Kafka to obtain an information processing result.

[0079] A saving module saves the information processing result to MySQL.

[0080] The container information includes:

[0081] Container operation logs, Container CPU usage, Container memory usage, Container JVM information, current node load information, network card traffic information, and disk information.

[0082] The processing module consumes the information in Kafka in real time and performs a comprehensive analysis of the task in combination with multiple logs.

[0083] It also includes a query module, which queries the processing results stored in MySQL.

[0084] Example 3:

[0085] Combine Figure 4 As shown, this embodiment discloses a specific implementation of a computer device. The computer device may include a processor 81 and a memory 82 storing computer program instructions.

[0086] Specifically, the processor 81 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0087] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0088] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0089] The processor 81 implements any one of the distributed computing task log analysis methods in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0090] In some embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 4 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0091] The communication interface 83 is used to implement communication between the various modules, devices, units, and / or devices in the embodiments of the present application. The communication interface 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0092] The bus 80 includes hardware, software, or both, and couples components of the computer device to each other. The bus 80 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0093] The computer device can be based on distributed computing task log analysis to achieve integration Figure 1 Described method.

[0094] In addition, in conjunction with the distributed computing task log analysis method in the above embodiment, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the distributed computing task log analysis methods in the above embodiment is implemented.

[0095] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] To sum up, based on the beneficial effects of the present invention, this patent provides a distributed computing task log analysis method, which reduces manual labor, can automatically analyze distributed computing task related logs, and display the results; it is efficient and real-time, and can perform real-time analysis of the running status of distributed computing tasks in the cluster.

[0097] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A distributed computing task log analysis method, characterized in that: include: Input step: Input any task to the Agent of the NodeManager node. Each NodeManager node deploys an Agent for real-time collection of task log information and machine hardware information; Collection step: The Agent collects Container information in real time and sends it to Kafka; Processing step: Spark Streaming processes the information in the Kafka to obtain information processing results; Saving step: saving the information processing results to MySQL; The distributed computing task is split into multiple subtasks, and the subtasks are run in the container on the NodeManager node; The container information includes: Container operation logs, Container CPU usage, Container memory usage, Container JVM information, current node load information, network card traffic information, and disk information; The processing steps include: The Spark Streaming consumes the information in the Kafka in real time and performs a comprehensive analysis of the task in combination with multiple logs.

2. The distributed computing task log analysis method according to claim 1, wherein: Also includes the query step: The processing results stored in the MySQL can be queried and displayed.

3. A distributed computing task log analysis system, characterized in that: include: An input module that inputs any task to the Agent of the NodeManager node. Each NodeManager node deploys an Agent for collecting task log information and machine hardware information in real time. A collection module collects Container information in real time and sends it to Kafka, wherein the distributed computing task is split into multiple subtasks, and the subtasks are run in the Container on the NodeManager node; A processing module processes the information in the Kafka to obtain an information processing result; A saving module, wherein the saving module saves the information processing result into MySQL; The container information includes: Container operation logs, Container CPU usage, Container memory usage, Container JVM information, current node load information, network card traffic information, and disk information; The processing module consumes the information in Kafka in real time and performs a comprehensive analysis of the task in combination with multiple logs.

4. The distributed computing task log analysis system according to claim 3, wherein: It also includes a query module, which queries the processing results stored in the MySQL.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the distributed computing task log analysis method according to any one of claims 1 to 2 is implemented.

6. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distributed computing task log analysis method according to any one of claims 1 to 2 is implemented.

Images