Yarn resource scheduling performance analysis method and device
By acquiring and analyzing log data in YARN and using machine learning models to identify YARN resource scheduling bottlenecks, the problem of low manual analysis efficiency is solved and automated optimization strategy provision is achieved.
Patent Information
- Application Number
- CN202410469227.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-18
- Publication Date
- 2025-10-24
AI Technical Summary
In the existing technology, the method of manually analyzing the performance of YARN resource scheduling is difficult to accurately determine the bottleneck, and the analysis efficiency is low.
By obtaining the log data of the application manager and container in YARN, extracting the container's identification number, running status and state switching time, and using the resource scheduling optimization model trained by the machine learning model, the diagnostic data is compared with the preset conditions to automatically provide resource scheduling optimization strategy information.
Accurately identify bottlenecks in YARN resource scheduling performance, automatically provide optimization strategies, and improve analysis efficiency.
Smart Images
Figure CN120832207A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of big data, and particularly relates to a Yarn resource scheduling performance analysis method and device. BACKGROUND
[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the present application recited in the claims. The description herein does not constitute admission of prior art.
[0003] Yarn (Yet Another Resource Negotiator) is a key component in the Hadoop ecosystem, which is used for resource management and task scheduling. In the running of complex distributed computing tasks, due to the limitation of resource allocation, the task running time is often too long, and technical personnel need to manually analyze the bottleneck affecting the Yarn resource scheduling performance and formulate resource scheduling optimization strategies based on the experience of technical personnel. The manual analysis method of Yarn resource scheduling performance is difficult to accurately determine the bottleneck of Yarn resource scheduling performance, and the analysis efficiency is low. SUMMARY
[0004] The present application provides a Yarn resource scheduling performance analysis method, which can accurately determine the bottleneck affecting the Yarn resource scheduling performance, automatically provide corresponding resource scheduling optimization strategy information for the bottleneck of Yarn resource scheduling performance, improve the analysis efficiency of Yarn resource scheduling performance, and comprises the following steps:
[0005] Obtaining log data of an application manager, log data of a plurality of containers, and log data of an application in Yarn, wherein the application manager is used to schedule resources of the plurality of containers required when the application processes task data;
[0006] Extracting an identification number of each container, a plurality of running states of each container, and switching time of the plurality of running states from the log data of the application manager and the log data of the plurality of containers; and determining a task data processing quantity according to the log data of the application;
[0007] Determining diagnosis data of a plurality of preset diagnosis items according to the identification number of each container, the plurality of running states of each container, the switching time of the plurality of running states, and the task data processing quantity, wherein the plurality of preset diagnosis items comprise a container usage quantity, a consumption time length of each container in different running states, and a task data processing quantity in a preset time length;
[0008] The diagnosis data of each preset diagnosis item is compared with the corresponding preset condition, the preset diagnosis item that does not meet the preset condition is input into a resource scheduling optimization model, and resource scheduling optimization strategy information is obtained;The preset condition is a threshold condition set in advance according to different preset diagnosis items;The resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information;The resource scheduling optimization strategy information includes resource optimization allocation scheme information.
[0009] The embodiment of the application provides a Yarn resource scheduling performance analysis device, which can accurately determine the bottleneck affecting the Yarn resource scheduling performance, automatically provide corresponding resource scheduling optimization strategy information for the bottleneck of the Yarn resource scheduling performance, and improve the analysis efficiency of the Yarn resource scheduling performance.
[0010] The data acquisition module is configured to acquire log data of an application manager, log data of a plurality of containers, and log data of an application in the Yarn, wherein the application manager is configured to schedule resources of the plurality of containers required when the application processes task data.
[0011] The log extraction module is configured to extract an identification number of each container, a plurality of running states of each container, and switching time of the plurality of running states from the log data of the application manager and the log data of the plurality of containers, and determine a task data processing quantity according to the log data of the application.
[0012] The diagnosis data determination module is configured to determine diagnosis data of a plurality of preset diagnosis items according to the identification number of each container, the plurality of running states of each container, the switching time of the plurality of running states, and the task data processing quantity, wherein the plurality of preset diagnosis items include a container usage quantity, a consumption time length of each container in different running states, and a task data processing quantity in a preset time length.
[0013] The diagnosis data comparison module is configured to compare the diagnosis data of each preset diagnosis item with the corresponding preset condition, input the preset diagnosis item that does not meet the preset condition into a resource scheduling optimization model, and obtain resource scheduling optimization strategy information, wherein the preset condition is a threshold condition set in advance according to different preset diagnosis items, the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information, and the resource scheduling optimization strategy information includes resource optimization allocation scheme information.
[0014] The embodiment of the application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the Yarn resource scheduling performance analysis method when executing the computer program.
[0015] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the Yarn resource scheduling performance analysis method.
[0016] The embodiment of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the Yarn resource scheduling performance analysis method.
[0017] The Yarn resource scheduling performance analysis method and device can solve the problem that the method of manually analyzing the Yarn resource scheduling performance cannot accurately determine the bottleneck of the Yarn resource scheduling performance and the analysis efficiency is low. The application obtains the log data of an application manager, the log data of a plurality of containers and the log data of an application in the Yarn. The application manager is used for scheduling resources of the plurality of containers required when the application processes task data. The identification number of each container, a plurality of running states of each container and switching time of the plurality of running states are extracted from the log data of the application manager and the log data of the plurality of containers. The number of task data processed is determined according to the log data of the application. The diagnosis data of a plurality of preset diagnosis items is determined according to the identification number of each container, the plurality of running states of each container, the switching time of the plurality of running states and the number of task data processed. The plurality of preset diagnosis items include the number of containers used, the consumption time length of each container in different running states and the number of task data processed in a preset time length. The diagnosis data of each preset diagnosis item is compared with a corresponding preset condition, and the preset diagnosis item that does not meet the preset condition is input into a resource scheduling optimization model to obtain resource scheduling optimization strategy information. The preset condition is a threshold condition that is set in advance according to different preset diagnosis items. The resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information. The resource scheduling optimization strategy information includes resource optimization allocation scheme information. The embodiment of the present application can determine the preset diagnosis item that does not meet the preset condition, determine the resource scheduling optimization strategy information corresponding to the preset diagnosis item that does not meet the preset condition by using the resource scheduling optimization model, and therefore can accurately determine the bottleneck that affects the Yarn resource scheduling performance, automatically provide corresponding resource scheduling optimization strategy information for the bottleneck of the Yarn resource scheduling performance and improve the analysis efficiency of the Yarn resource scheduling performance. BRIEF DESCRIPTION OF DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.
[0019] Figure 1 is a flow diagram of the Yarn resource scheduling performance analysis method of the embodiment of the present application;
[0020] Figure 2 is a specific example diagram of the Yarn resource scheduling performance analysis method of the embodiment of the present application;
[0021] Figure 3 is a specific example diagram of the Yarn resource scheduling performance analysis method of the embodiment of the present application;
[0022] Figure 4 is a schematic diagram of the Yarn resource scheduling performance analysis device of the embodiment of the present application;
[0023] Figure 5 is a schematic diagram of the computer device in the embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings. Herein, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application, but not as a limitation of the present application.
[0025] The term "and / or" in this paper is only to describe a relationship, which means that there are three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, the term "at least one" in this paper means any one of the plurality or any combination of at least two of the plurality, for example, including at least one of A, B and C, which means including any one or more elements selected from the set consisting of A, B and C.
[0026] In the description of the present specification, "include", "including", "have", "has", and the like are open terms, that is, mean to include but not limited to. The description of the terms "one embodiment", "one specific embodiment", "some embodiments", "for example", and the like means that the specific features, structures, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. The order of the steps involved in each embodiment is used to illustrate the implementation of the present application, and the order of the steps is not limited and can be appropriately adjusted as needed.
[0027] The principles and spirits of the present application will be explained in detail below with reference to several representative embodiments of the present application.
[0028] Figure 1 is a flowchart of a Yarn resource scheduling performance analysis method according to an embodiment of the present application. As shown in Figure 1 , the method comprises:
[0029] Step 101, obtaining log data of an application manager, log data of a plurality of containers, and log data of an application in Yarn; wherein the application manager is used to schedule resources of the plurality of containers required when the application processes task data;
[0030] Step 102, extracting an identification number of each container, a plurality of running states of each container, and switching time of the plurality of running states from the log data of the application manager and the log data of the plurality of containers; determining a task data processing quantity according to the log data of the application;
[0031] Step 103, determining diagnosis data of a plurality of preset diagnosis items according to the identification number of each container, the plurality of running states of each container, the switching time of the plurality of running states, and the task data processing quantity; wherein the plurality of preset diagnosis items include: a container usage quantity, a consumption time length of each container in different running states, and a task data processing quantity within a preset time length;
[0032] Step 104, comparing the diagnosis data of each preset diagnosis item with a corresponding preset condition, inputting the preset diagnosis item that does not meet the preset condition into a resource scheduling optimization model to obtain resource scheduling optimization strategy information; wherein the preset condition is a threshold condition set in advance according to different preset diagnosis items; the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information; and the resource scheduling optimization strategy information includes resource optimization allocation scheme information.
[0033] Depend on Figure 1 As can be seen from the process shown, the embodiment of the present invention obtains the log data of the application manager, the log data of multiple containers, and the log data of the application in Yarn; wherein the application manager is used to schedule the resources of multiple containers required when the application processes task data; extracts the identification number of each container, the multiple operating states of each container, and the switching time of the multiple operating states from the log data of the application manager and the log data of multiple containers; determines the number of task data to be processed according to the log data of the application; determines the number of task data to be processed according to the identification number of each container, the multiple operating states of each container, the switching time of the multiple operating states, and the number of task data to be processed Determine the diagnostic data of multiple preset diagnostic items; wherein, the multiple preset diagnostic items include: the number of containers used, the consumption time of each container in different operating states, and the number of task data processed within the preset time; compare the diagnostic data of each preset diagnostic item with the corresponding preset conditions, input the preset diagnostic items that do not meet the preset conditions into the resource scheduling optimization model, and obtain resource scheduling optimization strategy information; wherein, the preset conditions are threshold conditions pre-set according to different preset diagnostic items; the resource scheduling optimization model is obtained by training the machine learning model based on historical preset diagnostic items and corresponding historical resource scheduling optimization strategy information; the resource scheduling optimization strategy information includes resource optimization allocation plan information. The embodiment of the present invention can determine the preset diagnostic items that do not meet the preset conditions, and use the resource scheduling optimization model to determine the resource scheduling optimization strategy information corresponding to the preset diagnostic items that do not meet the preset conditions. It can be seen that the embodiment of the present invention can accurately judge the bottlenecks that affect the Yarn resource scheduling performance, automatically provide corresponding resource scheduling optimization strategy information for the bottlenecks of the Yarn resource scheduling performance, and improve the analysis efficiency of the Yarn resource scheduling performance.
[0034] To explain the above YARN resource scheduling performance analysis method more clearly, each step is described in detail below.
[0035] In the present invention, the resource scheduling node (resourcemanager) is the management node of Yarn, which is responsible for receiving task requests and assigning tasks to the task execution node (nodemanager); the task execution node (nodemanager) is responsible for executing the tasks assigned by the resource scheduling node (resourcemanager) and reporting the task execution status to the resource scheduling node (resourcemanager); a Yarn task will start multiple containers (containers), and the first container started (container) runs the application manager (app master).
[0036] In an embodiment of the present application, the log data of the application manager, the log data of the plurality of containers, and the log data of the application are obtained in Yarn, specifically, the log data of the application manager is obtained from a resource scheduling node of Yarn, the log data of the plurality of containers is obtained from a task execution node of Yarn, and the log data of the application is obtained from an interface of Yarn.
[0037] In an embodiment of the present application, the identification number of each container, the plurality of running states of each container, and the switching time of the plurality of running states are extracted from the log data of the application manager and the log data of the plurality of containers, including:
[0038] In the log data of the application manager and the log data of the plurality of containers, the identification number of each container, the plurality of running states of each container, and the switching time of the plurality of running states are extracted by using a preset regular expression.
[0039] In a specific implementation, the log data of the application manager (app master) is obtained from a resource scheduling node (resource manager) of Yarn, and a preset regular expression is used to extract the time of the running state switching of the container (container) in the log data; for example, the following log data:
[0040] 2023-12-27-18:09:11,874INFO
[0041] org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:container_1703046293324_120479_01_000001Container Transitioned from NEWto ALLOCATED;
[0042] The ID number of the container can be extracted as container_1703046293324_120479_01_000001, the state before the running state switching is NEW (new, i.e., no resource is allocated), and the state after the running state switching is ALLOCATED (resource is allocated); the log time is 2023-12-27 18:09:11, and the information is recorded in a data table.
[0043] In specific implementation, log data of multiple containers is obtained from a task execution node (nodemanager) of Yarn, time of switching of running states of the containers in the log data is extracted using a regular expression, and information is recorded into a data table. Log data of an application (application) is obtained from an interface of Yarn. A mapping relationship between a scheduled task and the application (application) is collected in the log data, log data is parsed through a python script, and a corresponding relationship between an SQL task and the application (application) is extracted. The corresponding relationship is recorded into the data table.
[0044] Figure 2 is a specific example diagram of the Yarn resource scheduling performance analysis method of the embodiment of the application.
[0045] In an embodiment of the application, with reference to Figure 2 , diagnosis data of multiple preset diagnosis items is determined according to an identification number of each container, multiple running states of each container, switching time of the multiple running states, and a task data processing quantity, and includes:
[0046] Step 201, a container usage quantity is determined according to the identification number of each container.
[0047] Step 202, a consumption time length of each container in different running states is determined according to the multiple running states of each container and the switching time of the multiple running states.
[0048] Step 203, a task data processing quantity in a preset time length is determined according to the task data processing quantity and the preset time length.
[0049] In specific implementation, the following diagnosis items are calculated using log data of an application manager and log data of multiple containers:
[0050] 1. Allocation time of an application manager (app master) is calculated. The first started container (container) when a task is executed is a container of the application manager (app master), and all are ended with 000001. Then, the time of the converted state of the container (container) with an identification number (Id) ended with 000001 being ALLOCATED (allocated resource) is subtracted from the time of the converted state being NEW (new, i.e., not allocated resource), which is the allocation time of the application manager (app master);
[0051] 2、Calculate the localizing time consumption of the application master, the difference between the time when the converted state of the container with the identification number (Id) ending with 000001 is LOCALIZING and the time when the converted state is RUNNING is the localizing time consumption of the application master;
[0052] 3、Calculate the allocating time consumption of the common container (i.e. the container other than the application master), the difference between the time when the converted state is ALLOCATED and the time when the converted state is LOCALIZING is the allocating time consumption of the common container;
[0053] 4、Calculate the localizing time consumption of the common container, the difference between the time when the state is LOCALIZING and the time when the converted state is RUNNING is the localizing time consumption of the common container;
[0054] 5、Calculate the running time consumption of the common container, the difference between the time when the converted state is RUNNING and the log time when the state is RUNNING is the running time consumption of the common container;
[0055] 6、Calculate the number of containers used, by removing the identification numbers (Ids) of the containers through the state switching record information, the number of containers used for executing tasks is obtained;
[0056] 7、Calculate the input data volume of the application and the data volume of the inter-container transmission (shuffle), the input data volume is obtained through the statistical index item hdfs_bytes_read in the log data of the application, and the data volume of the inter-container transmission (shuffle) is obtained through the statistical index item shuffle_bytes.
[0057] In an embodiment of the application, the diagnosis data of each preset diagnosis item is compared with the corresponding preset condition, and the preset diagnosis item that does not meet the preset condition is compared. The preset condition is set in advance according to the actual situation, and each preset diagnosis item corresponds to a preset condition. Specifically, the application pre-establishes an analysis rule library (i.e. a preset condition), compares the diagnosis data of the preset diagnosis item calculated with the historical statistical data, combines the analysis rule library, and judges the time consumption abnormality reason.
[0058] Table 1
[0059]
[0060] In an embodiment of the present application, the diagnosis data of each preset diagnosis item is compared with the corresponding preset condition, the preset diagnosis item that does not meet the preset condition is input into a resource scheduling optimization model, and resource scheduling optimization strategy information is obtained; wherein, the preset condition is a threshold condition set in advance according to different preset diagnosis items; the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information; and the resource scheduling optimization strategy information includes resource optimization allocation scheme information.
[0061] In specific implementation, the running time, resource consumption, and usage queue of the application program are collected, and the usage of the Yarn resource queue is also collected; according to the preset diagnosis item that does not meet the preset condition, the check item that needs to be checked can be determined, and the corresponding optimization suggestion can be proposed, as shown in Table 2.
[0062] Table 2
[0063]
[0064] In an embodiment of the present application, the diagnosis data of each preset diagnosis item is compared with the corresponding preset condition, and further includes:
[0065] If the diagnosis data of each preset diagnosis item meets the corresponding preset condition, a prompt message that the Yarn resource scheduling performance is good is sent.
[0066] Figure 3 is a specific example diagram of the Yarn resource scheduling performance analysis method of the embodiment of the present application.
[0067] In an embodiment of the present application, referring to Figure 3 , first, data collection is performed, the log data of the application program manager is obtained from the resource scheduling node of Yarn, the log data of multiple containers is obtained from the task execution node of Yarn, and the log data of the application program is obtained from the interface of Yarn; the collected data is processed to determine the diagnosis data of multiple preset diagnosis items; the preset diagnosis item that does not meet the preset condition is determined according to the pre-established analysis rule library (i.e., the preset condition); an expert knowledge base is established, wherein the expert knowledge base includes: historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information; a resource scheduling optimization model is trained according to the expert knowledge base, the preset diagnosis item that does not meet the preset condition is input into the resource scheduling optimization model, and resource scheduling optimization strategy information (i.e., optimization suggestion) is obtained.
[0068] It is to be understood that even though operations of the method of the present application are described in a particular, sequential order in the above embodiments and figures, this is not meant to be limiting or imply that the operations must be performed in this particular, sequential order, or that all of the operations be performed, to achieve desirable results. Additionally or alternatively, certain steps can be omitted, combined with another step, or performed in a different order.
[0069] The implementation of the yarn resource scheduling performance analysis device can refer to the implementation of the above method, and the repeated parts will not be described here. The term "module" or "unit" used below can be a combination of software and / or hardware that achieves a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.
[0070] Based on the same inventive concept, the present application also provides a yarn resource scheduling performance analysis device, as shown in the figure, the device comprises: Figure 4 As shown in the figure, the device comprises:
[0071] The data acquisition module 401 is configured to acquire log data of an application manager, log data of a plurality of containers, and log data of an application in the Yarn, wherein the application manager is configured to schedule resources of the plurality of containers required when processing task data of the application.
[0072] The log extraction module 402 is configured to extract an identification number of each container, a plurality of running states of each container, and switching time of the plurality of running states from the log data of the application manager and the log data of the plurality of containers, and determine a task data processing quantity according to the log data of the application.
[0073] The diagnostic data determination module 403 is configured to determine diagnostic data of a plurality of preset diagnostic items according to the identification number of each container, the plurality of running states of each container, the switching time of the plurality of running states, and the task data processing quantity, wherein the plurality of preset diagnostic items comprise a container usage quantity, a consumption time length of each container in different running states, and a task data processing quantity within a preset time length.
[0074] The diagnostic data comparison module 404 is configured to compare the diagnostic data of each preset diagnostic item with a corresponding preset condition, input a preset diagnostic item that does not meet the preset condition into a resource scheduling optimization model, and obtain resource scheduling optimization strategy information, wherein the preset condition is a threshold condition set in advance according to different preset diagnostic items, the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnostic items and corresponding historical resource scheduling optimization strategy information, and the resource scheduling optimization strategy information comprises resource optimization allocation scheme information.
[0075] In an embodiment of the present application, the data acquisition module 401 is specifically configured to:
[0076] The log data of the application manager is acquired from a resource scheduling node of Yarn, the log data of the plurality of containers is acquired from a task execution node of Yarn, and the log data of the application is acquired from an interface of Yarn.
[0077] In an embodiment of the present application, the log extraction module 402 is specifically configured to:
[0078] In the log data of the application manager and the log data of the plurality of containers, the identification number of each container, the plurality of running states of each container, and the switching time of the plurality of running states are extracted by using a preset regular expression.
[0079] In an embodiment of the present application, the diagnosis data determination module 403 is specifically configured to:
[0080] The number of containers is determined according to the identification number of each container.
[0081] The consumption duration of each container in different running states is determined according to the plurality of running states of each container and the switching time of the plurality of running states.
[0082] The number of task data processing in the preset duration is determined according to the number of task data processing and the preset duration.
[0083] In an embodiment of the present application, the present application further comprises:
[0084] The message prompting module is configured to issue a prompt message that the Yarn resource scheduling performance is good if the diagnosis data of each preset diagnosis item satisfies the corresponding preset condition.
[0085] It should be noted that, although several modules of the Yarn resource scheduling performance analysis apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into a plurality of modules.
[0086] Based on the foregoing inventive concept, as shown in Figure 5 The present application further proposes a computer device 500, which comprises a memory 501, a processor 502, and a computer program 503 stored in the memory 501 and capable of running on the processor 502, wherein the processor 502 implements the foregoing Yarn resource scheduling performance analysis method when executing the computer program 503.
[0087] Based on the foregoing inventive concept, the present application provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the foregoing Yarn resource scheduling performance analysis method.
[0088] Based on the foregoing inventive concept, the present application provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the Yarn resource scheduling performance analysis method.
[0089] The Yarn resource scheduling performance analysis method and device provided by the embodiments of the present application can solve the problem that the method of manually analyzing the Yarn resource scheduling performance cannot accurately determine the bottleneck of the Yarn resource scheduling performance and has low analysis efficiency; the embodiments of the present application obtain the log data of an application manager, the log data of a plurality of containers, and the log data of an application program in the Yarn; the application manager is used to schedule the resources of a plurality of containers required when the application program processes task data; the identification number of each container, the plurality of running states of each container, and the switching time of the plurality of running states are extracted from the log data of the application manager and the log data of the plurality of containers; the number of task data processed is determined according to the log data of the application program; the diagnosis data of a plurality of preset diagnosis items is determined according to the identification number of each container, the plurality of running states of each container, the switching time of the plurality of running states, and the number of task data processed; the plurality of preset diagnosis items include the number of containers used, the consumption time length of each container in different running states, and the number of task data processed in a preset time length; the diagnosis data of each preset diagnosis item is compared with a corresponding preset condition, and the preset diagnosis item that does not meet the preset condition is input into a resource scheduling optimization model to obtain resource scheduling optimization strategy information; the preset condition is a threshold condition set in advance according to different preset diagnosis items; the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information; and the resource scheduling optimization strategy information includes resource optimization allocation scheme information. The embodiments of the present application can determine the preset diagnosis item that does not meet the preset condition, determine the resource scheduling optimization strategy information corresponding to the preset diagnosis item that does not meet the preset condition by using the resource scheduling optimization model, and thus can accurately determine the bottleneck affecting the Yarn resource scheduling performance, automatically provide the corresponding resource scheduling optimization strategy information for the bottleneck of the Yarn resource scheduling performance, and improve the analysis efficiency of the Yarn resource scheduling performance.
[0090] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.
[0091] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.
[0092] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.
[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.
[0094] The specific embodiments described above are examples for purposes of explanation and illustration and are not intended to limit the scope of the application. The scope of the application is limited only by the claims that follow.
Claims
1. A method for performance analysis of Yarn resource scheduling, characterized in that, The application comprises the following steps: Obtaining log data of an application manager, log data of multiple containers, and log data of an application in Yarn, wherein the application manager is used to schedule resources of the multiple containers required when the application processes task data; Extracting an identification number of each container, multiple running states of each container, and switching time of the multiple running states from the log data of the application manager and the log data of the multiple containers, and determining a task data processing quantity according to the log data of the application; Determining diagnosis data of multiple preset diagnosis items according to the identification number of each container, the multiple running states of each container, the switching time of the multiple running states, and the task data processing quantity, wherein the multiple preset diagnosis items comprise a container usage quantity, a consumption time length of each container in different running states, and a task data processing quantity within a preset time length; Comparing the diagnosis data of each preset diagnosis item with a corresponding preset condition, inputting a preset diagnosis item that does not meet the preset condition into a resource scheduling optimization model, and obtaining resource scheduling optimization strategy information, wherein the preset condition is a threshold condition set in advance according to different preset diagnosis items, the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information, and the resource scheduling optimization strategy information comprises resource optimization allocation scheme information.
2. The method of claim 1, wherein, The application comprises the following steps: Obtaining log data of an application manager, log data of multiple containers, and log data of an application in Yarn, wherein the application manager is used to schedule resources of the multiple containers required when the application processes task data; 3. The method of claim 1, wherein, Extracting an identification number of each container, multiple running states of each container, and switching time of the multiple running states from the log data of the application manager and the log data of the multiple containers, and determining a task data processing quantity according to the log data of the application; Determining diagnosis data of multiple preset diagnosis items according to the identification number of each container, the multiple running states of each container, the switching time of the multiple running states, and the task data processing quantity, wherein the multiple preset diagnosis items comprise a container usage quantity, a consumption time length of each container in different running states, and a task data processing quantity within a preset time length; 4. The method of claim 1, wherein, Comparing the diagnosis data of each preset diagnosis item with a corresponding preset condition, inputting a preset diagnosis item that does not meet the preset condition into a resource scheduling optimization model, and obtaining resource scheduling optimization strategy information, wherein the preset condition is a threshold condition set in advance according to different preset diagnosis items, the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information, and the resource scheduling optimization strategy information comprises resource optimization allocation scheme information. The application comprises the following steps: Obtaining log data of an application manager, log data of multiple containers, and log data of an application in Yarn, wherein the application manager is used to schedule resources of the multiple containers required when the application processes task data; Extracting an identification number of each container, multiple running states of each container, and switching time of the multiple running states from the log data of the application manager and the log data of the multiple containers, and determining a task data processing quantity according to the log data of the application; 5. The method of claim 1, wherein, Determining diagnosis data of multiple preset diagnosis items according to the identification number of each container, the multiple running states of each container, the switching time of the multiple running states, and the task data processing quantity, wherein the multiple preset diagnosis items comprise a container usage quantity, a consumption time length of each container in different running states, and a task data processing quantity within a preset time length; Comparing the diagnosis data of each preset diagnosis item with a corresponding preset condition, inputting a preset diagnosis item that does not meet the preset condition into a resource scheduling optimization model, and obtaining resource scheduling optimization strategy information, wherein the preset condition is a threshold condition set in advance according to different preset diagnosis items, the resource scheduling optimization model is obtained by training a machine learning model according to historical preset diagnosis items and corresponding historical resource scheduling optimization strategy information, and the resource scheduling optimization strategy information comprises resource optimization allocation scheme information.
6. A Yarn resource scheduling performance analysis apparatus, characterized by, The data acquisition module is configured to acquire log data of an application manager, log data of a plurality of containers, and log data of an application in Yarn. The log extraction module is configured to extract, from the log data of the application manager and the log data of the plurality of containers, an identification number of each container, a plurality of running states of each container, and switching time of the plurality of running states. The diagnosis data determination module is configured to determine diagnosis data of a plurality of preset diagnosis items according to the identification number of each container, the plurality of running states of each container, the switching time of the plurality of running states, and the task data processing quantity. The diagnosis data comparison module is configured to compare the diagnosis data of each preset diagnosis item with a corresponding preset condition, input a preset diagnosis item that does not meet the preset condition into a resource scheduling optimization model, and obtain resource scheduling optimization strategy information.
7. The apparatus of claim 6, wherein, The data acquisition module is specifically configured to: acquire the log data of the application manager from a resource scheduling node of Yarn, acquire the log data of the plurality of containers from a task execution node of Yarn, and acquire the log data of the application from an interface of Yarn.
8. The apparatus of claim 6, wherein, The log extraction module is specifically configured to: extract, from the log data of the application manager and the log data of the plurality of containers, the identification number of each container, the plurality of running states of each container, and the switching time of the plurality of running states by using a preset regular expression.
9. The apparatus of claim 6, wherein, The diagnosis data determination module is specifically configured to: determine the container usage quantity according to the identification number of each container; determine the consumption time of each container in different running states according to the plurality of running states of each container and the switching time of the plurality of running states; and determine the task data processing quantity in a preset time period according to the task data processing quantity and the preset time period.
10. The apparatus of claim 6, wherein, The message prompting module is further configured to issue a prompt message that Yarn resource scheduling performance is good if the diagnosis data of each preset diagnosis item meets the corresponding preset condition. The processor executes the computer program to implement the method in any one of claims 1 to 5.
11. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 5.
12. A computer-readable storage medium, characterized in that, The computer program product includes a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 5.
13. A computer program product, characterised in that,