Slow node detection method during task running, apparatus, electronic device, and medium

Abstract

To provide a slow node detection method during task running, an apparatus, an electronic device, and a storage medium.SOLUTION: A method includes the steps of: obtaining running indicator information of a streaming data processing task, the streaming data processing task including sub-tasks of a plurality of operator types; determining a target operator type that is abnormal, on the basis of a topological relationship between respective operator types and abnormality reason information that is indicated by the running indicator information; filtering first sub-tasks of the target operator type from the sub-tasks of the plurality of operator types, and determining, from the running indicator information, to-be-detected indicator information respectively corresponding to the first sub-tasks; and determining, by using the to-be-detected indicator information, a second sub-task that runs slowly from the first sub-tasks, and determining a slow node machine, on the basis of an execution machine on which a computing node that executes the second sub-task is located.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the field of computer technology, and more particularly to a method, device, electronic device, and medium for detecting a slow node during task execution. [Background technology]

[0002] With the development of business, the processing of stream data has become more and more common. As a big data stream processing engine commonly used in the industry, the Flink engine can be used to process various Flink tasks. The Flink engine management has multiple computing nodes to process subtasks corresponding to Flink tasks, and the multiple computing nodes can run on different machines in the service cluster. However, in many cases, the same machine can run multiple types of subtasks corresponding to multiple computing nodes, and even run tasks corresponding to other engines, so the continuous running of various tasks may slow down the execution of certain subtasks running on the machine, affecting the execution efficiency of the task. At the same time, for the subtask, the machine will have the problem of a slow node.

[0003] Therefore, if machines with slow node problems can be detected and processed in a timely manner, the execution efficiency of Flink tasks can be greatly improved. However, traditional slow node detection solutions often have the problem of low detection accuracy and low detection effect. Summary of the Invention

[0004] SUMMARY OF THE DISCLOSURE The embodiments of the present disclosure provide a method, apparatus, electronic device, and medium for detecting a slow node during at least task execution.

[0005] In a first aspect, an embodiment of the present disclosure comprises: obtaining performance indicator information for a stream data processing task including a plurality of operator type subtasks; determining a target operator type having an abnormality according to the topology relationship of various operator types and the abnormality cause information indicated by the execution index information; selecting each first subtask of the target operator type from among the subtasks of the plurality of operator types, and determining index information of a detection target corresponding to each of the first subtasks from the execution index information; The present invention provides a method for detecting a slow node during task execution, the method including the steps of: determining a second subtask having a slow execution speed from the first subtask using index information of the detection target; and determining a slow node machine based on an execution machine in which a computation node executing the second subtask exists.

[0006] In a possible embodiment, after determining the slow node machine, pre-requesting target resources for each target computing node in the slow node machine based on an amount of resources required for each target computing node to execute the second subtask, using a pre-configured service migration interface; setting blocking information for the slow node machine, the blocking information being used to indicate that new subtasks of the target operator type cannot be scheduled to be executed by a compute node of the slow node machine; creating an alternative computing node to substitute for the target computing node using the requested target resources, and using the alternative computing node to execute the second subtask associated with the target computing node.

[0007] In a possible embodiment, the step of determining a slow node machine based on an execution machine on which a computing node executing the second subtask exists includes: When the target number of each of the second subtasks is greater than a preset number, determining the number of tasks corresponding to each of the execution machines based on the execution machines on which the computing nodes executing each of the second subtasks exist; determining the slow node machine based on a first ratio value between the number of tasks corresponding to each of the execution machines and the target number;

[0008] In a possible embodiment, the step of determining the slow node machine based on a first ratio value between the number of tasks corresponding to each of the execution machines and the target number includes: If there is a target ratio value that is equal to or greater than the first preset ratio value, use the execution machine that corresponds to the target ratio value as the slow node machine; or When each of the first ratio values ​​is smaller than the first preset ratio value, the step includes using an execution machine corresponding to the second subtask having the largest index anomaly degree as the slow node machine based on the index anomaly degree indicated by the index information of the detection target of each of the second subtasks.

[0009] In a possible embodiment, the step of determining the number of tasks corresponding to each execution machine based on an execution machine on which a computing node executing each of the second subtasks is present comprises: When the target operator type belongs to a data consumption type, determining a data delay time corresponding to each data partition from the index information of the detection target corresponding to the second subtask based on each data partition that each second subtask depends on when reading data from the message queue; filtering a plurality of target data partitions from the data partitions based on the data delay time; determining a number of data partitions corresponding to each of the data management servers based on a data management server to which each of the target data partitions belongs; and when a second ratio value between the number of the data partitions corresponding to each of the data management servers and the first number of the target data partitions is smaller than a second preset ratio value, determining the number of the tasks corresponding to each of the execution machines based on the execution machines on which the computation nodes of each of the second subtasks exist.

[0010] In a possible embodiment, the step of determining a second subtask having a slow execution speed from the first subtask using the index information of the detection target includes: When the target operator type belongs to a data consumption type, determining a data delay time corresponding to each data partition from the index information of the detection target corresponding to the first subtask based on each data partition that each first subtask depends on when reading data from the message queue; determining a second number of data partitions based on the data latency, the second number being greater than the predetermined number of data partitions; and determining the second subtask from the first subtask based on the data delay time corresponding to each of the data partitions when a third ratio value between the second number and a total number of partitions is smaller than a third preset ratio value, wherein the total number of partitions is the total number of data partitions on which each of the first subtasks depends.

[0011] In a possible embodiment, the step of determining the second subtask from the first subtask based on the data delay time corresponding to each of the data partitions comprises: determining whether or not there is a filtering target data partition whose data latency time satisfies a first abnormality condition in each of the data partitions based on the data latency time corresponding to each of the data partitions; If so, determining the second subtask based on queries per second of the first subtask on which the filtered data partition depends.

[0012] In a possible embodiment, the step of determining a second subtask having a slow execution speed from the first subtask using the index information of the detection target includes: When the target operator type belongs to an operator type other than a data-consuming type, determining whether or not there is a third subtask whose task idle state satisfies a second abnormality condition based on the task idle state indicated by the detection target index information of each of the first subtasks; If so, determining the second subtask based on the queries per second of the third subtask and a preset query rate.

[0013] In a second aspect, an embodiment of the present disclosure comprises: An acquisition module for acquiring performance indicator information of a stream data processing task including a plurality of operator type subtasks; A first determination module for determining a target operator type having an abnormality according to the topology relationship of various operator types and the abnormality cause information indicated by the performance indicator information; a second determination module for filtering each first subtask of the target operator type from a plurality of subtasks of the operator types, and determining, from the performance indicator information, indicator information of a detection target corresponding to each first subtask; The present invention further provides an apparatus for detecting slow nodes during task execution, the apparatus including: a third determination module for determining a second subtask having a slow execution speed from the first subtask using index information of the detection target; and determining a slow node machine based on an execution machine in which a computation node executing the second subtask is located.

[0014] In a third aspect, an optional implementation of the present disclosure further provides an electronic device including a processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor executing the machine-readable instructions stored in the memory, and the machine-readable instructions, when executed by the processor, performing the steps of the first aspect above, or any possible embodiment of the first aspect.

[0015] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium having stored thereon a computer program which, when executed, performs the steps of the first aspect described above, or any possible embodiment of the first aspect.

[0016] For the effects of the slow node detection device, electronic device, and computer-readable storage medium during task execution, please refer to the description of the slow node detection method during task execution, and the description will not be repeated here.

[0017] In the method, device, electronic device, and storage medium for detecting a slow node during task execution provided by the embodiment of the present disclosure, the execution indicator information can include indicator information for each subtask of each operator type during the execution of a stream data processing task, so that the acquired execution indicator information can be used to detect a slow node, thereby realizing comprehensive analysis and detection of various indicator information and improving the accuracy of the determined slow node machine. Based on the topology structure between operator types and the abnormality cause information, it can be accurately determined whether the task abnormality of the operator type is caused by the task abnormality of other operator types, and further, from multiple operator types, it can accurately filter the target operator type that may cause a slow node problem. When analyzing whether there is a slow node problem, since subtasks of different operator types need to use different indicator information, after determining the target operator type, it is possible to filter the indicator information of the detection target corresponding to each of the first subtasks from the execution indicator information, detect a slow node using the indicator information of the detection target, and accurately determine the second subtask with a slow execution speed from each first subtask. Then, since the subtask can be executed by the computation node of the machine included in the service cluster, after the second subtask is determined, the slow node machine with the slow node problem can be accurately determined based on the execution machine where the computation node executing the second subtask exists, so that the slow node problem can be accurately detected.In addition, since the execution indicator information can be uploaded in real time by the execution machine during the execution of the stream data processing task, the execution indicator information can be obtained in real time, and further, the execution indicator information can be used to grasp the problem of the slow node in real time, thereby improving the real-time detection.

[0018] In order to make the above-mentioned objects, features and advantages of the present disclosure more clear and understandable, preferred embodiments are given below and described in detail with reference to the accompanying drawings. [Brief description of the drawings]

[0019] In order to more clearly explain the technical solutions of the embodiments of the present disclosure, the drawings used in the embodiments are briefly introduced below. These drawings are incorporated in and constitute a part of this specification. These drawings show embodiments consistent with the present disclosure and are used to explain the technical solutions of the present disclosure together with the specification. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on these drawings without creative efforts.

[0020] [Figure 1] FIG. 1 shows a flowchart of a method for detecting a slow node during task execution provided by an embodiment of the present disclosure. [Diagram 2] FIG. 2 illustrates a specific flowchart for determining the slow-running second subtask provided by an embodiment of the present disclosure. [Diagram 3] FIG. 3 illustrates a specific flowchart for determining a slow node machine provided by an embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates an interaction schematic diagram of a slow node detection method provided by an embodiment of the present disclosure. [Diagram 5] FIG. 5 illustrates a schematic diagram of an apparatus for detecting a slow node during task execution provided by an embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic structural diagram of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure are described below clearly and completely with reference to the accompanying drawings in the embodiments of the present disclosure, and obviously, the described embodiments are only a part, not all, of the embodiments of the present disclosure. The components of the embodiments of the present disclosure generally described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative efforts are included in the scope of protection of the present disclosure.

[0022] Additionally, terms such as "first," "second," and the like in the specification and claims of the embodiments of the present disclosure, as well as in the figures described above, are used to distinguish between similar objects, but are not necessarily used to describe a particular order or sequence, and it should be understood that such terms may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced in orders other than those illustrated or described herein. As referred to herein, "multiple or several" means two or more. "And / or" represents a relationship between related objects and means that a three-way relationship may exist, for example, A and / or B may represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " usually represents that related objects are in an "or" relationship.

[0023] According to the survey, for the detection of slow nodes in stream data processing scenarios, the common detection method is to use some machine monitoring services to detect the core indicators of the machine, such as the utilization rate of the central processing unit (CPU) of the machine, and when the utilization rate reaches a threshold, the machine is deemed to have a slow node problem. Although this method can detect the problem of slow nodes, it usually has the problem that the detection is inaccurate. For example, under normal circumstances, a conservative threshold should be set, but because stream data processing tasks are very sensitive to delays and the sensitivity is also different for different types of stream data processing tasks, the conservative threshold cannot cover all types of stream data processing tasks, which affects the accuracy of detection. In addition, this method does not have the ability to monitor the operating environment of the cluster. For example, if a sudden fluctuation or delay occurs in the machine network of a certain network segment, this method cannot detect it, resulting in a deviation in the detection result.

[0024] Based on the above research, the present disclosure provides a method, device, electronic device, and storage medium for detecting a slow node during the execution of a task. Since the execution indicator information can include indicator information for each subtask of each operator type during the execution of a stream data processing task, the acquired execution indicator information can be used to detect a slow node, thereby realizing comprehensive analysis and detection of various indicator information and improving the accuracy of the determined slow node machine. Based on the topology structure between operator types and the abnormality cause information, it is possible to accurately determine whether the task abnormality of the operator type is caused by the task abnormality of other operator types, and accurately filter the target operator type that may cause a slow node problem from multiple operator types. When analyzing whether there is a slow node problem, since subtasks of different operator types need to use different indicator information, after determining the target operator type, the corresponding detection target indicator information of each first subtask is filtered from the execution indicator information, and the detection target indicator information is used to detect a slow node, and the second subtask with a slow execution speed can be accurately determined from each first subtask. And because the subtask can be executed by the computation node of the machine included in the service cluster, after the second subtask is determined, the slow node machine with the slow node problem can be accurately determined based on the execution machine where the computation node executing the second subtask is located, so that the slow node problem can be accurately detected.In addition, since the execution indicator information can be uploaded in real time by the execution machine during the execution of the stream data processing task, the execution indicator information can be obtained in real time, and further, the execution indicator information can be used to grasp the problem of the slow node in real time, thereby improving the real-time detection.

[0025] All the deficiencies in the above solutions are the results obtained by the inventor after practice and careful study, so the discovery process of the above problems and the following solutions proposed by this disclosure for the above problems are the contributions made by the inventor to this disclosure in the course of this disclosure.

[0026] It should be noted that like numerals and letters represent similar items in the following figures, so that once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

[0027] Before using the technical solutions disclosed in each embodiment of the present disclosure, the type, scope of use, and usage scenario of the personal information related to the present disclosure will be notified to the user by appropriate means in accordance with relevant laws and regulations, and the user's consent will be obtained.

[0028] It should be noted that specific nouns referred to in the embodiments of this disclosure include the following: Flink: Its Chinese name is Flink, and it is an open source stream processing framework. Its core is a distributed stream dataflow engine written in Java and Scala. Flink executes arbitrary stream data programs in a data-parallel and pipelined manner. Kubernetes, abbreviated as K8s, is an open source application used to manage containerized applications on multiple hosts in a cloud platform. The goal of Kubernetes is to make the distributed deployment of containerized service clusters simple and efficient. Yarn: A general resource management system that provides integrated resource management and scheduling for higher-layer applications.

[0029] In order to facilitate understanding of the present embodiment, the method for detecting a slow node during task execution disclosed in the embodiment of the present disclosure will be described in detail first. The execution subject of the method for detecting a slow node during task execution provided by the embodiment of the present disclosure is generally a terminal device with a certain computing capability or other processing device, and the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, an electronic device, etc., and in some possible implementation forms, the method for detecting a slow node during task execution can be realized by a processor calling a computer-readable instruction stored in a memory.

[0030] Hereinafter, a method for detecting a slow node during task execution provided by an embodiment of the present disclosure will be described, taking an example in which the execution subject is an electronic device.

[0031] As shown in FIG. 1, a flowchart of a method for detecting a slow node during task execution provided by an embodiment of the present disclosure, which may include the following steps:

[0032] S101: Acquire execution index information of a stream data processing task including subtasks of multiple operator types.

[0033] Here, the stream data processing task may be a task that uses a Flink engine to process continuous stream data according to data processing requirements. Stream data processing scenarios often include multiple types of stream data processing tasks, and different types of stream data processing tasks correspond to different data processing requirements. A type of stream data processing task is referred to as a type of Flink task, and the following Flink tasks all refer to stream data processing tasks. The method for detecting slow nodes during task execution provided by the embodiments of the present disclosure can be used to detect slow nodes for any type of stream data processing task. Illustratively, the Flink task may be, for example, a list calculation task, an order number calculation task, an interaction information calculation task, an information recommendation task, etc.

[0034] In the case of a stream data processing task, it may include subtasks of multiple operator types, and each operator type may include at least one subtask, and the number of subtasks of each operator type may be determined based on the parallelism of the task corresponding to the operator type. Each subtask may be executed by a computation node in a machine included in a service cluster, and one service cluster may include at least one machine, and one machine may include at least one computation node. A Flink task may be executed by at least one service cluster, and each Flink cluster may include one Flink engine, and one Flink engine may correspond to one Job Manager (JobManager, abbreviated as JM) and multiple Task Managers (TaskManager, abbreviated as TM), and one TM is one computation node in a machine, and one TM may execute at least one subtask, and the JM is used to manage each TM and schedule subtasks that need to be executed for the TM.

[0035] Subtasks of different operator types often have different computing capabilities and different data processing functions, and are used to process stream data in different ways. Exemplarily, the operator types may include, for example, a source operator of data consumption type, a sink operator of data output type, a map operator of data repetition type, and a sync operator of data synchronization type. Subtasks of different operator types have communication functions, and there are topological relationships between the various operator types, and the topological relationships are used to indicate the communication relationships and execution sequences between the subtasks of the various operator types, for example, the topological relationship may be source operator->sync operator->sink operator, and the communication relationship indicated by the topological relationship is as follows: there is a communication relationship between the subtasks of the source operator type and the subtasks of the sync operator, and there is a communication relationship between the subtasks of the sync operator and the subtasks of the sink operator, and the execution sequence indicated is to execute the subtasks of the source operator type, then execute the subtasks of the sync operator, and then execute the subtasks of the sync operator.

[0036] Optionally, an operator type can be subdivided into multiple subtypes, each subtype including the subtasks of that subtype. For example, a sink operator of data output type can include three subtypes: sink1 operator corresponding to data output type 1, sink2 operator corresponding to data output type 2, and sink3 operator corresponding to data output type 3. In this case, the topological relationship between operator types can be the topological relationship between various operator types and subtypes. For example, the topological relationship can be source operator->sync operator->sink1 operator->sink2 operator->sink3 operator, and the sync operator can be followed by sink1 as well as sink3 operator.

[0037] The execution indicator information is indicator information for determining whether there is a problem with a slow node, and during the execution of a stream data processing task, it can indicate engine indicator information of each type of each Flink engine associated with the stream data processing task, specifically, during the execution of a stream data processing task, it can include indicator information related to each subtask of each operator type, and some indicator information related to the entire stream data processing task. For ease of understanding, the execution indicator information of a Flink task can include, for example, some indicator information related to the entire Flink task, and indicator information related to the subtasks executed in each TM.

[0038] Exemplarily, the execution indicator information may include the data delay time of the Flink task, the data delay indicator of each subtask included in the Flink task (i.e., the delay time indicator), the abnormality cause indicator of various operator types (i.e., the backpressured indicator), and the idle state indicator of each subtask (i.e., the busy indicator), among which the data delay indicator is used to indicate the data delay time of the task, the abnormality cause indicator is used to indicate the backpressure degree of each operator type, and if the backpressure degree corresponding to the backpressured indicator of a certain operator type is greater than the backpressure threshold, it means that the subtask of the operator type is backpressured by the subtask of another operator type having a topological relationship, and therefore the abnormality cause indicator is used to represent the abnormality cause information of various operator types, and the idle state indicator is used to represent the busy degree of the task.

[0039] In a specific implementation, a machine in each service cluster related to a Flink task can upload the execution indicator information generated during the execution of the Flink task to the indicator database. Among them, the indicator database may be an external storage system independent of the machine providing the slow node automatic detection service and each service cluster, and is used to dynamically maintain the execution indicator information generated by each service cluster during the execution of the Flink task. When it is determined that slow node detection is required for a service cluster executing a Flink task, the execution indicator information generated by the service cluster within the corresponding execution time can be obtained from the indicator database. Here, the execution time may be a preset period such as the past 15 minutes, the past 30 minutes, etc., or may be the duration during which the service cluster has executed the Flink task.

[0040] It should be noted that the method for detecting slow nodes during task execution provided by the embodiment of the present disclosure can be set on a machine in the form of an automatic detection service for slow nodes, and when the stream data processing task that needs to be detected is a large-scale or very large-scale task, the slow node automatic detection service can be distributed and set in at least one service cluster. When slow node detection needs to be performed for any type of Flink task, any idle machine can be determined from the service cluster according to a load balancing strategy, and the machine is used to detect slow nodes for the Flink task according to the method for detecting slow nodes during task execution provided by the embodiment of the present disclosure based on the configured automatic detection service for slow nodes. For example, a detection period can be preset, and slow node detection can be performed for the service cluster corresponding to the Flink task according to the preset detection period. The detected slow nodes may specifically be one or more machines in the service cluster.

[0041] S102: Determine a target operator type having an abnormality based on the topology relationships between various operator types and the abnormality cause information indicated by the performance index information.

[0042] Here, the topological relationship is used to indicate the communication relationship and execution order between tasks of each operator type. The abnormality cause information is used to indicate the association relationship between the task abnormality of a specific operator type and the task abnormality of other operator types, and the association relationship can reflect whether the task abnormality of a specific operator type is caused by the task abnormality of other operator types. The abnormality cause information is information indicated by the backpressure indicators of various operator types in the execution indicator information, and when the degree of backpressure corresponding to the backpressure indicator of an operator type is greater than the backpressure threshold, it can indicate whether the subtask abnormality of an operator type is caused by abnormal backpressure from the subtask of other operator types having a topological relationship, so that the abnormality cause information indicated by the backpressure indicator can indicate whether the task abnormality of an operator type is related to the task abnormality of other operator types having a topological relationship.

[0043] One operator type can correspond to the abnormality cause information, and is determined according to the degree of backpressure indicated by the backpressure indicator of the operator type.

[0044] Exemplarily, the topological relationship may be source operator->sync operator->sink1 operator->sink2 operator, and if the backpressure indicators corresponding to the source operator and the sync operator are both greater than the backpressure threshold, and the degree of backpressure indicated by the sink1 operator indicator is less than the backpressure threshold, it can be determined that the abnormality of the subtask of the source operator type is caused by the abnormal backpressure of the subtask of the sync operator type, and the abnormality of the subtask of the sync operator type is caused by the abnormal backpressure of the subtask of the sink1 operator type. Thus, the abnormality cause information corresponding to the source operator type may be "yes", the sync operator may be "yes", and the abnormality cause information corresponding to the sink1 operator type and the sink2 operator type are both "no".

[0045] In specific implementation, after obtaining the execution indicator information of the Flink task, the task delay corresponding to the Flink task can be determined based on the data delay indicator related to the Flink task in the execution indicator information. If the task delay is less than or equal to the task delay threshold, it can be determined that there is no problem with a slow node. Conversely, if the task delay is greater than the task delay threshold, it can be determined that the Flink task has a large delay problem, and the idle application programming interface (API) of the Flink engine is used to determine the abnormality cause information corresponding to each operator type based on the degree of backpressure indicated by the backpressure indicator corresponding to each operator type in the execution indicator information.

[0046] Then, based on the anomaly cause information corresponding to the various operator types, it is determined whether backpressure exists in the subtask of each operator type, i.e., whether the anomaly cause information corresponding to the various operator types includes anomaly cause information indicating that a task anomaly of a specific operator type is related to a task anomaly of another operator type. If so, it can be determined that there is backpressure in the subtask, and based on the topological relationships between the various operator types, a root type that causes backpressure is determined from the multiple operator types, and the type is used as the target operator type. For example, when the topological relationship is source operator->sync operator->sink1 operator->sink2 operator->sink3 operator, if the anomaly cause information corresponding to the source operator type and the sync operator all indicate that backpressure is applied, and the anomaly cause information corresponding to the sink1 operator type, the sink2 operator type, and the sink3 operator type all indicate that backpressure is not applied, it can be determined that the root type is the sink1 operator type, i.e., the sink1 operator type is determined to be the target operator type. The anomaly corresponding to the target operator type is used to indicate that there is an anomaly in the subtask of the operator type.

[0047] If it is determined that the anomaly cause information for each operator type indicates that no backpressure is being applied, i.e., if it is determined that no backpressure is being applied to the subtasks of the various operator types, then it can be determined that the target operator type is a data consuming type (i.e., a source operator type).

[0048] S103: From the subtasks of the multiple operator types, each first subtask of the target operator type is filtered, and index information of the detection target corresponding to each first subtask is determined from the execution index information.

[0049] Here, the first subtask is a subtask having a target operator type, and the index information to be detected may be index information for determining whether the first subtask of the target operator type has a slow node problem in the execution index information.

[0050] Subtasks of different operator types require different slow node indicator information, i.e., different execution indicator information is used when detecting whether subtasks of different operator types have a slow node problem. For example, if the target operator type is a source operator type, it can be determined by the data delay time indicated by the data delay indicator, and if the target operator type is an operator type other than the source operator type, it can be determined by the task idle state indicated by the idle state indicator.

[0051] Exemplarily, when the target operator type is a source operator type, a data delay indicator corresponding to each of the first subtasks of the source operator type can be obtained from the execution indicator information, and when the target operator type is an operator type other than the source operator type, an idle state indicator corresponding to each of the first subtasks can be obtained from the execution indicator information.

[0052] S104: Using the index information of the detection target, a second subtask having a slower execution speed is determined from the first subtask, and a slow node machine is determined based on an execution machine in which a calculation node executing the second subtask exists.

[0053] Here, the second subtask may be the first subtask in the TM that has a slow execution speed, and the execution machine is the machine in which the TM that executes the second subtask is located. The slow node machine may be a machine that has a slow node problem among the machines included in each service cluster that executes the Flink task.

[0054] In a specific implementation, for each first subtask, it can be determined whether there is an abnormality in the index information of the detection target of the first subtask, and if there is an abnormality, the first subtask can be used as the second subtask, and if there is no abnormality, it can be determined that the first subtask is executed normally. Alternatively, for each first subtask, a detection method for a slow node of the target operator type can be determined based on the target operator type of the first subtask, and the detection method for the slow node is used to detect the index information of the detection target of the first subtask and determine whether the first subtask can be used as the second subtask. The detection methods for slow nodes corresponding to various target operator types will be described in detail later.

[0055] After determining the second subtask, the execution machine on which the TM for executing the second subtask is located can be determined first, and then the execution machine can be directly used as the slow node machine. Alternatively, after determining the execution machine on which the TM for executing the second subtask is located, the indicator to be detected can be used to further detect the execution machine to determine whether the execution machine has a slow node problem, and thereby determine the accurate slow node machine.

[0056] Optionally, if Flink itself does not have a slow node problem, the number of determined second subtasks may be 0, and the number of slow node machines may also be 0.

[0057] In one embodiment, for the step of "determining the second subtask" in S104, if the target operator type is different, it can be completed using different slow node detection methods. Specifically, if the target operator type is a data consumption type, the second subtask can be determined according to the slow node detection methods shown in the following S104-1 to S104-3.

[0058] S104-1: If the target operator type belongs to a data consumption type, based on each data partition on which each first subtask relies when reading data from the message queue, determine a data delay time corresponding to each data partition from the index information of the detection target corresponding to the first subtask.

[0059] Here, the first subtask of the source operator type is used to read data from a message queue (abbreviated as Message Queue, MQ), where there can be multiple data partitions in the message queue, and different data partitions are used to store stream data of different topics. The message queue may be any type of message queue in the prior art, such as RabbitMQ, RocketMQ, ActiveMQ, Kafka Message Queue (kafka), etc. In the following, we will use kafka queue as a message queue as an example for explanation. The data partition may be each partition in the kafka queue.

[0060] A subtask can read and consume data in at least one data partition of the message queue, and different subtasks depend on different data partitions when reading data from the message queue.

[0061] When the target operator type belongs to a source operator type, the index information to be detected may include a data delay index corresponding to each data partition on which each first subtask depends, the data delay index may indicate a data delay time corresponding to the data partition, and the data delay time of the data partition may be expressed as a partition delay (Partition lag).

[0062] In a specific implementation, when the target operator type is a source operator type, the data partitions used by each first subtask of the source operator type when reading stream data from the Kafka queue can be determined first. Here, one subtask corresponds to at least one data partition. Next, for each subtask, a data delay indicator corresponding to each data partition used by the first subtask is determined from the index information of the detection target of the first subtask, and the data delay time corresponding to each data partition is determined based on the data delay indicator, that is, the partition lag of each partition is determined.

[0063] S104-2: Based on the data latency, determine a second number of data partitions whose data latency is greater than the predetermined time.

[0064] Here, the preset time may be set based on experience, and is not specifically limited in the embodiment of the present disclosure. Exemplarily, the preset time may be 0. The second number is the number of data partitions whose partition lag is greater than the preset time.

[0065] In particular, when implemented, after obtaining the partition lag of each data partition corresponding to each first subtask, it can be determined whether there is a data partition whose partition lag is greater than a preset time, that is, whether there is a data partition that is delayed. If not, it can be determined that each first subtask is executed normally and there is no problem of slow execution speed. If so, it can be determined that a second number of data partitions whose partition lag is greater than a preset time.

[0066] S104-3: If a third ratio value between the second number and the total number of partitions is smaller than a third preset ratio value, determine a second subtask from the first subtask based on the data delay time corresponding to each data partition, and the total number of partitions is the total number of data partitions on which each first subtask depends.

[0067] Here, the total number of partitions is the sum of the number of data partitions that each first subtask depends on. Here, the third preset ratio value can be set based on experience, and is not specifically limited in the embodiment of the present disclosure. For example, the third preset ratio value may be 1 / 5.

[0068] In a specific implementation, after determining the second number, determine whether a third ratio value between the second number and the total number of partitions is smaller than a third preset ratio value. If not, it can be determined that the Flink task will experience a large delay unrelated to the machine, and it can be determined that there is no second subtask in the first subtask. If so, it can be determined that the data delay of the second number of data partitions may be caused by the machine executing the first subtask, and filter the second subtasks from the first subtasks based on the partition lag of the data partition corresponding to each first subtask. For example, the first subtasks corresponding to the data partitions whose partition lag is greater than the preset partition time can be used as the filtered second subtasks.

[0069] In one embodiment, the step of "determining a second subtask from the first subtask based on the data delay time corresponding to each data partition" in S104-3 can be implemented according to the following steps:

[0070] S104-3-1: Based on the data latency corresponding to each data partition, it is determined whether or not there is a data partition to be filtered, the data latency of which satisfies a first abnormality condition, in each data partition.

[0071] Here, the first abnormal condition is used to filter abnormal delay times from the data delay times of each data partition. Exemplarily, the first abnormal condition may be that the data delay time is greater than a first designated time, or that the difference between the data delay time of each data partition and the median value of the data delay time is greater than a second designated time, etc. In specific implementation, the first abnormal condition may be set based on experience, and is not specifically limited in the embodiment of the present disclosure.

[0072] In a specific implementation, an outlier filtering algorithm can be used to perform anomaly filtering on the data delay time corresponding to each data partition, and determine whether there is a target delay time that meets the first abnormality condition. If so, the data partition corresponding to the target delay time can be used as the data partition to be filtered, and the following S104-3-2 can be performed. The determined number of data partitions to be filtered may be one or more. If not, it can be considered as a normal delay of each first subtask, and there is no problem of slow execution speed.

[0073] For example, if the data latency corresponding to data partitions 1 to 10 are 10, 10, 100, 1000, and 10000, respectively, the target latency can be determined to be 100, 1000, and 10000, and the data partitions corresponding to these three target latency times can be used as the data partitions to be filtered.

[0074] S104-3-2: If so, determine a second subtask based on the queries per second of the first subtask that depends on the data partition to be filtered.

[0075] Here, Queries-per-second, or QPS for short, is used to indicate the amount of task traffic processed in one second by the machine executing the task.

[0076] Specifically, when it is implemented, after it is determined that there is a data partition to be filtered that has an abnormal data latency, it determines the incoming QPS of a first subtask that uses the stream data of the data partition to be filtered, and then it can be determined whether the incoming QPS is equal to or greater than a predetermined multiple of the average QPS of the first subtask. If not, it uses the first subtask with the incoming QPS as the second subtask, and if so, it can be determined that the partition lag of the first subtask with the incoming QPS is caused by data imbalance, and the latency problem cannot be solved by migrating the machine, so it determines the first subtask as a normally running subtask and does not process it.

[0077] In this way, the subtasks with large delays can be determined by filtering outliers in data delay time, and the slow node machines with slow node problems can be accurately determined based on the subtasks with large delays.

[0078] In another embodiment, for the step of “determining the second subtask” in S104, if the target operator type belongs to an operator type other than the data consumption type, the second subtask is determined according to the detection method of the slow node shown in step 1 and step 2.

[0079] Step 1: if the target operator type belongs to an operator type other than a data-consuming type, determine whether there is a third subtask whose task idle state satisfies a second abnormal condition based on the task idle state indicated by the index information of the detection target of each first subtask.

[0080] Here, when the target operator type does not belong to the source operator type, regardless of which operator type the target operator type belongs to among other operator types other than the source operator type, all of the index information to be detected by the first subtask may be an idle state index of each first subtask.

[0081] The second abnormal condition is used to filter the abnormally busy third subtask from the first subtask, and exemplarily, the second abnormal condition may be that the idle state of the task is less than a first predetermined degree, or the difference between the idle state of each first subtask and the median value is greater than a second predetermined degree, etc. In specific implementation, the second abnormal condition can be set based on experience, and is not specifically limited in the embodiment of the present disclosure.

[0082] In specific implementation, if the target operator type does not belong to the source operator type, an outlier filtering algorithm can be used to determine whether there is a third subtask whose idle state meets the second abnormal condition based on the task idle state indicated by the detection target index information of each first subtask. If so, it can be determined that there is an abnormally busy third subtask, and step 2 can be performed based on the determined third subtask.

[0083] Otherwise, each of the first subtasks may be relatively idle, the backpressure of each subtask is within a normal range, each of the first subtasks may be considered a normally running subtask, and there is no slow node issue, so no processing may be required.

[0084] Step 2, if yes, determine the second subtask based on the queries per second of the third subtask and the preset query rate.

[0085] Here, the preset query rate may be a preset multiple of the average QPS of the third subtask, and the preset multiple may be set based on experience, and is not particularly limited here. For example, the preset multiple may be 1.5.

[0086] In a specific implementation, after determining that there is an abnormally busy third subtask, the incoming QPS of the third subtask can be determined, and whether the incoming QPS is equal to or greater than a preset query rate can be determined. If so, it can be determined that the third subtask is busy due to data skew, and machine migration cannot solve the delay problem, and therefore the third subtask can be determined to be a normally running subtask, and if not, it can be determined that the third subtask is a slow-running second subtask.

[0087] In this way, based on filtering outliers of the idleness of the tasks, the abnormally busy third subtasks can be determined, and based on the third subtasks, the slow node machines having the slow node problem can be accurately determined.

[0088] As shown in FIG. 2, a specific flowchart for determining a slow-running second subtask provided by an embodiment of the present disclosure is shown, which may include the following steps:

[0089] S201: Based on the data delay indicator of a Flink task, determine whether the Flink task has a very large delay.

[0090] If not, it can be determined that there is no slow node problem, and if so, the next steps from S202 onwards are executed.

[0091] S202: Determine whether there is backpressure on the subtasks of each operator type.

[0092] If not, execute S203 to S205, and if yes, execute S206 to S208.

[0093] S203: Determine whether the ratio value between the delayed data partition and the total number of partitions is less than a third predetermined ratio value.

[0094] If not, it can be determined that the first subtask is delayed extensively and there is no slow node problem, and if so, execute the next S204.

[0095] S204: It is determined whether there is a data partition to be filtered that has an abnormal data latency time.

[0096] If not, it can be determined that the first subtask is normally delayed and there is no slow node problem, and if so, execute the next S205.

[0097] S205: Determine whether the queries per second QPS of the first subtask corresponding to the data partition to be filtered is greater than or equal to a predetermined multiple of the average QPS.

[0098] If not, it can be determined that the first subtask is the slower executing second subtask, and if not, it can be determined that the delay is caused by data imbalance and can be left unprocessed.

[0099] For specific implementation steps of the above S203 to S205, reference can be made to the above S104-1 to S104-3, and the description will not be repeated here.

[0100] S206: Determine a first subtask of the target operator type based on the topological relationships between various operator types.

[0101] For the specific implementation process of this step, please refer to S102, and it will not be described here repeatedly.

[0102] S207: It is determined whether or not there is an abnormally busy third subtask in the first subtask.

[0103] If not, it can be determined that the first subtask is in a normal backpressure state, and no processing is performed. If so, the next step S208 is executed.

[0104] S208: Determine whether the queries per second QPS of the third subtask is greater than or equal to a predetermined multiple of the average QPS.

[0105] If so, it can be determined that data skew exists and no processing is performed; if not, the third subtask is determined as the slower executing second subtask.

[0106] For the specific implementation steps of the above S207 to S208, reference can be made to the above steps 1 and 2, and the description will not be repeated here.

[0107] In one embodiment, the step of S104 "determine slow node machines having slow node problems" can be implemented according to the following steps.

[0108] S1: If the target number of the second subtasks is greater than a preset number, determine the number of tasks corresponding to each execution machine based on the execution machine on which the computing node executing each second subtask resides.

[0109] Here, the target number is the total number of the determined second subtasks. Here, the preset number can be set based on experience, and is not specifically limited in the embodiment of the present disclosure. Exemplarily, the preset number may be 1. The number of tasks corresponding to one execution machine is the total number of second subtasks executed by the execution machine.

[0110] In specific implementation, after determining the second subtasks, it can first be determined whether the number of the second subtasks is greater than 1, and if not, the execution machine on which the computation node executing the second subtask exists is used as the slow node machine. If so, the number of tasks corresponding to each execution machine is determined according to the execution machine on which the computation node executing each of the multiple second subtasks exists.

[0111] Optionally, it may further determine whether a no-aggregation migration function is set when the number of second subtasks is one, which migrates a machine corresponding to the only second subtask when the number of second subtasks is one, or migrates a machine when there are multiple second subtasks even if the determined second subtasks do not exist many times on one machine. If the no-aggregation migration function is set, it may determine that an execution machine on which a TM executing the only second subtask is located is a slow node machine. If the no-aggregation migration function is not set, it may determine that there is no slow node machine, and do not process the slow-executing second subtask.

[0112] S2: Determine a slow node machine based on a first ratio value between the number of tasks corresponding to each execution machine and the target number.

[0113] In particular, for each execution machine for executing the second subtask, a first ratio value between the number of tasks and the target number corresponding to the execution machine can be determined. After obtaining the first ratio value corresponding to each execution machine, a slow node machine can be determined based on each first ratio value. For example, the execution machine corresponding to the largest first ratio value can be used as the slow node machine.

[0114] In one embodiment, after obtaining the first ratio value corresponding to each execution machine, the magnitude relationship between the first ratio value and the first predetermined ratio value can be determined, and the slow node machine can be determined in various ways. Here, the first predetermined ratio value can be set based on experience, and is not specifically limited in the embodiment of the present disclosure. For example, the first predetermined ratio value may be 2 / 3.

[0115] Specifically, if there is a target ratio value that is equal to or greater than the first preset ratio value, the execution machine corresponding to the target ratio value is used as the slow node machine.

[0116] Here, the target ratio value may be a first ratio value equal to or greater than a first preset ratio value. Specifically, when there is a target ratio value, the execution machine corresponding to the target ratio value can be used as a slow node machine. In this way, since the target ratio value is equal to or greater than the first preset ratio value, the execution machine corresponding to the target ratio value is a machine having many second subtasks with slow execution speeds, and the machine is treated as a slow node machine and migrated using the next step, thereby improving the rationality of machine migration and the execution efficiency of multiple second subtasks.

[0117] Alternatively, when each of the first ratio values ​​is smaller than a first preset ratio value, the execution machine corresponding to the second subtask with the largest index anomaly degree is used as the slow node machine based on the index anomaly degree indicated by the index information of the detection target of each second subtask.

[0118] Here, the index anomaly degree may be the anomaly degree of the index value in the index information of the detection target. For example, when the target operator type is a source operator type, the index anomaly degree indicated by the index information of the detection target may reflect the length of the data delay time corresponding to each data partition. When the source operator type does not belong to the source operator type, the index anomaly degree indicated by the index information of the detection target may reflect the degree of the task idle state of each first subtask.

[0119] In a specific implementation, if each first ratio value is smaller than the first preset ratio value, that is, if there is no first ratio value equal to or greater than the first preset ratio value, it indicates that there is no machine aggregation (i.e., the target number of second subtasks are not gathered in large numbers on the same machine) for the target number of second subtasks. In this case, based on the index anomaly degree indicated by the index information of the detection target of each second subtask, the execution machine corresponding to the second subtask with the largest index anomaly degree is used as the slow node machine. For example, if the second subtask is a source operator type subtask, the execution machine in which the second subtask corresponding to the data partition with the longest data latency exists can be used as the slow node machine. If the second subtask does not belong to a source operator type subtask, the execution machine in which the second subtask with the least task idle state exists can be used as the slow node machine. In this way, through the machine aggregation and outlier filtering algorithm, it is possible to filter out situations with slow execution speeds other than machine engines, and the accuracy of the determined slow execution speed second subtasks can be improved.

[0120] Optionally, if each of the first ratio values ​​is smaller than a first predetermined ratio value, it can also be determined whether a no-aggregation transition function is set, and if so, use the execution machine corresponding to the second subtask with the largest index abnormality degree as a slow node machine; if not, if the second subtasks are not aggregated in large numbers on the same machine, it can be determined that the slow execution speed of the task is within a reasonable range, and it can be determined that there is no slow node machine, and each subtask with a slow execution speed is not processed.

[0121] In one embodiment, S1 can be performed according to the following steps:

[0122] S1-1: If the target operator type belongs to a data consumption type, based on each data partition on which each second subtask relies when reading data from a message queue, determine a data delay time corresponding to each data partition from the index information of the detection target corresponding to the second subtask.

[0123] In specific implementation, before executing S1, it can be determined first whether the target operator type belongs to a data consumption type, and if so, first execute S1-1 to S1-4 in sequence and then execute S2; if not, S1 and S2 can be executed directly, and in this case, S1 does not include S1-1 to S1-4.

[0124] The specific implementation process of S1-1 is similar to the implementation process of S104-1 described above, and is not described here repeatedly. For example, when the target operator type belongs to the source operator type, the partition lag corresponding to the data partition on which each second subtask depends can be obtained from the index information of the detection target.

[0125] S1-2: Filter multiple target data partitions from the data partitions based on data delay time.

[0126] In particular, the partition lag of each data partition can be used to sort each data partition from the longest time to the shortest time to obtain the sort order of each data partition. Then, the data partition whose sort order is smaller than the preset order can be used as the target data partition. In this way, multiple partitions with longer partition lag can be filtered from the data partition. For example, the preset order can be 11, that is, each of the top 10 data partitions with partition lag can be filtered.

[0127] It can be understood that if the total number of data partitions is less than the value corresponding to the preset order, each data partition can be used as the target data partition.

[0128] S1-3: Based on the data management server to which each target data partition belongs, the number of data partitions corresponding to each data management server is determined.

[0129] Here, different data partitions can be managed by different data management servers, and the data management servers are each broker corresponding to a message queue. The number of data partitions is the number of target data partitions managed by each broker.

[0130] When specifically implemented, after filtering each target data partition, the data management server to which each target data partition belongs can be determined, and then the number of target data partitions (i.e., the number of data partitions) managed by each data management server can be further determined.

[0131] S1-4: If the second ratio values ​​between the number of data partitions corresponding to each data management server and the first number of target data partitions are both smaller than a second preset ratio value, determine the number of the tasks corresponding to each execution machine based on the execution machine on which the computing node of each second subtask exists.

[0132] Here, the second preset ratio value can be set based on experience, and is not specifically limited in the embodiment of the present disclosure. For example, the second preset ratio value may be 2 / 3. The first number is the number of target data partitions.

[0133] When specifically implemented, the second ratio value corresponding to each data management server can be determined according to the number of data partitions corresponding to each data management server and the first number. Then, determine whether there is a second ratio value greater than the second predetermined ratio value. If so, the slow execution of the Flink task may not be caused by the machine, but may be caused by a problem of the data management server itself corresponding to the second ratio value equal to or greater than the second predetermined ratio value, and therefore, it can be determined that there is no slow node machine, and do not process the second subtask. If not, that is, if each second ratio value is smaller than the second predetermined ratio value, it indicates that the reason for the slow execution of the Flink task at present is not caused by a problem of the data management server itself, but is caused by the machine, and therefore, the slow node machine can be determined based on the first ratio value between the number of tasks and the target number corresponding to each execution machine, and the magnitude relationship between the first ratio value and the first preset ratio value.

[0134] In this way, by using the second ratio value corresponding to the data management server, the problem of slow task execution caused by a malfunction of the data management server itself can be eliminated, and the accuracy of determining slow nodes can be improved.

[0135] As shown in FIG. 3 , a specific flowchart for determining a slow node machine provided by an embodiment of the present disclosure is shown, which may include the following steps:

[0136] S301: Determine whether the number of second subtasks is greater than one.

[0137] If not, execute S302; if yes, execute S303.

[0138] S302: It is determined whether or not a transition without aggregation function is set.

[0139] If so, the execution machine on which the second subtask exists is used as the slow node machine; if not, it is determined that there is no slow node machine, and is not processed.

[0140] S303: Determine whether the target operator type belongs to a data consumption type.

[0141] If so, execute S304 to S306; if not, execute S307.

[0142] S304: Filter multiple target data partitions from the data partitions on which each second subtask depends.

[0143] For the specific implementation process of this step, please refer to S1-1 and S1-2 above.

[0144] S305: Determine whether a second ratio value of the target data partition associated with the same data management server is greater than or equal to a second predetermined ratio value.

[0145] If so, it is determined that there is no slow node machine and no processing is performed; otherwise, execute S306.

[0146] For the specific implementation process of this step, please refer to S1-1 and S1-4.

[0147] S306: Determine whether a first ratio value of the number of second subtasks executed by the same execution machine is greater than or equal to a first predetermined ratio value.

[0148] If so, determine that machine aggregation exists, and use the execution machine on which a large number of second subtasks are aggregated as the slow node machine; otherwise, if the no-aggregation transition function is set, use the execution machine corresponding to the second subtask with the longest data latency as the slow node machine.

[0149] S307: Determine whether a first ratio value of the number of second subtasks executed by the same execution machine is greater than or equal to a first predetermined ratio value.

[0150] If so, it determines that machine aggregation exists, and uses the execution machine on which a large number of second subtasks are aggregated as the slow node machine; otherwise, if the no-aggregation transition function is set, it uses the execution machine corresponding to the least idle second subtask of the task as the slow node machine.

[0151] For the specific implementation steps of the above S301 to S307, reference can be made to the above-mentioned embodiments, and the description will not be repeated here.

[0152] In one embodiment, after determining the slow node machine, the slow node machine can be migrated to solve the slow node problem. Specifically, the machine migration can be performed according to the following steps:

[0153] P1: Using a pre-configured service migration interface, pre-apply for target resources for each target computing node based on the amount of resources required for each target computing node to execute the second subtask in the slow node machine.

[0154] Here, the service migration interface may be a pre-developed API in the Flink engine that has the ability to call a machine migration function. The machine migration function can migrate the TM that executes the second subtask to a new machine, i.e., the second subtask that runs slowly on a slow node machine that has a slow node problem can be migrated to a new machine for execution.

[0155] It should be noted that the machine migration function is not a function already present in the conventional technology, but a function specially developed to solve the problem of slow nodes, which not only realizes machine migration, but also has optimization functions such as "blocking + resource pre-application" to optimize migration work. Blocking, hereinafter, means setting blocking information, and resource pre-application, hereinafter, means applying for target resources in advance.

[0156] The target computation node is a computation node for executing the second subtask, which has a slower execution speed, in the slow node machine, that is, a TM for executing the second subtask in the slow node machine.

[0157] The amount of resources required for the target computing node refers to the container resources required to run one target computing node, such as Pod container resources or Container container resources in K8s, and the target resources are container resources that have been applied for and obtained in advance.

[0158] In specific implementation, the machine migration function is invoked through the service migration interface to first determine the amount of resources required for each target computing node, and then, based on the resource amount, a corresponding amount of container resources can be pre-applied for in K8s for each target computing node.

[0159] P2: Set blocking information for a slow node machine, which is used to indicate that new subtasks of the target operator type cannot be scheduled to be executed by the computation nodes of the slow node machine.

[0160] Here, after deciding to set the blocking information, the new subtask can newly generate a subtask having the target operator type. For example, a new source operator type subtask and a new sink operator type subtask. The blocking information is used to indicate that the new subtask of the target operator type cannot be scheduled to be executed by the computation node of the slow node machine. That is, the blocking information instructs the slow node machine to block the subtask of the target operator type under the Flink task, and ensures that the subtask of the new target operator type will not be scheduled to the slow node machine after it is generated, and the Flink task for slow node detection will not be scheduled to the slow node machine.

[0161] Optionally, the blocking information may include a blocking period during which new subtasks of the target operator type cannot be scheduled to be executed by the computation nodes of the slow node machine, after which new subtasks of the target operator type may again be scheduled to be executed on the computation nodes of the slow node machine.

[0162] In particular, after determining the slow node machine, the migration function of the called machine can be used to set the blocking information of the Flink task currently performing slow node detection and the new subtask of the target operator type under the Flink task to the slow node machine. In this way, after detecting the slow node machine corresponding to the Flink task, the slow node machine can be blocked, and the slow node machine can be prevented from being called again to perform the Flink task during the blocking period.

[0163] It can be understood that there is no strict execution order for P1 and P2 above and they can be executed synchronously or asynchronously.

[0164] In addition, in the case of a slow node machine, in addition to the tasks related to the blocking information not being scheduled to be executed by the slow node machine, the second subtask, which is executing slowly on the slow node machine, is migrated to and executed by the alternative node machine, and other engine type tasks (tasks other than the second subtask) already being executed by the slow node machine are not affected and can continue to be executed by the slow node machine before the slow node machine goes offline.

[0165] P3: Using the requested target resources, create an alternative computing node to substitute for the target computing node, and use the alternative computing node to execute a second subtask related to the target computing node.

[0166] In particular, when the target resource pre-application is successful, when a new machine comes online, the invoked machine migration function is used to create each alternative computing node in the newly online machine to replace each target computing node using the applied target resource. Furthermore, based on the failover mechanism of the Flink engine, the alternative computing node can be used to resume the execution of the second subtask related to the target computing node, and at the same time, the alternative computing node can be used to execute a new subtask of the target operator type. In addition, each target computing node in the slow node machine can be released, and the specific release step is to request the JM side that manages the TM to realize the resource release of the target computing node by the JM using the resource release interface developed on the target computing node (TM or Container).

[0167] In this way, using the service migration interface, not only can the entire machine migration process be performed asynchronously within the service cluster, but the target resource pre-application method can also be used to realize pre-application of TM resources in the machine that needs to be migrated, and finally, based on the failover mechanism and the pre-application of the target resources, the machine is migrated, and the migration is usually completed in a few seconds, which greatly improves the migration speed and stability of the slow node and improves the efficiency of solving the problem of slow nodes.

[0168] Optionally, after invoking the machine migration function using the service migration interface, the machine migration work can be directly performed to migrate the slow-running second subtask to a new machine for execution without setting blocking information and without pre-application of resources. The specific process is as follows: when the new machine is online, the machine migration function is used to apply for target resources in the new machine, an alternative execution node is created based on the applied target resources, and then, based on the failover mechanism, the execution of the second subtask related to the target computing node is resumed and a new subtask with the target operator type is executed. At the same time, after the machine migration is completed, the resource release interface can be used to request the JM side managing the TM to realize the resource release of the target computing node by the JM. In this way, the second subtask executed on the slow node machine can be migrated to a new machine for execution using the machine migration function without using technical means such as setting blocking information and pre-application of resources, so that the problem of the slow node can be solved. At the same time, in the process of performing machine migration using the machine migration function, using blocking information and pre-application of resources as auxiliary means can improve the migration efficiency and further improve the migration effect.

[0169] As shown in FIG. 4, it is an interaction schematic diagram of the slow node detection method provided by the embodiment of the present disclosure. In FIG. 4, three different types of Flink tasks are shown, and the resource cluster may be a K8s cluster or Yarn. The service cluster of each Flink task can send and store the execution indicator information generated during the execution of the Flink task to the indicator database. Specifically, the slow node auto-detection service can obtain the execution indicator information of the Flink task to be detected from the indicator database. Then, based on the execution indicator information, a slow node analysis can be performed to determine the slow node machine with the slow node problem. The analysis of the slow node can include: 1, using an outlier filtering algorithm to determine the second subtask with a slow execution speed; and 2, based on the machine aggregation, determining the slow node machine. For the specific implementation process of the slow node analysis step, reference can be made to the above embodiments. After obtaining the slow node machine, a blocking operation and a migration operation can be performed, and the blocking operation can generate blocking information for the slow node machine, and the migration operation can migrate the slow node machine based on the pre-application of target resources and the failover mechanism. The blocking operation is used to optimize the migration work and improve the migration speed by pre-application of target resources without calling and executing the Flink task on the slow node machine in a short period of time. Optionally, the slow node machine can also be migrated without pre-application of target resources and / or setting blocking information. For the specific implementation process of the blocking and migration operations, please refer to the above P1 to P3, and will not be repeated here.

[0170] Those skilled in the art will understand that in the above-described methods of the specific embodiments, the described order of each step is not a strict execution order, and does not constitute any restriction on the implementation process, and the specific execution order of each step is determined by its function and possible internal logic.

[0171] Based on a similar inventive concept, an embodiment of the present disclosure further provides an apparatus for detecting a slow node during task execution, which corresponds to the method for detecting a slow node during task execution, and since the problem-solving principle of the apparatus in the embodiment of the present disclosure is similar to the method for detecting a slow node during task execution described above in the embodiment of the present disclosure, the implementation of the apparatus can be referred to, and repeated details will be omitted.

[0172] FIG. 5 is a schematic diagram of a slow node detection device during task execution provided by an embodiment of the present disclosure, the device including: An acquisition module 501 used to acquire performance indicator information of a stream data processing task including subtasks of multiple operator types; A first determination module 502, which is used to determine a target operator type having an abnormality according to the topology relationship of various operator types and the abnormality cause information indicated by the performance indicator information; a second determination module 503, which is used to filter each first subtask of the target operator type from the subtasks of the plurality of operator types, and determine, from the performance indicator information, indicator information of detection targets corresponding to each first subtask respectively; A third determination module 504 is used to determine a second subtask having a slow execution speed from the first subtask using the index information of the detection target, and to determine a slow node machine based on an execution machine in which a computing node executing the second subtask exists; Includes.

[0173] In a possible embodiment, the device comprises: a migration module 505; After determining the slow node machine, the migration module 505: pre-requesting target resources for each target computing node in the slow node machine based on an amount of resources required for each target computing node to execute a second subtask using a pre-configured service migration interface; setting blocking information for the slow node machine, the blocking information being used to indicate that new subtasks of the target operator type cannot be scheduled to be executed by a compute node of the slow node machine; creating an alternative computing node to substitute for the target computing node using the requested target resources, and using the alternative computing node to execute the second subtask associated with the target computing node; It is used for this purpose.

[0174] In a possible embodiment, the third determination module 504, when determining a slow node machine based on an execution machine in which a computing node executing the second subtask is present, If the target number of the second subtasks is greater than a preset number, determine the number of tasks corresponding to each execution machine based on an execution machine on which a computing node executing each of the second subtasks exists; determining the slow node machine based on a first ratio value between the number of tasks corresponding to each of the execution machines and the target number; It is used for this purpose.

[0175] In a possible embodiment, the third determination module 504, when determining the slow node machine, based on a first ratio value between the number of tasks corresponding to each of the execution machines and the target number, If there is a target ratio value that is equal to or greater than the first preset ratio value, use the execution machine that corresponds to the target ratio value as the slow node machine; or When each of the first ratio values ​​is smaller than the first preset ratio value, an execution machine corresponding to a second subtask having the largest index anomaly degree is used as the slow node machine based on an index anomaly degree indicated by the index information of the detection target of each of the second subtasks. It is used for this purpose.

[0176] In a possible embodiment, the third determination module 504, when determining the number of tasks corresponding to each of the execution machines based on the execution machines on which the computing nodes executing each of the second subtasks are present, When the target operator type belongs to a data consumption type, determine a data delay time corresponding to each of the data partitions from the index information of the detection target corresponding to the second subtask according to each data partition that each of the second subtasks depends on when reading data from the message queue; filtering a plurality of target data partitions from the data partitions based on the data latency; Determine a number of data partitions corresponding to each of the data management servers based on a data management server to which each of the target data partitions belongs; When the second ratio value between the number of the data partitions corresponding to each of the data management servers and the first number of the target data partitions is smaller than a second preset ratio value, determine the number of the tasks corresponding to each execution machine based on an execution machine on which a computing node executing each of the second sub-tasks exists; It is used for this purpose.

[0177] In a possible embodiment, the third determination module 504, when determining a second subtask having a slow execution speed from the first subtask using the index information of the detection target, If the target operator type belongs to a data consumption type, determine a data delay time corresponding to each of the data partitions from the index information of the detection target corresponding to the first subtask based on each data partition that each of the first subtasks depends on when reading data from a message queue; determining a second number of data partitions in response to the data latency, the second number being greater than the predetermined data latency; If a third ratio value between the second number and a total number of partitions is smaller than a third preset ratio value, determining the second subtask from the first subtask based on the data delay time corresponding to each of the data partitions, wherein the total number of partitions is the total number of data partitions on which each of the first subtasks depends; It is used for this purpose.

[0178] In a possible embodiment, when the third determination module 504 determines the second subtask from the first subtask based on the data delay time corresponding to each of the data partitions, the third determination module 504: Determine whether or not there is a filtering target data partition whose data latency time satisfies a first abnormality condition in each of the data partitions based on the data latency time corresponding to each of the data partitions; if so, determining the second subtask based on queries per second of the first subtask that depends on the data partition being filtered; It is used for this purpose.

[0179] In a possible embodiment, the third determination module 504, when determining a second subtask having a slow execution speed from the first subtask using the index information of the detection target, When the target operator type belongs to an operator type other than a data consuming type, determine whether or not there is a third subtask whose task idle state satisfies a second abnormality condition based on the task idle state indicated by the detection target index information of each of the first subtasks; if so, determining the second subtask based on the queries per second of the third subtask and a preset query rate; It is used for this purpose.

[0180] For the description of the processing flow of each module in the apparatus and the interaction flow between each module, please refer to the relevant description in the above method embodiment, and will not be described in detail here.

[0181] Based on the same technical concept, the embodiment of the present application further provides an electronic device. As shown in FIG. 6, a schematic structural diagram of the electronic device provided by the embodiment of the present application is shown, which includes: The system includes a processor 601, a memory 602, and a bus 603. The memory 602 stores machine-readable instructions executable by the processor 601, and the processor 601 is used to execute the machine-readable instructions stored in the memory 602. When the machine-readable instructions are executed by the processor 601, the processor 601 executes the following steps: S101, acquiring execution indicator information of a stream data processing task including subtasks of a plurality of operator types; S102, determining a target operator type having an abnormality based on the topology relationship of various operator types and abnormality cause information indicated by the execution indicator information; S103, filtering each first subtask of the target operator type from the subtasks of the plurality of operator types, and determining detection target indicator information corresponding to each of the first subtasks from the execution indicator information; and S104, determining a second subtask having a slow execution speed from the first subtask using the detection target indicator information, and determining a slow node machine based on an execution machine in which a computation node executing the second subtask exists.

[0182] The memory 602 includes an internal memory 6021 and an external memory 6022, where the internal memory 6021 is also called an internal storage and is used to temporarily store calculation data in the processor 601 and data exchanged with an external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 via the internal memory 6021, and during operation of the electronic device, the processor 601 and the memory 602 communicate with each other via a bus 603, whereby the processor 601 executes the execution instructions described in the above method embodiments.

[0183] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method for detecting a slow node during task execution described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0184] A computer program product of a method for detecting a slow node during task execution provided by an embodiment of the present disclosure includes a computer-readable storage medium storing program code, and instructions contained in the program code perform steps of the method for detecting a slow node during task execution described in the above method embodiment; specifically, reference may be made to the above method embodiment, and the description will not be repeated here.

[0185] The computer program product can be specifically realized through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is realized as a computer storage medium, and in another alternative embodiment, the computer program product is realized as a software product such as a Software Development Kit (SDK).

[0186] Those skilled in the art can clearly understand that for convenience and simplicity of description, the specific operation processes of the above-mentioned system and device can refer to the corresponding processes in the above-mentioned method embodiments, and will not be described repeatedly here. In some embodiments provided by the present disclosure, it should be understood that the disclosed system, device, and method can be realized in other ways. The above-mentioned device embodiments are merely examples, for example, the division of the units is merely a logical functional division, and there may be other division methods when actually realized, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not executed. Meanwhile, the coupling or direct coupling or communication connection between each other shown or described may be an indirect coupling or communication connection via some communication interfaces, devices or units, and may be in an electrical, mechanical, or other form.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solution of this embodiment.

[0188] Furthermore, each functional unit in each embodiment of the present disclosure may be integrated into a single processing unit, each unit may exist physically alone, or two or more units may be integrated into a single unit.

[0189] When the function is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure can essentially realize a part that contributes to the prior art or a part of the technical solution in the form of a software product, and the computer software product includes some instructions stored in a storage medium to make an electronic device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method according to each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program code, such as U disk, mobile hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk.

[0190] Where the technical solution of the present application involves personal information, the product using the technical solution of the present application has clearly notified the personal information processing rules and obtained the individual's independent consent before processing the personal information. Where the technical solution of the present application involves sensitive personal information, the product using the technical solution of the present application has obtained individual consent before processing the sensitive personal information, and at the same time has also met the requirement of "explicit consent". For example, a clear and conspicuous sign is set on a personal information collection device such as a camera to inform the individual that the personal information will be collected by entering the personal information collection range, and when the individual voluntarily enters the collection range, the individual is deemed to have consented to the collection of the personal information; or, when a conspicuous logo / information is used to inform the personal information processing rules on the device processing the personal information, the individual's permission is obtained through a pop-up message or by asking the individual to upload personal information, where the personal information processing rules include the personal information controller, the personal information processing purpose, the processing method, the type of personal information processed, etc.

[0191] Finally, the above examples are merely specific embodiments of the present disclosure, do not limit the technical solutions of the present disclosure, are used to explain the technical solutions of the present disclosure, and the protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above examples, those skilled in the art can still easily think of modifications or changes to the technical solutions recorded in the above examples within the technical scope disclosed herein, or replace some of the technical features with equivalents, and these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the examples of the present disclosure, and are included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. obtaining performance indicator information for a stream data processing task including a plurality of operator type subtasks; determining a target operator type having an abnormality according to the topology relationship of various operator types and the abnormality cause information indicated by the execution index information; filtering each first subtask of the target operator type from the subtasks of the plurality of operator types; and determining, from the performance indicator information, indicator information of a detection target corresponding to each first subtask; using index information of the detection target to determine a second subtask having a slow execution speed from the first subtask, and determining a slow node machine based on an execution machine in which a computation node executing the second subtask exists.

2. After determining the slow node machine, pre-requesting target resources for each target computing node in the slow node machine based on an amount of resources required for each target computing node to execute the second subtask, using a pre-configured service migration interface; setting blocking information for the slow node machine, the blocking information being used to indicate that new subtasks of the target operator type cannot be scheduled to be executed by a compute node of the slow node machine; and creating an alternative computing node to substitute for the target computing node using the requested target resources, and using the alternative computing node to execute the second subtask associated with the target computing node.

3. The step of determining a slow node machine based on an execution machine in which a computing node that executes the second subtask exists, If the target number of the second subtasks is greater than a preset number, determining the number of tasks corresponding to each execution machine based on an execution machine on which a computing node executing each of the second subtasks exists; 2. The method of claim 1, further comprising: determining the slow node machine based on a first ratio value between the number of tasks corresponding to each of the execution machines and the target number.

4. The step of determining the slow node machine based on a first ratio value between the number of tasks corresponding to each of the execution machines and the target number, If there is a target ratio value that is equal to or greater than the first preset ratio value, use the execution machine that corresponds to the target ratio value as the slow node machine; or The method according to claim 3, further comprising a step of: when each of the first ratio values ​​is smaller than the first predetermined ratio value, using as the slow node machine an execution machine corresponding to the second subtask having the largest index anomaly degree based on the index anomaly degree indicated by the index information of the detection target of each of the second subtasks.

5. The step of determining the number of tasks corresponding to each of the execution machines based on the execution machines on which the computing nodes executing each of the second subtasks are present includes: When the target operator type belongs to a data consumption type, determining a data delay time corresponding to each of the data partitions from the index information of the detection target corresponding to the second subtask based on each data partition that each of the second subtasks depends on when reading data from a message queue; filtering a plurality of target data partitions from the data partitions based on the data delay time; determining a number of data partitions corresponding to each of the data management servers based on a data management server to which each of the target data partitions belongs; 4. The method of claim 3, further comprising: determining the number of tasks corresponding to each of the execution machines based on an execution machine on which a computing node of each of the second subtasks exists when a second ratio value between the number of data partitions corresponding to each of the data management servers and the first number of the target data partitions is smaller than a second preset ratio value.

6. The step of determining a second subtask having a slow execution speed from the first subtask by using the index information of the detection target includes: When the target operator type belongs to a data consumption type, determining a data delay time corresponding to each of the data partitions from the index information of the detection target corresponding to the first subtask based on each data partition that each of the first subtasks depends on when reading data from a message queue; determining a second number of data partitions, the second number being greater than a predetermined data latency, based on the data latency; 2. The method of claim 1, further comprising: determining the second subtask from the first subtask based on the data delay time corresponding to each of the data partitions when a third ratio value between the second number and a total number of partitions is less than a third preset ratio value, wherein the total number of partitions is a total number of data partitions on which each of the first subtasks depends.

7. determining the second subtask from the first subtask based on the data delay time corresponding to each of the data partitions, determining whether or not there is a filtering target data partition whose data latency time satisfies a first abnormality condition in each of the data partitions based on the data latency time corresponding to each of the data partitions; and if so, determining the second subtask based on queries per second of the first subtask that depends on the data partition being filtered.

8. The step of determining a second subtask having a slow execution speed from the first subtask by using the index information of the detection target includes: When the target operator type belongs to an operator type other than a data-consuming type, determining whether or not there is a third subtask whose task idle state satisfies a second abnormality condition based on the task idle state indicated by the detection target index information of each of the first subtasks; and if so, determining the second subtask based on queries per second of the third subtask and a preset query rate.

9. An acquisition module for acquiring performance indicator information of a stream data processing task including a plurality of operator type subtasks; A first determination module for determining a target operator type having an abnormality according to the topology relationship of various operator types and the abnormality cause information indicated by the performance indicator information; a second determination module for filtering each first subtask of the target operator type from the subtasks of the plurality of operator types, and determining, from the performance indicator information, indicator information of a detection target corresponding to each first subtask; and a third determination module for determining a second subtask having a slow execution speed from the first subtask using index information of the detection target, and for determining a slow node machine based on an execution machine in which a computation node executing the second subtask is located.

10. An electronic device comprising a processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor executing the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the processor performs steps of a method for detecting slow nodes during task execution according to any one of claims 1 to 8.

11. A computer-readable storage medium, the computer-readable storage medium storing a computer program, the computer-readable storage medium being characterized in that, when the computer program is executed by an electronic device, the electronic device executes the steps of the method for detecting slow nodes during task execution described in any one of claims 1 to 8.

Images