Equipment exception detection method and device, storage medium and electronic equipment

Abstract

The invention discloses an equipment exception detection method and device, a storage medium and electronic equipment. The method comprises the following steps: determining a first task parameter for performing data interaction with an equipment disk according to first operation data corresponding to a target task; comparing the historical operation data corresponding to the historical tasks with the first operation data to determine a minimum task parameter, the task types of the target task and the historical tasks being the same, and the first operation data and the historical operation data satisfying a preset same-family condition; the minimum task parameter is used for indicating the disk operation rate of the disk of the specified equipment in the process of executing the target task; and under the condition that the first task parameter and the minimum task parameter meet a preset value condition, generating a disk exception prompt message. According to the invention, the technical problem of low detection accuracy of equipment abnormity is solved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for detecting device malfunctions, a storage medium, and an electronic device. Background Technology

[0002] In distributed computing frameworks like Spark, the accuracy of detecting device anomalies, especially disk I / O anomalies, is low. These anomalies are often subtle, typically manifesting only as extended task execution time or leaving vague anomaly records in logs, such as network connection failure logs due to disk I / O blocking. However, simply examining the execution time of a single task in isolation is insufficient to effectively identify I / O bottlenecks or anomalies, as task execution time is influenced by various factors, including data size, disk type and quantity, and network conditions, failing to provide a direct solution for accurate device anomaly detection.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method and apparatus for detecting equipment malfunctions, a storage medium, and an electronic device, to at least solve the technical problem of low detection accuracy of equipment malfunctions.

[0005] According to one aspect of the embodiments of this application, a method for detecting device anomalies is provided, comprising: determining a first task parameter for data interaction with a device disk based on first operation data corresponding to a target task, wherein the first task parameter is used to indicate the disk operation rate of the device disk during the execution of the target task; performing a comparison operation between historical operation data corresponding to multiple historical tasks and the first operation data to determine a minimum task parameter, wherein the target task and the multiple historical tasks have the same task type, the first operation data and the historical operation data satisfy a preset family condition, and the minimum task parameter is used to indicate the disk operation rate of the specified device disk during the execution of the target task; and generating a disk anomaly prompt message when a preset value condition is satisfied between the first task parameter and the minimum task parameter.

[0006] According to another aspect of the embodiments of this application, a device for detecting device anomalies is also provided, comprising: a determining module, configured to determine a first task parameter for data interaction with a device disk based on first operation data corresponding to a target task, wherein the first task parameter is used to indicate the disk operation rate of the device disk during the execution of the target task; an execution module, configured to perform a comparison operation between historical operation data corresponding to multiple historical tasks and the first operation data respectively, and determine a minimum task parameter, wherein the target task and the multiple historical tasks have the same task type, the first operation data and the historical operation data satisfy a preset family condition, and the minimum task parameter is used to indicate the disk operation rate of the specified device disk during the execution of the target task; and a generating module, configured to generate a disk anomaly prompt message when a preset value condition is met between the first task parameter and the minimum task parameter.

[0007] Optionally, the device is configured to perform a comparison operation between the historical operation data corresponding to each of the multiple historical tasks and the first operation data to determine the minimum task parameter by: obtaining the historical hash value of the historical operation data and the target hash value of the first operation data; if the historical hash value and the target hash value are the same, determining the historical operation data as the second operation data; determining a historical task parameter sequence based on the second operation data, and performing a target merging operation on the historical task parameter sequence to obtain the minimum task parameter, wherein one of the historical task parameter sequences is used to indicate the disk operation rate of the device disk during the execution of one of the historical tasks.

[0008] Optionally, the apparatus is further configured to: generate a first dependency tree based on the first operation data, wherein the first dependency tree includes a plurality of first task nodes, the first task nodes being used to interact with the device disk based on the first operation data; perform a concatenation operation on each of the first task nodes in the first dependency tree to obtain a second dependency tree; determine the corresponding initial hash value of each of the second task nodes in the second dependency tree based on the attribute value of each of the second task nodes; perform a sorting operation on the initial hash values ​​to obtain an attribute string, including: determining the position of the initial hash value of the corresponding second task node in the attribute string based on the connection edge relationship between each of the second task nodes in the second dependency tree and its adjacent task nodes; and determine the target hash value based on the attribute string.

[0009] Optionally, the device is configured to perform a concatenation operation on each of the first task nodes in the first dependency tree to obtain a second dependency tree by: determining a first hash value corresponding to each of the attribute values ​​of each first task node; sequentially determining each of the first task nodes as the current task node and performing the following steps until the traversal of the first task nodes is completed: concatenating the first hash value of the current task node, the first hash value of the task node preceding the current task node, and the first hash value of the task node following the current task node to obtain a second hash value; updating the attribute value of the current task node according to the second hash value to obtain a second task node; and determining the second dependency tree according to the second task node.

[0010] Optionally, the device is used to perform a sorting operation on the initial hash value to obtain an attribute string by: determining the position of the initial hash value of the corresponding second task node in the attribute string according to the connection edge relationship between each second task node in the second dependency tree and its adjacent task nodes.

[0011] Optionally, the apparatus is further configured to: obtain a first number of connection edges originating from the first child node and connecting to any second task node other than itself, and obtain a second number of connection edges originating from the second child node and connecting to any second task node other than itself, wherein the first child node and the second child node are different, and the second task node includes the first child node and the second child node; and, if the first number is 0 and the second number is not 0, determine that the initial hash value of the first child node is in a first position in the attribute string, and the initial hash value of the second child node is in a second position in the attribute string, wherein the first position precedes the second position.

[0012] Optionally, the apparatus is further configured to: generate a third dependency tree based on historical operation data corresponding to one of the plurality of historical tasks, wherein the tree structure of the first dependency tree and the tree structure of the third dependency tree are the same, the third dependency tree includes a plurality of third task nodes, the third task nodes being used to interact with the device disk based on one of the historical operation data; perform a concatenation operation on each of the third task nodes in the third dependency tree to obtain a fourth dependency tree; determine an initial hash value corresponding to each of the fourth task nodes in the fourth dependency tree based on the attribute value of each; perform a sorting operation on the initial hash values ​​to obtain an attribute string; and determine a historical hash value based on the attribute string.

[0013] Optionally, the device is configured to determine a historical task parameter sequence based on the second operation data in the following manner, and perform a target merging operation on the historical task parameter sequence to obtain the minimum task parameter: obtaining the first quartile and the third quartile in the historical task parameter sequence, wherein 25% of the task parameters in the historical task parameter sequence are less than the first quartile, and 25% of the task parameters in the historical task parameter sequence are greater than the third quartile; and determining the minimum task parameter based on the first quartile and the third quartile.

[0014] Optionally, the apparatus is further configured to: set environment variables according to the device address of the target device where the device disk is located before executing the target task; and set the environment variables in the container used to start the target task.

[0015] Optionally, the device is further configured to: obtain disk status data based on the event log of the target task after the target task is executed; and generate a disk anomaly message based on the disk status data if the first task parameter is less than the minimum task parameter.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described device malfunction detection method when running.

[0017] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the device malfunction detection method described above.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described device malfunction detection method through the computer program.

[0019] In this embodiment, a first task parameter for data interaction with the device disk is determined based on the first operation data corresponding to the target task. This parameter specifically reflects the disk operation rate of the device disk during the execution of the target task. Next, historical operation data corresponding to multiple historical tasks are compared with the first operation data to filter out multiple second operation data that meet preset family conditions, i.e., historical tasks with the same type and hash value as the target task. Then, based on these second operation data, a historical task parameter sequence is determined, where each historical task parameter also indicates the disk operation rate during the execution of the historical task. A target merging operation is performed on the historical task parameter sequence to calculate the minimum task parameter, i.e., the slowest disk operation rate threshold in the historical tasks. If the difference between the first task parameter (the disk operation rate of the target task) and the calculated minimum task parameter meets a preset value condition, it means that the disk operation rate of the target task is significantly lower than the lowest normal rate of historical tasks of the same type. At this time, a disk anomaly message is generated to warn of possible disk I / O anomalies. In other words, by comparing and analyzing the disk operation rates of the target task and historical tasks, the purpose of accurately identifying device disk I / O anomalies is achieved, thereby improving the technical effect of improving the accuracy of device anomaly detection. It can promptly and accurately identify task execution anomalies caused by disk problems, thereby solving the technical problem of low detection accuracy of device anomalies. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram of the application environment of an optional device anomaly detection method according to an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating an optional device anomaly detection method according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of an optional node according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of an optional device anomaly detection method according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of another optional device malfunction detection method according to an embodiment of this application;

[0026] Figure 6This is a schematic diagram of another optional device malfunction detection method according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of another optional device malfunction detection method according to an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of an optional device for detecting device malfunctions according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] The present application will be described below with reference to embodiments:

[0032] According to one aspect of the embodiments of this application, a method for detecting equipment malfunctions is provided. Optionally, in this embodiment, the above-described method for detecting equipment malfunctions can be applied to, for example... Figure 1 The hardware environment shown consists of server 101 and terminal device 103. For example... Figure 1As shown, server 101 is connected to terminal device 103 via a network and can be used to provide services to terminal device or applications installed on terminal device. Application 107 can be video application, instant messaging application, browser application, educational application, game application, etc. Database 105 can be set up on the server or independently of the server to provide data storage services for server 101, such as a game data storage server. The network mentioned above can include, but is not limited to, wired networks and wireless networks. The wired network includes local area networks, metropolitan area networks, and wide area networks. The wireless network includes Bluetooth, WIFI, and other networks that enable wireless communication. Terminal device 103 can be a terminal configured with an application, and can include, but is not limited to, at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), laptops, tablets, handheld computers, MID (Mobile Internet Devices), PADs, desktop computers, smart TVs, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, virtual reality (VR) terminals, augmented reality (AR) terminals, mixed reality (MR) terminals, and other computer devices. The server mentioned above can be a single server, a server cluster composed of multiple servers, or a cloud server.

[0033] Combination Figure 1 As shown, the above-mentioned device anomaly detection method can be executed by an electronic device, which can be a terminal device or a server. The above-mentioned device anomaly detection method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0034] The above is merely an example, and this embodiment does not impose any specific limitations.

[0035] Alternatively, as an alternative implementation method, such as Figure 2 As shown, the detection methods for the above-mentioned equipment malfunctions include:

[0036] S202, determine the first task parameters for data interaction with the device disk based on the first operation data corresponding to the target task, wherein the first task parameters are used to indicate the disk operation rate of the device disk during the execution of the target task;

[0037] S204, compare the historical operation data corresponding to each of the multiple historical tasks with the first operation data to determine the minimum task parameter. The target task and the multiple historical tasks have the same task type, the first operation data and the historical operation data meet the preset family condition, and the minimum task parameter is used to indicate the disk operation rate of the specified device disk during the execution of the target task.

[0038] S206, if the preset value conditions are met between the first task parameter and the minimum task parameter, a disk error message is generated.

[0039] Optionally, the target task can be a task under any distributed computing framework, such as a task in the Spark framework. A total task job (i.e., a Spark application) can be composed of multiple stages. Each stage has its own stage data, which is the first operation data mentioned above. In other words, a stage corresponds to the target task mentioned above.

[0040] Furthermore, each stage can contain multiple sub-tasks. The operation data corresponding to the sub-tasks in a stage is the first operation data mentioned above, and the sub-tasks in a stage are the target tasks mentioned above.

[0041] In other words, a job can be broken down into several stages, each with its own independent stage data, namely the first operation data mentioned above. Furthermore, each stage contains multiple sub-tasks, and the operation data of these sub-tasks are closely related to the stage.

[0042] It should be noted that the IO operation types of subtasks within a stage (i.e., the task types of the target task mentioned above) are the same, and a subtask can have multiple IO operation types. For example, the IO operation types of subtasks within a stage can be both input and shuffle read. The IO operation types here can include, but are not limited to, shuffle read, shuffle write, input, and output.

[0043] Optionally, the first operation data refers to the disk interaction details of the subtask during execution, including but not limited to the specific rate values ​​of shuffle read, shuffle write, input, output, etc.

[0044] It should also be noted that the above comparison operation refers to the algorithm process used to determine whether the current task and the historical task are similar in execution characteristics, including but not limited to hash value comparison, statistical index comparison, time series analysis comparison and other methods.

[0045] Specifically, hash value comparison is applied to task similarity judgment based on DAG graph. Node attributes are updated iteratively through multiple rounds of the Weisfeiler-Lehman algorithm, and the resulting dag_hash is used to quickly compare whether the task structures are the same or similar. Statistical index comparison evaluates the stability of execution rate by calculating statistical measures such as data mean and standard deviation. Time series analysis comparison uses techniques such as autocorrelation function and wavelet transform to analyze the dynamic changes in disk I / O rate during task execution, thereby determining whether the required data exists. The embodiments of this application are not limited to the above methods, and the most suitable comparison strategy can be selected according to the specific scenario.

[0046] For example, after all the jobs in the main task containing the target task have been executed, the first operation data is obtained from the event log of the main task.

[0047] It should also be noted that, in this embodiment, the startup process of the Spark executor can be modified. In cloud-native scenarios, the Spark executor is often started in a container and can only be aware of the container's virtual address. In this embodiment, the physical machine IP can be set as an environment variable in the container, and the physical machine information can be obtained from the system environment variables when the executor starts.

[0048] Furthermore, it is agreed that the task data directory for Yarn is a fixed path on each server, such as / app / spark / data0, / app / spark / data1, etc. In this way, the disk drive letter can be obtained by executing the command `df -h / app / spark / data0` using Java's ProcessBuilder when the executor starts. The physical machine information and disk drive letter information are then attached to the executor's registration event and written to the Spark event log.

[0049] Optionally, the first task parameter reflects the disk operation rate of the device disk during the execution of subtasks. By deeply analyzing the Spark event logs, the read and write rate data of each subtask involving IO operations can be extracted. By statistically analyzing these rates, the average disk operation rate of the entire stage can be obtained and used as the first task parameter.

[0050] It should also be noted that obtaining the first task parameter involves deep analysis of the Spark event logs to ensure that the actual read and write rates of each task in the I / O-related stages are extracted. Therefore, the first task parameter reflects the I / O rate performance of all tasks within the entire stage, and the overall efficiency of disk operations can be more comprehensively evaluated by calculating the average I / O rate. This application is not limited to a specific rate calculation method; any method that accurately reflects disk operation efficiency is applicable.

[0051] Furthermore, after determining the first task parameters, a further filtering operation can be performed from multiple historical tasks to obtain multiple second operation data that meet the comparison conditions from the historical operation data.

[0052] Optionally, in this embodiment of the application, the aforementioned historical operation data refers to the operation data generated during the execution of multiple historical tasks that have the same task type as the target task and meet the preset family conditions, including but not limited to disk operation rate data such as shuffle read, shuffle write, disk read and disk write, and the device disk did not experience any abnormalities during the execution of the historical tasks corresponding to the historical data, that is, all historical tasks were executed successfully.

[0053] For example, the above-mentioned preset family condition can be understood as:

[0054] Suppose that the first operation data is the operation data corresponding to a stage A, and there exists a historical task B with the same business processing logic as the target task. The operation data corresponding to the historical task B is a historical operation data B. In this case, the first operation data and the historical operation data B satisfy the preset family condition.

[0055] For example, Figure 3 This is a schematic diagram of an optional task according to an embodiment of this application, such as... Figure 3 As shown, assume the target task corresponds to stage A, which includes two processing nodes: node A for data acquisition and node B for data transformation. The historical task B corresponds to stage B, which also includes two processing nodes: node C for data acquisition and node D for data transformation. The task type of the target task in stage A is input (that is, the IO operation type of the subtasks in stage A is input), and the task type of the historical task in stage B is also input (that is, the IO operation type of the subtasks in stage B is input). Since the business processing logic of the two is the same, the first operation data and the historical operation data B satisfy the preset family condition.

[0056] It should be noted that the target task can correspond to a stage in Spark application 1, and historical task B can correspond to a stage in Spark application 2. Spark application 1 and Spark application 2 can be the same or different.

[0057] In an exemplary embodiment, when a user submits a Spark application, they can actively set the name of the family. That is, the user can independently determine the family to which the Spark application belongs. Spark applications belonging to the same family have the same business logic, including but not limited to the same code implementation logic. Then, Stage data (i.e., the aforementioned second operation data) with the same hash value as the first operation data is determined from the Spark applications belonging to the same family.

[0058] For example, a user first submits three Spark applications: the first Spark application, the second Spark application, and the third Spark application. Specifically:

[0059] When a user submits their first Spark application, the family to which the first Spark application belongs is set to family A; when a user submits their second Spark application, the family to which the second Spark application belongs is set to family A; when a user submits their third Spark application, the family to which the third Spark application belongs is set to family B.

[0060] It is understandable that the first and second Spark applications both belong to family A, and their business processing logic is the same, for example, both perform the same data analysis and processing on the data of website A. However, the third Spark application belongs to family B, and its business processing logic is different from that of the first and second Spark applications.

[0061] Furthermore, assuming the first operation data is Stage data from the first Spark application, then, based on the relevant application information of the first Spark application, we know that the first Spark application belongs to family A. Therefore, we can identify other Spark applications that also belong to family A from among the other Spark applications. Furthermore, since the second Spark application also belongs to family A, while the third Spark application does not, therefore:

[0062] The Stage data in the second Spark application meets the preset family condition, that is, the Stage data in the second Spark application is the aforementioned historical operation data.

[0063] The Stage data in the third Spark application does not meet the preset family condition, that is, the Stage data in the third Spark application is not the aforementioned historical operation data.

[0064] Furthermore, in the example above, historical operation data that meets the preset family condition with the first operation data is selected from multiple historical tasks. Then, the historical operation data corresponding to the multiple historical tasks are compared with the first operation data to obtain multiple second operation data.

[0065] For example, taking the hash value comparison operation as an example, firstly, the Weisfeiler-Lehman algorithm is executed multiple times on the RDD lineage tree maintained for the target task. The attributes of the upstream neighboring nodes of the current node are sorted by dictionary, and the downstream neighboring nodes are also sorted by dictionary. The target hash value corresponding to the data of the first operation is calculated for the attribute value of the current node (node ​​name, such as the filout operation), the string concatenated with the upstream sorting result and the downstream sorting result.

[0066] In the business scenario of obtaining the target hash value corresponding to the first operation data mentioned above, assuming the RDD lineage tree corresponding to the Stage is as follows, initially, the attribute of each node is the RDD name. After multiple rounds of WL algorithm iteration, the attribute of each node corresponds to a string of numbers. These numbers are topologically sorted. The sorting principle is to first enqueue the nodes with an out-degree of 0 in the current graph. Finally, the sorting result is obtained, and the hash value of the sorting result is obtained to obtain the target hash value.

[0067] Similarly, firstly, for each historical operation data corresponding to the historical task's maintained RDD lineage tree, multiple rounds of the Weisfeiler-Lehman algorithm are executed. The attributes of the upstream neighboring nodes of the current node are sorted lexicographically, and the downstream neighboring nodes are also sorted lexicographically. The historical hash value corresponding to the first operation data is calculated for the string concatenated with the attribute value of the current node (node ​​name, such as the filout operation), the upstream sorting result, and the downstream sorting result.

[0068] In the business scenario of obtaining the historical hash value corresponding to the above historical operation data, assuming that the RDD lineage tree corresponding to the Stage is as follows, initially, the attribute of each node is the RDD name. After multiple rounds of WL algorithm iteration, the attribute of each node corresponds to a string of numbers. These numbers are topologically sorted. The sorting principle is to first enqueue the nodes with an out-degree of 0 in the current graph. Finally, the sorting result is obtained, and the hash value of the sorting result is obtained to obtain the historical hash value.

[0069] If the target hash value and the historical hash value are the same, the historical operation data corresponding to the historical hash value is determined as the second operation data.

[0070] Furthermore, after obtaining a specified number of second operation data, the second task parameters corresponding to each second operation data can be determined, and these multiple second task parameters can be stored in the historical task parameter array.

[0071] Furthermore, if the number of second operation data is too small, for example, if it is pre-set that the second task parameters corresponding to each second operation data are determined only when the number of second operation data is greater than or equal to 5, and these multiple second task parameters are stored in the historical task parameter array, then if there is only one second operation data, the parsing result of the target task can be directly saved to the database. It is also possible to mark whether the first operation data is normal based on whether the target task execution status is successful, without having to perform subsequent operations.

[0072] Optionally, the second task parameter reflects the disk operation rate of the device disk during the execution of historical tasks. By deeply analyzing the Spark event logs, the read and write rate data of each subtask involving IO operations can be extracted. By statistically analyzing these rates, the average disk operation rate of the entire stage can be obtained and used as the second task parameter.

[0073] It should also be noted that obtaining the second task parameter involves deep analysis of the Spark event logs to ensure that the actual read and write rates of each task within the I / O-related stages are extracted. Therefore, the second task parameter reflects the I / O rate performance of all tasks throughout the entire stage, and the overall efficiency of disk operations can be more comprehensively evaluated by calculating the average I / O rate. This application is not limited to a specific rate calculation method; any method that accurately reflects disk operation efficiency is applicable.

[0074] Next, the minimum task parameters mentioned above can be determined from the historical task parameter series, including but not limited to using the box method.

[0075] Specifically, the minimum task parameter is determined from the historical task parameter series using the box plot method. This involves calculating a series of statistics for the series, including the minimum, first quartile (Q1), median (Q2), third quartile (Q3), and maximum.

[0076] For example, assuming the sequence of historical task parameters is {1, 2, 3, 4, 5, 6}, the box plot method is calculated as follows:

[0077] The minimum value of this sequence is 1, the maximum value is 6, and the median is 3.5 ((3+4) / 2, because the sequence has an even number of elements). The first quartile, Q1, is 1.5, which is the value at the (1 / 4)n position. Here, n is 6, so it is the 1.5th number. In actual calculation, the nearest smaller value is rounded up, so the result is the second value, 2. Rounding down gives the first value, 1. Q1 is the interpolation between the two, 1.5. The third quartile, Q3, is 4.5, which is the value at the (3 / 4)n position. Here, n is 6, so it is the 4.5th number. The nearest smaller value is rounded up, so the result is the fifth value, 5. Rounding down gives the fourth value, 4. Q3 is the interpolation between the two, 4.5.

[0078] After obtaining Q1 and Q3, according to box plot theory, outliers outside the normal range can be further calculated, namely the upper and lower limits. The formulas for calculating the upper and lower limits are as follows:

[0079] Upper limit=Q3+1.5×(Q3-Q1);

[0080] Lower limit=Q1-1.5×(Q3-Q1);

[0081] For the sequence {1, 2, 3, 4, 5, 6}, the upper limit = 4.5 + 1.5 × (4.5 - 1.5) = 4.5 + 1.5 × 3 = 9; the lower limit = 1.5 - 1.5 × (4.5 - 1.5) = 1.5 - 1.5 × 3 = -4.5. Since all elements in the sequence are between the lower and upper limits, Q1 is directly defined as the minimum task parameter, i.e., the minimum task parameter is 1.5.

[0082] Finally, after obtaining the first task parameter and the minimum task parameter, it is determined whether they meet the preset value conditions. Specifically:

[0083] If the first task parameter is less than the minimum task parameter, it means that the first task parameter and the minimum task parameter meet the preset value conditions. At this time, since the Spark container has been modified in advance in this embodiment, the disk status information of the corresponding disk during the execution of the target task can be directly obtained in the event log. Then, the disk status information is combined to generate a disk abnormality prompt message.

[0084] For example, if it is detected that the first task parameter is indeed less than the minimum task parameter calculated by the box plot method, it means that the device disk operation rate of the target task during execution is significantly lower than the lower limit of the normal rate range of similar historical tasks, thus satisfying the preset value condition.

[0085] At this point, since the Spark container has been modified in advance in this embodiment, the event log can directly record the disk status information of the corresponding disk during the execution of the target task, such as key indicators such as disk IO utilization, IOPS, and read / write speed.

[0086] By combining disk status information obtained from the event log, disk anomaly messages can be automatically generated. For example, the message might include: "During the execution phase [specific stage number] of the Spark task with task ID [specific task ID], the disk IO rate is abnormally slow, below the historical normal range. Preliminary analysis suggests a possible disk performance bottleneck or hardware failure. It is recommended to check the health status and load of the disk [specific physical machine IP and disk drive letter]."

[0087] It should also be noted that, in this embodiment, if the target task has multiple task types, subsequent processing can be performed according to the task type. For example, if the target task's task type is input or shuffle read, then taking the target task's task type as input as an example, the first task parameter is determined, and the corresponding historical task type is also input. Similarly, taking the target task's task type as shuffle read as an example, the first task parameter is determined, and the corresponding historical task type is also shuffle read. Ultimately, two sets of comparison data will be obtained:

[0088] The first set of comparison data: the first and minimum task parameters corresponding to the input type.

[0089] The second set of comparison data: the first task parameters and the minimum task parameters corresponding to the shuffle read type.

[0090] At this point, as long as the first task parameter and the minimum task parameter in either the first set of comparison data or the second set of comparison data meet the above preset value conditions, the above disk abnormality prompt message can be generated.

[0091] In one exemplary embodiment, Figure 4 This is a schematic diagram of an optional device malfunction detection method according to an embodiment of this application, such as... Figure 4 As shown, let's take the target task corresponding to a certain stage in Spark as an example:

[0092] S1: After the user starts the Spark task analysis process, the event log of the corresponding Spark task is obtained, and the data in the stage is extracted.

[0093] S2: If a stage does not involve I / O operations, such as shuffling or reading / writing data from the file system, it is skipped. Otherwise, task metric data (performance statistics collected during Spark task execution, including data volume, execution time, memory consumption, read / write operations, execution status, etc., as mentioned in the first operation data) is further extracted from the stage. I / O metrics for each task are categorized and discussed separately; for example, shuffle read, shuffle write, input, and output all require separate rate calculations. Tasks within the same stage have the same I / O type.

[0094] S3, extracts historical execution data from the database for comparison:

[0095] If the number of historical executions (i.e. the number of second operation data) is less than a certain threshold, the parsing results of the task will be directly saved to the database, but the sample needs to be marked as normal based on whether the task execution status is successful.

[0096] If there are a sufficient number of normal samples, the anomaly threshold is calculated using box plot theory, and then it is determined whether the average I / O rate of tasks in the current stage is less than the threshold.

[0097] S3-1, If ​​not, update the indicators to the database, mark the sample as a normal sample, and end;

[0098] S3-2, if so, mark the sample as an abnormal sample, and send a request to the hardware monitoring system to query the IO utilization, IOPS, disk health status and other information of the specified disk (disk on some executor or disk on HDFS node) within the specified time (stage execution time period), and execute S4;

[0099] S4 provides users with disk error messages indicating slow I / O through the front end, and displays disk monitoring data as additional content, allowing users to quickly determine whether there are bad disks or disk I / O bottlenecks in the cluster.

[0100] In this embodiment, a first task parameter for data interaction with the device disk is determined based on the first operation data corresponding to the target task. This parameter specifically reflects the disk operation rate of the device disk during the execution of the target task. Next, historical operation data corresponding to multiple historical tasks are compared with the first operation data to filter out multiple second operation data that meet preset family conditions, i.e., historical tasks with the same type and hash value as the target task. Based on these second operation data, a historical task parameter sequence is determined, where each historical task parameter also indicates the disk operation rate during the execution of the historical task. A target merging operation is performed on the historical task parameter sequence to calculate the minimum task parameter, i.e., the slowest disk operation rate threshold in the historical tasks. If the difference between the first task parameter (the disk operation rate of the target task) and the calculated minimum task parameter meets a preset value condition, it means that the disk operation rate of the target task is significantly lower than the lowest normal rate of historical tasks of the same type. At this time, a disk anomaly warning message is generated, indicating a possible disk I / O anomaly. In other words, by comparing and analyzing the disk operation rates of the target task and historical tasks, the goal of accurately identifying device disk I / O anomalies is achieved, thereby improving the technical effect of improving the accuracy of device anomaly detection. It can promptly and accurately identify task execution anomalies caused by disk problems, thereby solving the technical problem of low detection accuracy of device anomalies.

[0101] As an optional approach, the above-mentioned comparison operation between the historical operation data corresponding to each of the multiple historical tasks and the first operation data is performed to determine the minimum task parameter includes: obtaining the historical hash value of the historical operation data and the target hash value of the first operation data; if the historical hash value and the target hash value are the same, the historical operation data is determined as the second operation data; a historical task parameter sequence is determined based on the second operation data, and a target merging operation is performed on the historical task parameter sequence to obtain the minimum task parameter, wherein one of the historical task parameters in the historical task parameter sequence is used to indicate the disk operation rate of the device disk during the execution of one of the historical tasks.

[0102] Optionally, in the embodiments of this application, the comparison operation performed on the historical operation data corresponding to multiple historical tasks refers to comparing the historical hash value of the historical operation data with the target hash value of the first operation data, including but not limited to the comparison of the historical hash value and the target hash value of operation types such as shuffle read, shuffle write, disk read and disk write.

[0103] It should be noted that both the historical hash value and the target hash value are calculated based on the Weisfeiler-Lehman algorithm, which is obtained through multiple iterations of the DAG graphs of the target and historical tasks, combined with the node attribute values ​​after topological sorting. In practice, the calculation of the historical hash value and the target hash value may involve the optimization of various data structures and algorithms. As long as the isomorphism of the task DAG graph can be ensured, this application does not limit the specific implementation details.

[0104] For example, the historical hash value of the historical operation data and the target hash value of the first operation data are obtained. If the hash values ​​of the two match, the historical operation data is regarded as the second operation data and used for the construction of the subsequent historical task parameter sequence.

[0105] In an exemplary embodiment, taking the DAG graph analysis of a Spark task as an example, the system first parses the event log, extracts the DAG graph of the target task, and calculates its target hash value. Then, the system traverses the historical task database to obtain the historical hash value corresponding to the DAG graph of each historical task. If the historical hash value equals the target hash value, the system extracts the IO operation rate data of the historical task and determines it as the second operation data, used to construct the historical task parameter sequence.

[0106] Through the embodiments of this application, the method of hash value comparison is used to accurately determine the isomorphism between historical tasks and target tasks, further ensuring the reasonable construction of the parameter sequence of historical tasks, and achieving the purpose of accurately identifying the IO anomalies of the device disk in the target task.

[0107] As an optional approach, the method further includes: generating a first dependency tree based on the first operation data, wherein the first dependency tree includes multiple first task nodes, which are used to interact with the device disk based on the first operation data; performing a concatenation operation on each of the first task nodes in the first dependency tree to obtain a second dependency tree; determining the corresponding initial hash value of each second task node in the second dependency tree based on its attribute value; performing a sorting operation on the initial hash values ​​to obtain an attribute string, including: determining the position of the initial hash value of the corresponding second task node in the attribute string based on the connection edge relationship between each second task node and its adjacent task nodes in the second dependency tree; and determining the target hash value based on the attribute string.

[0108] Optionally, in the embodiments of this application, the first dependency tree refers to a data structure composed of multiple first task nodes. These first task nodes are associated with the data interaction of the device disk during the execution of the target task, including but not limited to associated nodes of operation types such as shuffle read, shuffle write, disk read, and disk write.

[0109] It should be noted that the construction of the first dependency tree is based on the execution data of the Spark task. Possible dependencies include, but are not limited to, dependencies between RDDs and dependencies between stages. By analyzing the interaction between the first operation data and the device disk, a first dependency tree reflecting the data processing flow is constructed. This application does not limit the specific method for constructing the dependency tree.

[0110] For example, the system generates a first dependency tree based on the first operation data, where each task node represents a specific operation that interacts with the device disk. A concatenation operation is then performed on each first task node to form a second dependency tree containing node attribute values. These node attribute values ​​reflect the characteristics of the task node's interaction with the disk, such as operation type and speed.

[0111] In an exemplary embodiment, taking the Spark shuffle phase as an example, the system parses the first operation data of the shuffle write operation from the event log. Based on this data, a first dependency tree is constructed, where each node represents a different task in the shuffle write operation. Then, a concatenation operation is performed on each node to form a second dependency tree. Next, based on the attribute values ​​of each node in the second dependency tree, such as task ID, amount of data written, and write time, the initial hash value corresponding to the node is determined. The system sorts these initial hash values ​​to form attribute strings, and determines the target hash value based on the attribute strings.

[0112] As an optional approach, the above sorting operation on the initial hash value to obtain the attribute string includes: determining the position of the initial hash value of the corresponding second task node in the attribute string based on the connection edge relationship between each second task node in the second dependency tree and its adjacent task nodes.

[0113] Optionally, in this embodiment, the process of performing a sorting operation on the initial hash value to obtain the attribute string refers to determining the relative position of these values ​​in the attribute string based on the initial hash value of each second task node in the second dependency tree, combined with the connection edge relationships between nodes. Specifically, this includes, but is not limited to, finding the correct sorting position of the initial hash value of each node in the final attribute string during Spark task DAG graph parsing.

[0114] It should be noted that the connection relationship may include the node dependency relationship, execution order, etc., and the initial hash value sorting of the nodes may be based on different rules, such as topological sorting, dependency depth sorting, etc., to ensure that the attribute string can accurately reflect the structural characteristics of the second dependency tree. This application embodiment does not limit this.

[0115] For example, the position of the corresponding initial hash value in the attribute string is determined based on the connection edge relationship of each second task node in the second dependency tree.

[0116] For example, for each node, examine the connection edges between its previous and next task nodes, and determine the position of the node's initial hash value in the attribute string based on these relationships. This ensures that the attribute string can fully represent the structural information of the dependency tree, thereby accurately reflecting the isomorphism of task-disk interaction in subsequent hash value calculations.

[0117] Through the embodiments of this application, the method of generating a dependency tree and calculating a hash value is used to achieve an accurate description and isomorphism judgment of the interaction process between distributed tasks and device disks, thereby achieving the ability to quickly identify and diagnose disk I / O anomalies and improving the accuracy and efficiency of Spark task anomaly detection.

[0118] As an optional approach, the above-mentioned concatenation operation is performed on each of the first task nodes in the first dependency tree to obtain the second dependency tree, including: determining the corresponding first hash value based on the attribute value of each of the first task nodes; sequentially determining each of the first task nodes as the current task node and performing the following steps until the traversal of the first task nodes is completed: concatenating the first hash value of the current task node, the first hash value of the task node preceding the current task node, and the first hash value of the task node following the current task node to obtain a second hash value; updating the attribute value of the current task node based on the second hash value to obtain a second task node; and determining the second dependency tree based on the second task node.

[0119] Optionally, in this embodiment, the process of performing a concatenation operation on each first task node in the first dependency tree to obtain the second dependency tree refers to determining a first hash value based on the attribute value of each first task node, and updating the node attribute values ​​iteratively to form a new dependency tree structure. Specifically, this includes, but is not limited to, updating and reorganizing the hash value of each node in the DAG graph parsing of Spark tasks. Figure 5 This is a schematic diagram of another optional device anomaly detection method according to an embodiment of this application, where the first dependency tree and the second dependency tree are as follows: Figure 5 As shown.

[0120] It should be noted that attribute values ​​may include node ID, type of interaction with the device disk, rate, etc., which are not limited in this application, as long as they reflect the node characteristics. Furthermore, the splicing operation during the iterative update process may be adjusted according to different dependencies and node order to adapt to various distributed computing scenarios.

[0121] For example, for each first task node, a first hash value is first calculated based on its attribute values. Then, nodes are sequentially identified as the current task nodes, and a concatenation operation is performed to combine the first hash values ​​of the current node, its preceding task node, and its following task node to generate a second hash value. Subsequently, the attribute values ​​of the current task node are updated with the second hash value to form a second task node. This process is iterated until all first task nodes have been traversed, ultimately resulting in the second dependency tree.

[0122] In an exemplary embodiment, taking the Spark shuffle stage as an example, the system first calculates a first hash value based on the attribute values ​​(such as data volume, write time, etc.) of the first task node of each shuffle write operation. Then, through an iterative process, the first hash value of the current node is concatenated with the first hash values ​​of its neighboring nodes to form a second hash value, thereby updating the node attributes and generating a second task node. This operation continues until all the first task nodes of all shuffle write operations have been traversed, thus constructing a second dependency tree. The attributes of each node in the tree reflect the adjusted hash value, more accurately describing the isomorphism of the shuffle write operations.

[0123] Through the embodiments of this application, a hash iteration splicing method based on the attribute values ​​of dependency tree nodes is adopted to realize the dynamic description and isomorphism judgment of the interaction process between distributed tasks and device disks, thereby achieving the purpose of fine-grained identification and diagnosis of disk I / O anomalies and optimizing the accuracy and speed of task anomaly detection.

[0124] As an optional approach, the method further includes: obtaining a first number of connecting edges originating from the first child node and connecting to any of the aforementioned second task nodes other than itself, and obtaining a second number of connecting edges originating from the second child node and connecting to any of the aforementioned second task nodes other than itself, wherein the first child node and the second child node are different, and the second task node includes both the first child node and the second child node; when the first number is 0 and the second number is not 0, determining that the initial hash value of the first child node is in the first position of the aforementioned attribute string, and the initial hash value of the second child node is in the second position of the aforementioned attribute string, wherein the first position precedes the second position.

[0125] Optionally, in this embodiment, the process of obtaining the number of connection edges and determining the position of the initial hash value refers to, based on the topology of the second dependency tree, counting the number of connection edges from any two child nodes to other second task nodes, and then determining the position of the initial hash value in the attribute string based on these numbers. This involves, but is not limited to, analyzing the out-degree and in-degree of nodes in the DAG graph parsing of Spark tasks to ensure that the attribute string accurately reflects the structure of the dependency tree.

[0126] It should be noted that the first and second quantities may vary depending on the specific structure of the tree, the number of nodes, and the connection relationships between nodes. Additional logical judgments may even be needed to handle special cases, such as loops and parallel processing. This application does not limit the specific application scenarios and processing details, as long as the sorting position of the initial hash value can be determined based on the number of connected edges.

[0127] For example, the system obtains a first count of connecting edges originating from the first child node and a second count of connecting edges originating from the second child node. If the first count is 0 (meaning the node has no outgoing edges, i.e., no subsequent dependent nodes), and the second count is not 0 (indicating the node has at least one outgoing edge), then the initial hash value of the first child node is determined to be in the first position in the attribute string, while the initial hash value of the second child node is in the second position, with the first position preceding the second position. This process applies to all child nodes, ensuring that the construction of the attribute string follows the dependencies and execution order between nodes.

[0128] In an exemplary embodiment, taking the application scenario of Spark executing MapPartitions operations as an example, the second dependency tree constructed by the system contains multiple second task nodes, including first child nodes and second child nodes. The system counts that the number of connection edges originating from the first child node is 0, indicating that it is a leaf node in the dependency tree; while the number of connection edges originating from the second child node is not 0, indicating that it has subsequent dependent nodes. In this case, the initial hash value of the first child node is determined in the first position of the attribute string, and the initial hash value of the second child node is located in the second position, ensuring that the attribute string can accurately reflect the topology of the dependency tree and the execution order of the nodes.

[0129] Through the embodiments of this application, a sorting strategy based on the number of node connection edges is adopted to determine the position of the initial hash value in the attribute string, thereby achieving accurate capture and description of the characteristics of the dependency tree structure, and achieving the purpose of improving the accuracy of hash value calculation and quickly identifying disk I / O anomalies in distributed tasks.

[0130] As an optional approach, the method further includes: generating a third dependency tree based on one historical operation data corresponding to one of the multiple historical tasks, wherein the tree structure of the first dependency tree is the same as that of the third dependency tree, the third dependency tree includes multiple third task nodes, and the third task nodes are used to interact with the device disk based on one of the historical operation data; performing a concatenation operation on each of the third task nodes in the third dependency tree to obtain a fourth dependency tree; determining the initial hash value corresponding to each of the fourth task nodes in the fourth dependency tree based on the attribute value of each node; performing a sorting operation on the initial hash values ​​to obtain an attribute string; and determining a historical hash value based on the attribute string.

[0131] Optionally, in the embodiments of this application, the aforementioned historical operation data refers to information extracted from multiple historical tasks that interacts with the device disk, including but not limited to operation data such as shuffle read, shuffle write, disk read, and disk write, and their attribute values ​​during execution.

[0132] It should be noted that the rules for generating the third dependency tree are the same as those for the first dependency tree, and the shape and hierarchical relationship of the tree structure remain consistent. However, the attribute values ​​and hash values ​​of the nodes will be calculated based on historical operation data. This application embodiment does not limit this.

[0133] For example, a third dependency tree is generated based on the historical operation data of a single historical task, where the third task nodes reflect the specific disk interactions in the historical task. Then, a concatenation operation is performed on each third task node in the third dependency tree to obtain a fourth dependency tree, updating the attribute values ​​of each node. By sorting the updated attribute values ​​and combining them to form an attribute string, the system calculates a historical hash value for subsequent isomorphism determination.

[0134] In an exemplary embodiment, taking the application of Spark's shuffle write operation in historical tasks as an example, the system first constructs a third dependency tree based on historical shuffle write operation data. Each third task node in the tree represents a shuffle write operation in a historical task. Subsequently, the system performs a concatenation operation, converting the attribute values ​​of each node into attributes of a fourth task node. Through sorting and combination, an attribute string is obtained, and the historical hash value is calculated accordingly. This process ensures that the historical hash value accurately reflects the data processing flow and disk interaction characteristics of the shuffle write operation in the historical task.

[0135] Through the embodiments of this application, the method of generating a dependency tree and calculating historical hash values ​​is used to determine the isomorphism between historical tasks and current tasks, which further improves the diagnostic accuracy and speed of disk I / O anomalies, and achieves the goal of efficiently and accurately identifying abnormal disk conditions of devices in a distributed computing environment.

[0136] As an optional approach, the above-mentioned determination of the historical task parameter series based on the second operation data, and the execution of a target merging operation on the historical task parameter series to obtain the minimum task parameter, includes: obtaining the first quartile and the third quartile in the historical task parameter series, wherein 25% of the task parameters in the historical task parameter series are less than the first quartile, and 25% of the task parameters in the historical task parameter series are greater than the third quartile; and determining the minimum task parameter based on the first quartile and the third quartile.

[0137] Optionally, in this embodiment, the aforementioned historical task parameter series refers to a set of execution rate data based on similar stages in historical Spark tasks, including but not limited to task shuffle read rate, shuffle write rate, disk read rate, and disk write rate. These rate parameters form the basis of the series for subsequent statistical analysis.

[0138] It should be noted that the number of tasks and the distribution of rates in the historical task parameter series may vary, but the core of this application's embodiment is to use quartiles to determine the threshold boundary of abnormal rates. This method is applicable even in diverse situations such as varying series size and uneven rate distribution, and this application does not limit it.

[0139] For example, the system obtains the first quartile (Q1) and the third quartile (Q3) of the historical task parameter sequence. 25% of the task parameters are less than Q1, representing the lower-rate portion of the sequence; similarly, 25% of the task parameters are greater than Q3, representing the higher-rate portion. Based on Q1 and Q3, the system further determines the minimum task parameter, i.e., the boundary value of an abnormally low rate, for subsequent anomaly detection.

[0140] In an exemplary embodiment, taking the rate analysis of Spark shuffle write operations as an example, the system collects the task execution rates of the past 10 shuffle write operations, forming a historical task parameter sequence. Next, the system calculates the first and third quartiles of this sequence. Assuming the first quartile is 2 GB / s and the third quartile is 10 GB / s, this indicates that 25% of the tasks in the sequence have a rate below 2 GB / s, while the other 25% have a rate above 10 GB / s. Based on Q1 and Q3, the system determines the minimum task parameter to be 2 GB / s, used to determine whether the rate of the shuffle write operation in the current task is abnormally low; that is, a rate below 2 GB / s is considered potentially abnormal.

[0141] By employing the first and third quartiles of historical task parameter sequences to determine the minimum task parameters through the embodiments of this application, accurate anomaly detection of the current task execution rate is achieved, enabling rapid identification and response to disk I / O anomalies, and improving the stability and operational efficiency of large-scale data processing systems.

[0142] As an optional approach, the above method further includes: setting environment variables based on the device address of the target device where the device disk is located before executing the target task; and setting the environment variables in the container used to start the target task.

[0143] Optionally, in the embodiments of this application, the above-mentioned device address refers to the IP address of the physical machine where the storage device (such as a disk) is located, and the environment variable refers to the variable used to pass specific information when the container starts. They are used to ensure that the Spark executor can access the correct local disk information in a distributed environment, including but not limited to the environment variable settings in the container startup configuration.

[0144] It should be noted that the settings of device addresses and environment variables may depend on different cloud computing platforms or cluster management software, such as Kubernetes, Docker Swarm, Mesos, etc., as well as the parameter adjustments of various container runtimes. This application does not limit these settings.

[0145] For example, before executing the target task, environment variables are set based on the target device's device address (i.e., the physical machine's IP address), and these environment variables are injected into the container configuration that starts the target task. This allows the Spark executor in the container to access the environment variables to obtain the physical machine's IP address information, thereby locating the correct local disk path.

[0146] In an exemplary embodiment, taking a Spark task scheduling application scenario in a cloud-native environment as an example, the system sets the target device's address (physical machine IP) as an environment variable in the container's startup parameters when the container starts. As the container initializes, the environment variable is injected into the container, and applications, including the Spark executor, can access this information. After starting, the Spark executor determines its local disk path based on the physical machine IP provided in the environment variable and records this information in the event log, enabling subsequent IO anomaly diagnosis to accurately associate it with the specific disk device.

[0147] By using the embodiments of this application, the method of setting the target device address as an environment variable in the container startup configuration is adopted to achieve accurate binding between the Spark executor and the local disk, thereby improving the accuracy of IO anomaly detection and enhancing the flexibility and reliability of containerized Spark applications in complex cloud environments.

[0148] As an optional approach, the above method further includes: after the target task is completed, obtaining disk status data based on the event log of the target task; and if the first task parameter is less than the minimum task parameter, generating the disk error message based on the disk status data.

[0149] Optionally, in this embodiment, the aforementioned disk status data refers to various performance indicators of the device disk, including but not limited to disk utilization, IOPS, throughput, disk health status, etc. The minimum task parameter is a defined threshold used to compare the IO rate of the current task, thereby determining whether any anomalies exist.

[0150] It should be noted that there may be various ways to obtain disk status data. It can be done by directly reading relevant entries in the event log or indirectly by making real-time queries through interface calls with cluster monitoring systems (such as Prometheus, InfluxDB, etc.). This application does not limit this.

[0151] For example, after the target task is completed, its event log is parsed to extract disk status data. If the first task parameter (such as average IO rate) of a certain IO operation of the current task is found to be less than the preset minimum task parameter, a disk anomaly message is generated based on the extracted disk status data to inform the operations and maintenance personnel or developers that there may be a disk performance bottleneck.

[0152] In an exemplary embodiment, taking the monitoring of shuffle write rate during Spark task execution as an example, after the task ends, the event logs generated by Spark are first analyzed to obtain disk status data related to shuffle write. If the system calculates that the first task parameter (i.e., average write rate) of the current shuffle write operation is less than the previously calculated minimum task parameter, the system will trigger a disk anomaly detection process to query the usage of the corresponding disk during task execution. If the usage rate exceeds 85%, IOPS is lower than a preset baseline, or the disk health status is flagged as red, the system will synthesize this disk status data and generate a detailed disk anomaly message to notify relevant personnel to check and maintain it.

[0153] By combining task event log analysis with disk status data to generate anomaly alert messages through the embodiments of this application, timely warnings of potential disk I / O anomalies in Spark tasks are achieved. This enhances the operation and maintenance capabilities and troubleshooting efficiency of the distributed computing platform, effectively ensuring the stable operation of big data processing tasks.

[0154] In an exemplary embodiment, the above-described device anomaly detection method can be applied to business scenarios where Spark task IO anomalies are determined by combining historical task execution data. Specifically, the workflow of a distributed computing engine inevitably involves disk IO operations. Taking Spark as an example, firstly, input / output operations require reading and writing files from the distributed file system, which typically involves disk IO. Secondly, Spark's shuffle operation requires classifying data according to keywords, during which data is categorized and written to disk, and subsequent reduce tasks read the corresponding data from each executor's local disk. Furthermore, Spark's spill mechanism also involves disk IO. When performing operations such as sorting and broadcasting, if memory is insufficient, Spark will temporarily write data to disk, which also involves disk IO operations.

[0155] When disk anomalies occur, the PaaS computing engine may exhibit very subtle signs. It might simply mean that task execution time has increased, or it might show obscure anomaly logs, such as a TCP connection failure due to disk I / O blocking network information exchange. If the disk anomaly occurs during the shuffle process, it could even lead to a "Fetch failed" exception. Furthermore, it's difficult to determine if a task has an I / O bottleneck or anomaly based solely on its execution time. For example, tasks processing larger amounts of data will inevitably take longer, and the number and type of disks will also affect execution time. Even network conditions can impact task speed and interfere with the assessment.

[0156] Therefore, in this embodiment, by combining the analysis results of historical event logs of multiple identical tasks, it is determined whether the IO operation of the current task is abnormally slow. If the IO rate is considered slow during the execution of the current task, an alarm is issued, and the hardware monitoring system is requested to provide the corresponding disk monitoring metrics for display to the operations and maintenance personnel. First, the content of the corresponding Spark event messages is modified so that it can obtain relevant hardware information and record it in the event log. After each task ends, the Spark event information is analyzed, and the historical execution result data of the task is obtained from the database. The IO read and write rates in the stage are compared. If it is found that the IO rate of the current task is less than a certain threshold compared with the historical execution results, it is considered that the IO is slow during the current execution. Subsequently, with the help of the monitoring system, the monitoring data of the working disk during the execution of the stage is obtained, and the monitoring metrics are fed back to Spark developers or operations and maintenance personnel for analysis and reference. This process can help Spark application developers quickly check whether Spark task anomalies are caused by disk problems, improving the efficiency of problem investigation.

[0157] In an exemplary embodiment, the content of the corresponding Spark event messages is first modified to enable them to obtain relevant hardware information and record it in the event log. Specifically:

[0158] The physical machine IP is set as an environment variable in the container. When the executor starts, it can obtain the physical machine information from the system environment variables. It is agreed that the task data directory of YARN is a fixed path on each server.

[0159] After a user initiates the Spark task analysis process, the event logs of the corresponding Spark task are retrieved, and data extraction from the stage begins. If a stage does not involve I / O operations (e.g., no shuffle or file system read / write operations), it is skipped. Otherwise, task metric data (performance statistics collected during Spark task execution, including data volume, execution time, memory consumption, read / write counts, and execution status) is further extracted from the stage. I / O metrics for each task are categorized and discussed separately; for example, shuffle read, shuffle write, input, and output rates need to be calculated individually. For tasks within the same stage with the same I / O type, historical execution data is extracted from the database for comparison. If the historical execution count is less than a certain threshold, the task's parsing results are directly saved to the database, but the sample's normality is marked based on the task's execution success status. If there are a sufficient number of normal samples, the anomaly threshold is calculated using box plot theory. Then, it is determined whether the average IO rate of the task in the current stage is less than the threshold. If not, the metrics are updated to the database, and the sample is marked as a normal sample. If yes, the sample is marked as an abnormal sample. At the same time, a request is sent to the hardware monitoring system to query the IO utilization, IOPS, disk health status, and other information of the specified disk (disks on certain executors or disks on HDFS nodes) within a specified time period (stage execution time period). The front end provides the user with an alarm for slow IO and displays the disk monitoring data as additional content. The user can quickly determine whether there are bad disks or disk IO bottlenecks in the cluster.

[0160] S1, Retrieving Spark Temporary Data Disk Information as follows: Figure 6 As shown:

[0161] Understandably, locating a disk device requires two pieces of information: the physical machine IP address where the disk resides and the disk drive letter. The disks used for Spark shuffle or other runtime temporary data are local disks on the physical machine where the executor resides. Therefore, after the executor starts, it needs to confirm which temporary data disk it is using.

[0162] After a Spark task is submitted, the executor starts. Modifying the Spark executor startup process is necessary. In cloud-native scenarios, Spark executors are often started in containers and can only perceive the container's virtual address. The physical machine IP needs to be set as an environment variable in the container so that the executor can obtain physical machine information from the system environment variables when it starts.

[0163] Furthermore, it is agreed that the task data directory of Yarn is a fixed path on each server, such as / app / spark / data0, / app / spark / data1, etc. In this way, when the executor starts, the disk drive letter can be obtained by executing the command `df -h / app / spark / data0` through Java's ProcessBuilder. The physical machine information and disk drive letter information are then attached to the executor's registration event and written to the Spark event log.

[0164] S2, Task I / O Analysis and Diagnosis:

[0165] After the task execution is complete, the diagnostic module reads the task's event log and extracts the relevant information to store in the database. Specifically, it iterates through each stage involving I / O operations, obtaining the data volume and time consumed for each task's shufflewrite, shuffleread, input, or output read / write operations within that stage. Dividing the data volume by the time consumed yields the I / O rate of the task. The analyzed data, along with application information, is saved to the database, with the following table design:

[0166] First, the physical machine information table is shown in Table 1, which includes the IP address assigned to each physical machine in the cluster and the set of all disks on that physical machine used as temporary data directories for YARN:

[0167] Table 1

[0168]

[0169]

[0170] Secondly, the basic information of the application is shown in Table 2. Here, "family" indicates the same task, meaning the business logic of the two tasks is exactly the same. This field needs to be explicitly specified by the user when starting the Spark task. "app_id" and "app_name" are the basic information of the Spark application. "exec_map" records the corresponding physical machine IPs to which the executors of this task are assigned.

[0171] Table 2

[0172]

[0173] Finally, there is the Stage information table, which includes information about all stages in an app that involve shuffle or reading / writing to HDFS. The io_type defines the type of I / O, such as shuffle read, shuffle write, or reading from or writing to HDFS. The dag_hash field is a hash value calculated based on the DAG relationship of the Stage, which can be used as a unique identifier for the Stage. That is, if two Stages have the same dag_hash value, it is considered that the two Stages are performing the same computation logic.

[0174] Furthermore, in the process of deriving the dag_hash value, we perform a multi-round Weisfeiler-Lehman algorithm on the RDD lineage tree maintained in this stage.

[0175] Specifically: Sort the attributes of the upstream neighboring nodes of the current node by dictionary order, and sort the downstream neighboring nodes by dictionary order as well. Calculate the hash value of the string concatenated with the attribute value of the current node (node ​​name, such as the filout operation), the upstream sorting result, and the downstream sorting result.

[0176] For example, such as Figure 7 As shown, note that nodes with an attribute value of 2 will produce the same result after the first hash value calculation. This may affect subsequent sorting decisions, so multiple rounds of algorithm iteration are required. After multiple rounds of iteration, unless the subtree structure is exactly the same, nodes will often produce different attribute values.

[0177] In scenarios where the hash value of a DAG graph is calculated, such as... Figure 5 As shown, assuming the RDD lineage tree corresponding to Stage is as follows, initially, the attribute of each node is the RDD name. After multiple rounds of WL algorithm iteration, the attribute of each node corresponds to a string of numbers, and these numbers are topologically sorted.

[0178] The sorting principle is to first enqueue the nodes with an out-degree of 0 in the current graph (the number of edges pointing from that node to other nodes). The first node enqueueed is "204". After "204" is enqueued, it is immediately removed from the graph. Therefore, the next node enqueueed is "84". After "84" is removed, two nodes with an out-degree of 0 will appear in the graph, "178" and "884". At this time, the node with the smaller number is enqueueed first, that is, "178" is enqueueed first. Finally, the sorting result is obtained. Usually, after multiple rounds of iteration, the heterogeneous DAG graphs have enough differences to greatly reduce the probability of hash collisions.

[0179] Next, the hash value of the sorting result is calculated and stored as an attribute in the dag_hash field of the Stage information table. The task_rate_avg is the average rate of all tasks in this stage, which can be obtained by analyzing the event logs and calculating the rate of each task. In addition, the start and end times of the stage are also recorded.

[0180] The last field indicates whether the stage is an I / O exception; whether a task is abnormal is determined using a box plot. The specific approach is as follows:

[0181] First, find all stage records of the same family in the current environment from the database. The dag_hash must be equal to the dag_hash value of the stage being compared, and they must belong to the same IO type. Additionally, the abnormal flag must be false, meaning there are no stages with IO abnormalities.

[0182] Collect all stages that meet these conditions, and obtain a sequence of average task execution rates (shuffle read, shuffle write, input, or output rates) for each stage.

[0183] For this sequence, the formula for calculating the quartiles is as follows:

[0184] Q_1 (first quartile) is the number in the sequence at position (n+1) / 4, where n is the length of the sequence; Q_3 (third quartile) is the number in the sequence at position (3×(n+1)) / 4.

[0185] Then calculate the threshold: IQR = Q_3 - Q_1;

[0186] Threshold (minimum task parameter) = Q_1 - 1.5 × IQR;

[0187] It can be assumed that if the average task rate of the stage of the currently analyzed task is less than this threshold, then the current task has an IO slowness exception when executing in this stage, and the abnormal field is set to true.

[0188] The structure of the Spark application IO information table is shown in Table 3:

[0189] Table 3

[0190]

[0191] The S3 monitoring module can use time-series databases such as InfluxDB and Prometheus to monitor the performance and running status of each node, including performance-related metrics of each disk on the physical machine.

[0192] S4. When a user calls the task's diagnostic analysis interface, the system determines whether it needs to obtain additional information from the disk monitoring module and display it on the front end based on the abnormal flag recorded in the database.

[0193] When abnormal is true, an alarm is issued, and monitoring data for the corresponding time period is retrieved from the time-series database of the disk monitoring module based on the execution time period of the stage for display, such as key indicators like IO utilization, IOPS, and throughput.

[0194] This application provides a solution for quickly identifying and diagnosing IO anomalies in Spark tasks. Combined with a hardware monitoring system, it can quickly determine whether the task anomaly is caused by underlying disk IO anomalies. It is particularly suitable for big data cluster scenarios where disks are bottlenecks. By iterating the DAG graph of the stage using the WL algorithm and calculating the hash value after topological sorting, it can determine whether the stage belongs to a homogeneous task. By comparing the task execution rate performance between homogeneous stages of the same task, it can determine whether the IO during the execution of a certain stage is slow. If so, an alarm is issued, and the monitoring data (disk status data) of the corresponding disk during the task execution period is displayed to the operation and maintenance personnel.

[0195] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0196] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0197] According to another aspect of the embodiments of this application, an apparatus for detecting equipment malfunctions for implementing the above-described equipment malfunction detection method is also provided. For example... Figure 8 As shown, the device includes:

[0198] The determining module 802 is used to determine the first task parameters for data interaction with the device disk based on the first operation data corresponding to the target task, wherein the first task parameters are used to indicate the disk operation rate of the device disk during the execution of the target task;

[0199] The execution module 804 is used to perform a comparison operation between the historical operation data corresponding to each of the multiple historical tasks and the first operation data to determine the minimum task parameter. The target task and the multiple historical tasks have the same task type, the first operation data and the historical operation data meet the preset family condition, and the minimum task parameter is used to indicate the disk operation rate of the specified device disk during the execution of the target task.

[0200] The generation module 806 is used to generate a disk error message when the preset value conditions between the first task parameter and the minimum task parameter are met.

[0201] As an optional solution, the above-mentioned device is used to perform a comparison operation between the historical operation data corresponding to each of the multiple historical tasks and the first operation data to determine the minimum task parameter in the following manner: obtaining the historical hash value of the historical operation data and the target hash value of the first operation data; if the historical hash value and the target hash value are the same, the historical operation data is determined as the second operation data; a historical task parameter sequence is determined according to the second operation data, and a target merging operation is performed on the historical task parameter sequence to obtain the minimum task parameter, wherein one of the historical task parameters in the historical task parameter sequence is used to indicate the disk operation rate of the device disk during the execution of a historical task.

[0202] As an optional solution, the above-mentioned apparatus is further configured to: generate a first dependency tree based on first operation data, wherein the first dependency tree includes multiple first task nodes, the first task nodes being used to interact with the device disk based on the first operation data; perform a concatenation operation on each first task node in the first dependency tree to obtain a second dependency tree; determine the corresponding initial hash value of each second task node in the second dependency tree based on the attribute value of each second task node; perform a sorting operation on the initial hash values ​​to obtain an attribute string, including: determining the position of the initial hash value of the corresponding second task node in the attribute string based on the connection edge relationship between each second task node in the second dependency tree and its adjacent task nodes; and determine the target hash value based on the attribute string.

[0203] As an optional approach, the aforementioned apparatus is used to perform a concatenation operation on each first task node in the first dependency tree to obtain a second dependency tree by: determining the corresponding first hash value based on the attribute value of each first task node; sequentially determining each first task node as the current task node and performing the following steps until the traversal of the first task nodes is completed: concatenating the first hash value of the current task node, the first hash value of the task node preceding the current task node, and the first hash value of the task node following the current task node to obtain a second hash value; updating the attribute value of the current task node based on the second hash value to obtain a second task node; and determining the second dependency tree based on the second task node.

[0204] As an alternative, the above-mentioned device is used to perform a sorting operation on the initial hash value to obtain the attribute string in the following manner: based on the connection edge relationship between each second task node in the second dependency tree and its adjacent task nodes, the position of the initial hash value of the corresponding second task node in the attribute string is determined.

[0205] As an optional solution, the above-mentioned device is further configured to: obtain a first number of connection edges from the first child node to any second task node other than itself, and obtain a second number of connection edges from the second child node to any second task node other than itself, wherein the first child node and the second child node are different, and the second task node includes the first child node and the second child node; when the first number is 0 and the second number is not 0, determine that the initial hash value of the first child node is in the first position in the attribute string, and the initial hash value of the second child node is in the second position in the attribute string, wherein the first position is before the second position.

[0206] As an optional solution, the above-mentioned device is further configured to: generate a third dependency tree based on historical operation data corresponding to one of the multiple historical tasks, wherein the tree structure of the first dependency tree is the same as that of the third dependency tree, the third dependency tree includes multiple third task nodes, and the third task nodes are used to interact with the device disk based on historical operation data; perform a concatenation operation on each third task node in the third dependency tree to obtain a fourth dependency tree; determine the initial hash value corresponding to each fourth task node in the fourth dependency tree based on the attribute value of each fourth task node; perform a sorting operation on the initial hash values ​​to obtain an attribute string; and determine a historical hash value based on the attribute string.

[0207] As an optional approach, the aforementioned apparatus is used to determine a historical task parameter sequence based on second operational data in the following manner, and to perform a target merging operation on the historical task parameter sequence to obtain the minimum task parameter: obtaining the first quartile and the third quartile in the historical task parameter sequence, wherein 25% of the task parameters in the historical task parameter sequence are less than the first quartile, and 25% of the task parameters in the historical task parameter sequence are greater than the third quartile; and determining the minimum task parameter based on the first quartile and the third quartile.

[0208] As an optional solution, the above-mentioned apparatus is also used to: set environment variables according to the device address of the target device where the device disk is located before executing the target task; and set the environment variables in the container used to start the target task.

[0209] As an optional solution, the above-mentioned device is also used to: obtain disk status data according to the event log of the target task after the target task is completed; and generate a disk abnormality prompt message according to the disk status data if the first task parameter is less than the minimum task parameter.

[0210] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0211] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0212] According to one aspect of this application, a computer program product is provided, the computer program product comprising a computer program.

[0213] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0214] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions defined in the system of this application.

[0215] In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs the various functions provided in the embodiments of this application.

[0216] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0217] According to one aspect of this application, a computer-readable storage medium is provided, from which a processor of an electronic device reads computer instructions, and the processor executes the computer instructions, causing the electronic device to perform the device anomaly detection method provided in various alternative implementations of the device anomaly detection aspect described above.

[0218] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store methods for performing the embodiments of this application.

[0219] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0220] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0221] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more electronic devices to execute all or part of the steps of the methods described in the various embodiments of this application.

[0222] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0223] In the several embodiments provided in this application, it should be understood that the disclosed application can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0224] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0226] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting equipment malfunctions, characterized in that, include: The first task parameters for data interaction with the device disk are determined based on the first operation data corresponding to the target task, wherein the first task parameters are used to indicate the disk operation rate of the device disk during the execution of the target task; The historical operation data corresponding to each of the multiple historical tasks are compared with the first operation data to determine the minimum task parameter. The target task and the multiple historical tasks have the same task type, the first operation data and the historical operation data meet the preset family condition, and the minimum task parameter is used to indicate the disk operation rate of the device disk during the execution of the target task. If the preset value conditions are met between the first task parameter and the minimum task parameter, a disk error message is generated.

2. The method according to claim 1, characterized in that, The step of comparing the historical operation data corresponding to each of the multiple historical tasks with the first operation data to determine the minimum task parameters includes: Obtain the historical hash value of the historical operation data and the target hash value of the first operation data; If the historical hash value and the target hash value are the same, the historical operation data will be determined as the second operation data. Based on the second operation data, a sequence of historical task parameters is determined, and a target merging operation is performed on the sequence of historical task parameters to obtain the minimum task parameters. In this sequence, one historical task parameter is used to indicate the disk operation rate of the device disk during the execution of one of the historical tasks.

3. The method according to claim 2, characterized in that, The method further includes: A first dependency tree is generated based on the first operation data, wherein the first dependency tree includes a plurality of first task nodes, and the first task nodes are used to perform data interaction with the device disk based on the first operation data; Perform a concatenation operation on each of the first task nodes in the first dependency tree to obtain the second dependency tree; The initial hash value is determined based on the attribute value of each second task node in the second dependency tree. Perform a sorting operation on the initial hash value to obtain an attribute string, including: determining the position of the initial hash value of the corresponding second task node in the attribute string based on the connection edge relationship between each second task node in the second dependency tree and its adjacent task nodes; The target hash value is determined based on the attribute string.

4. The method according to claim 3, characterized in that, The step of performing a concatenation operation on each of the first task nodes in the first dependency tree to obtain the second dependency tree includes: The first hash value is determined based on the attribute value of each of the first task nodes; Each of the first task nodes is sequentially determined as the current task node, and the following steps are performed until the first task node traversal is completed: concatenate the first hash value of the current task node, the first hash value of the task node preceding the current task node, and the first hash value of the task node following the current task node to obtain a second hash value; update the attribute value of the current task node according to the second hash value to obtain a second task node; The second dependency tree is determined based on the second task node.

5. The method according to claim 3, characterized in that, The method further includes: Get the first number of connecting edges from the first child node to any second task node other than itself, and get the second number of connecting edges from the second child node to any second task node other than itself, wherein the first child node and the second child node are different, and the second task node includes the first child node and the second child node; If the first quantity is 0 and the second quantity is not 0, the initial hash value of the first child node is determined to be in the first position of the attribute string, and the initial hash value of the second child node is determined to be in the second position of the attribute string, wherein the first position is before the second position.

6. The method according to claim 2, characterized in that, The step of determining the historical task parameter sequence based on the second operation data and performing a target merging operation on the historical task parameter sequence to obtain the minimum task parameter includes: Obtain the first quartile and the third quartile in the historical task parameter sequence, wherein 25% of the task parameters in the historical task parameter sequence are less than the first quartile, and 25% of the task parameters in the historical task parameter sequence are greater than the third quartile; The minimum task parameters are determined based on the first quartile and the third quartile.

7. A device for detecting equipment malfunctions, characterized in that, include: The determining module is used to determine the first task parameters for data interaction with the device disk based on the first operation data corresponding to the target task, wherein the first task parameters are used to indicate the disk operation rate of the device disk during the execution of the target task; An execution module is used to perform a comparison operation between the historical operation data corresponding to each of the multiple historical tasks and the first operation data to determine the minimum task parameter. The target task and the multiple historical tasks have the same task type, the first operation data and the historical operation data meet a preset family condition, and the minimum task parameter is used to indicate the disk operation rate of the specified device disk during the execution of the target task. The generation module is used to generate a disk error message when the first task parameter and the minimum task parameter meet a preset value condition.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 6.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 6 through the computer program.

Images