Task data processing method, device, electronic device and storage medium
By acquiring the task relationship graph and the set of key tasks, the execution information of related tasks is determined, and related tasks are iteratively added for anomaly detection. This solves the problem of time-consuming and inaccurate anomaly analysis in task scheduling scenarios, and achieves the effect of quickly locating and repairing task anomalies.
Patent Information
- Application Number
- CN202111565622.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-20
Smart Images

Figure CN114238055B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data analysis, and in particular to a task data processing method, device, electronic device and storage medium. Background Art
[0002] Currently, in the task scheduling scenario, if an exception occurs in the endpoint task, an exception analysis is required for the endpoint task. Existing exception analysis usually traverses the task path with the largest total execution time and relies on manual processing and judgment. The analysis process is time-consuming, the analysis results are inaccurate, and the exception cannot be quickly located and repaired. Therefore, how to improve the efficiency of exception analysis for tasks is an urgent problem to be solved. Summary of the invention
[0003] The embodiments of the present application provide a task data processing method, device, electronic device and storage medium, which can improve the efficiency and accuracy of abnormal analysis for tasks.
[0004] On the one hand, an embodiment of the present application provides a task data processing method, the method comprising:
[0005] Obtain a task relationship diagram and a key task set of the task to be detected; the task relationship diagram includes N tasks and data flows between the N tasks, the key task set includes k key tasks, and the key tasks are connected to the task to be detected in the task relationship diagram, N is a positive integer, and k is a positive integer less than N;
[0006] Determine at least one associated task of the kth key task according to the data flow between the N tasks;
[0007] Acquire execution information of each associated task in the at least one associated task;
[0008] Determine a target associated task from the at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task;
[0009] According to the key task set to which the target associated task is added, anomaly detection is performed on the task to be detected, a detection result for the task to be detected is obtained, and the detection result is sent to the terminal device.
[0010] On the one hand, an embodiment of the present application provides a task data processing device, the device comprising:
[0011] An acquisition module is used to acquire a task relationship diagram and a key task set of the task to be detected; the task relationship diagram includes N tasks and data flows between the N tasks, the key task set includes k key tasks, and the key tasks are connected to the task to be detected in the task relationship diagram, N is a positive integer, and k is a positive integer less than N;
[0012] A determination module, configured to determine at least one associated task of the kth key task according to the data flow between the N tasks;
[0013] The acquisition module is further used to acquire execution information of each associated task in the at least one associated task;
[0014] The determining module is further configured to determine a target associated task from the at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task;
[0015] The detection module is used to perform anomaly detection on the task to be detected according to the key task set to which the target associated task is added, obtain the detection result for the task to be detected, and send the detection result to the terminal device.
[0016] On the one hand, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute some or all of the steps in the above method.
[0017] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions, which, when executed by a processor, are used to execute some or all of the steps in the above method.
[0018] In the embodiment of the present application, a task relationship diagram and a key task set of the task to be detected are obtained, at least one associated task of the kth key task is determined according to the data flow between N tasks, the execution information of each associated task is obtained, the target associated task is determined from at least one associated task according to the execution information of each associated task, and the target associated task is added to the key task set as the k+1th key task. According to the key task set with the target associated task added, an abnormality detection is performed on the task to be detected, and the detection result for the task to be detected is obtained and sent to the terminal device. By implementing the above method, the execution information of each associated task corresponding to the kth key task of the task to be detected can be obtained. Since the execution of the associated task will affect the execution of the kth key task, the k+1th key task that has a greater impact on the execution of the kth key task can be determined through the execution information, and each key task has a connection relationship with the task to be detected, which will affect the execution of the task to be detected. The key task set of the task to be detected obtained can be understood as a key task set that has a greater impact on the task to be detected, and the detection result obtained by using the key task set for abnormality detection is more accurate, which can improve the efficiency of abnormal analysis for tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 A flowchart of a task data processing method provided in an embodiment of the present application;
[0021] Figure 2 A schematic diagram of a task relationship diagram provided in an embodiment of the present application;
[0022] Figure 3 A flowchart of a task data processing method provided in an embodiment of the present application;
[0023] Figure 4a A schematic diagram of a scenario for determining a key task path provided in an embodiment of the present application;
[0024] Figure 4b A schematic diagram of a scenario for determining a key task path provided in an embodiment of the present application;
[0025] Figure 4c A schematic diagram of a scenario for determining a key task path provided in an embodiment of the present application;
[0026] Figure 4d A schematic diagram of a scenario for determining a key task path provided in an embodiment of the present application;
[0027] Figure 4e A schematic diagram of a scenario for determining a key task path provided in an embodiment of the present application;
[0028] Figure 5 A schematic diagram of a first mapping image provided in an embodiment of the present application;
[0029] Figure 6 A schematic diagram of the structure of a task data processing device provided in an embodiment of the present application;
[0030] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0032] The task data processing method proposed in the embodiment of the present application is implemented in an electronic device, which can be a terminal device or a server. Among them, the terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. The server can be an independent server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, but is not limited to this. The present application relates to blockchain technology, and electronic devices can write the data involved, such as task relationship diagrams, task execution information, etc., into the blockchain, so that the electronic device can obtain the required information on the blockchain, such as obtaining the execution information of the associated tasks on the blockchain.
[0033] In some embodiments, the electronic device can execute the task data processing method according to actual business needs to improve the efficiency of abnormal analysis for tasks. The technical solution of the present application can be applied to any abnormal analysis scenario for task scheduling. For example, in the abnormal analysis scenario of system task scheduling, or in the abnormal analysis scenario of database task scheduling, etc. In the applied scenario, the electronic device can determine the key task set of the task to be detected according to the task relationship diagram used to characterize task scheduling, and use the key task set to perform abnormal detection on the task to be detected, so as to realize abnormal analysis of the task to be detected and improve the accuracy of abnormal analysis.
[0034] It is understandable that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0035] Based on the above description, the present application embodiment proposes a task data processing method, which can be executed by the electronic device mentioned above. Figure 1 As shown, the process of the task data processing method of the embodiment of the present application may include the following:
[0036] S101, obtaining a task relationship diagram and a key task set of the task to be detected.
[0037] Among them, the task relationship diagram includes N tasks and the data flow between the N tasks, the key task set includes k key tasks, and the key tasks are connected with the tasks to be detected in the task relationship diagram, that is, the tasks to be detected and the key tasks both belong to N tasks, N is a positive integer, and k is a positive integer less than N. The task to be detected is a task in an abnormal execution state among the N tasks. The task relationship diagram can be used to characterize the scheduling relationship between tasks. And, there is no restriction on the specific type of tasks here, such as data collection tasks for databases, or data analysis tasks for systems, etc.
[0038] In some embodiments, the connection relationship between N tasks can be obtained through the data flow between N tasks. Assume that N tasks include a first task, a second task, and a third task. If the first task has a data flow pointing to the second task, it means that the task result output by the first task is the input of the second task, that is, the first task and the second task have a connection relationship (in this case, a direct connection relationship); if the second task has a data flow pointing to the third task, it means that the task result output by the second task is the input of the third task, that is, the second task and the third task have a connection relationship (in this case, a direct connection relationship); and since the first task and the second task have a connection relationship, and the second task and the third task have a connection relationship, it means that the task result output by the first task will affect the input of the third task, so the first task and the third task also have a connection relationship (in this case, an indirect connection relationship). Among them, the connection relationship between the key task and the task to be detected in the task relationship diagram can be a direct connection relationship or an indirect connection relationship.
[0039] For example, Figure 2 As shown, Figure 2A schematic diagram of a task relationship diagram provided for an embodiment of the present application; wherein the task relationship diagram includes three types of tasks, namely a starting task, an intermediate task and an end task, the starting task is a task that only has a data flow pointing to other tasks, the intermediate task is a task that has a data flow pointing to other tasks and has a data flow pointed to by other tasks, and the end task is a task that does not have a data flow pointing to other tasks and has a data flow pointed to by other tasks; intermediate task 6 has a data flow pointing to end task 2, so intermediate task 6 has a connection relationship (direct connection relationship) with end task 2, intermediate task 4 has a data flow pointing to intermediate task 6, so intermediate task 4 has a connection relationship (direct connection relationship) with intermediate task 6, and intermediate task 6 has a data flow indirectly pointing to end task 2, so intermediate task 4 has a connection relationship (direct connection relationship) with end task 2.
[0040] S102: Determine at least one associated task of the kth key task according to the data flow between the N tasks.
[0041] In some embodiments, the electronic device can determine the target task (or tasks) with data flow pointing to the kth key task from the N tasks according to the data flow between the N tasks, and obtain the at least one associated task according to the target task. That is, taking the target task as at least one associated task means that the at least one associated task is a task that has a direct connection relationship with the kth key task, and the at least one associated task can also be called the predecessor task of the kth key task. For example, Figure 2 As shown, assuming that intermediate task 8 is the kth key task, the at least one associated task obtained is intermediate task 4 and intermediate task 5. In addition, if the kth key task is the starting task, that is, the kth key task does not have a data flow pointed to by other tasks, and therefore the kth key task does not have at least one associated task, then the task to be detected is directly detected for anomalies based on the key task set including the kth key task, that is, step S105 is executed.
[0042] In particular, when determining the first key task, at least one associated task of the task to be detected is determined based on the data flow between the N tasks, that is, a target task with a data flow pointing to the task to be detected is determined from the N tasks, and the target task with the data flow pointing to the task to be detected is used as at least one associated task of the task to be detected, so as to determine the first key task from the at least one associated task of the task to be detected.
[0043] S103: Obtain execution information of each associated task in at least one associated task.
[0044] In some embodiments, the electronic device can obtain execution information of each associated task, which execution information includes the execution time and execution status of the associated task. The execution time includes the start time and end time of the associated task execution, and the execution status includes a completed status or an uncompleted status.
[0045] S104: Determine a target associated task from at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task.
[0046] In some embodiments, determining the target associated task from at least one associated task based on the execution information of each associated task may specifically be, if an associated task with an execution status of an unfinished state is determined as the target associated task; optionally, if there are multiple associated tasks with an execution status of an unfinished state at this time, the target associated task may be further determined based on the execution time of the multiple associated tasks; determining the target associated task based on the execution time of the multiple associated tasks may be, if the associated task has the latest end time and / or the latest start time as the target associated task; or, if the execution status of the associated tasks are all in a completed state, determining the target associated task based on the execution time of the multiple associated tasks may be, if the associated task has the latest end time as the target associated task. Determining the target associated task based on the execution time of the multiple associated tasks may be, if the associated task has the latest end time as the target associated task.
[0047] In some embodiments, after determining the target associated task, the target associated task can be added to the key task set as the k+1th key task. Subsequently, the k+2th key task can be determined based on the method of determining the k+1th key task. The k+1th key task can be one or more. If there are multiple, when determining the k+2th key task, the above method is performed on each k+1th key task to obtain the k+2th associated task associated with each k+1th key task. For example, Figure 2 As shown, let the kth key task be intermediate task 8, and the k+1th key task be intermediate task 4 and intermediate task 5. Therefore, continue to determine the k+2th associated task associated with intermediate task 4 from at least one associated task of intermediate task 4 (intermediate task 1 and intermediate task 2) based on the above method, and determine the k+2th associated task associated with intermediate task 5 from at least one associated task of intermediate task 5 (intermediate task 2, intermediate task 3 and intermediate task 4).
[0048] It can be understood that the kth key task can only be executed when at least one associated task of the kth key task is completed. If the target associated task is in an unfinished state, or has the latest end time among the completed states, then it has the greatest impact on the kth key task. Therefore, the k+1th key task is the predecessor task that has the greatest impact on the execution of the kth key task. Since the k+1th key task is connected to the task to be detected, it is one of the tasks that has the greatest impact on the execution of the task to be detected.
[0049] S105 . Perform anomaly detection on the task to be detected according to the key task set to which the target associated task is added, obtain a detection result for the task to be detected, and send the detection result to the terminal device.
[0050] In some embodiments, it can be understood that the process of determining the key tasks in the present application is a continuously iterative process, and the target associated task of the last key task in the key task set is determined in each round and added to the key task set, that is, the last key task in the key task set is determined in each round, until the last key task in the key task set is the starting task (that is, there is no associated task) in the next round, which means that the key task set of this round is the final key task set, and the electronic device performs anomaly detection on the task to be detected based on the final key task set.
[0051] In some embodiments, the electronic device performs anomaly detection on the task to be detected based on a key task set to which target-associated tasks are added, and obtains a detection result for the task to be detected. Specifically, it may be that based on the execution information of each key task included in the key task set to which target-associated tasks are added, anomaly detection is performed on the task to be detected, and a detection result for the task to be detected is obtained. The detection result for the task to be detected includes the detection result obtained after performing anomaly detection on each key task. Through the detection result, rapid positioning of task anomalies can be achieved, that is, accurate identification of abnormal tasks that affect the task to be detected, and the detection result for the task to be detected is sent to the terminal device, and relevant business personnel perform anomaly repair based on the abnormal tasks indicated by the detection result.
[0052] In an embodiment of the present application, the electronic device can obtain a task relationship diagram and a key task set of the task to be detected, determine at least one associated task of the kth key task according to the data flow between N tasks, obtain the execution information of each associated task in the at least one associated task, determine the target associated task from at least one associated task according to the execution information of each associated task, add the target associated task to the key task set as the k+1th key task, perform anomaly detection on the task to be detected based on the key task set to which the target associated task is added, obtain the detection result for the task to be detected, and send the detection result to the terminal device. By implementing the above method, the execution information of each associated task corresponding to the kth key task of the task to be detected can be obtained. Since the execution status of the associated tasks will affect the execution of the kth key task, the k+1th key task that has a greater impact on the execution of the kth key task can be determined through the execution information, and each key task has a connection relationship with the task to be detected, which will have an impact on the execution of the task to be detected. The key task set of the task to be detected thus obtained can be understood as a key task set that has a greater impact on the task to be detected, and the detection results obtained by using this key task set for anomaly detection are more accurate, which can improve the efficiency of anomaly analysis for tasks.
[0053] See also Figure 3 , Figure 3 The following is a flow chart of a task data processing method provided in an embodiment of the present application, which method can be executed by the electronic device mentioned above. Figure 3 As shown, the process of the task data processing method in the embodiment of the present application may include the following:
[0054] S301, obtaining a task relationship diagram and a key task set of the task to be detected.
[0055] Among them, the task relationship diagram includes N tasks and the data flow between the N tasks, the key task set includes k key tasks, and the key tasks are connected with the tasks to be detected in the task relationship diagram. N is a positive integer, and k is a positive integer less than N.
[0056] In some embodiments, the above-mentioned N tasks may include an endpoint task set and a non-endpoint task set. The endpoint tasks included in the endpoint task set are tasks for which there is no data flow directed to any of the N tasks, and the non-endpoint tasks included in the non-endpoint task set are tasks for which there is data flow directed to any of the N tasks. The non-endpoint tasks of the non-endpoint task set include starting tasks and intermediate tasks.
[0057] In some embodiments, before obtaining the task relationship diagram and the key task set of the task to be detected, the electronic device can determine the task to be detected from the terminal task set. Wherein, determining the task to be detected from the terminal task set can be, obtaining the execution information of each terminal task in the terminal task set, the execution information is the execution information obtained when the terminal task is executed, including the execution time and the execution status, the execution time includes the start time and the end time, and the execution status includes the completed state and the uncompleted state; according to the execution time and the execution status of each terminal task, determine the abnormal terminal task that meets the abnormal execution condition from the terminal task set, and determine the abnormal terminal task as the task to be detected. The electronic device determines the abnormal terminal task that meets the abnormal execution condition from the terminal task set according to the execution time and the execution status of each terminal task, specifically, determining the terminal task whose execution status is uncompleted in the terminal task set as the abnormal terminal task, or determining the terminal task whose execution status is completed in the terminal task set and whose start time and end time meet the abnormal time condition as the abnormal terminal task.
[0058] In some embodiments, the electronic device can obtain the execution information of each terminal task at a target time point (e.g., a fixed time point every day). If an abnormal terminal task is found, an abnormal detection is performed on the abnormal terminal task. The target time point can be set by relevant business personnel according to the actual task scheduling situation. The target time point is based on the normal execution time when the terminal task is in a normal execution state and is in a completed state, that is, when the terminal task is detected at the target time point, the normal terminal task is ensured to be completed. The start time and end time in the execution time are the actual start time and actual end time of the terminal task when it is executed.
[0059] Among them, the above-mentioned determination of whether the start time and the end time meet the abnormal time condition can specifically be to obtain the expected execution time for the endpoint task, the expected execution time includes the expected start time and the expected end time, determine a first time difference between the expected start time and the start time, and determine a second time difference between the expected end time and the end time, and determine whether the abnormal time condition is met based on the first time difference and the second time difference.
[0060] In some embodiments, determining whether the abnormal time condition is met according to the first time difference and the second time difference can be specifically that if the first time difference is greater than or equal to the first time difference threshold and / or the second time difference is greater than or equal to the second time difference threshold, it is determined that the abnormal time condition is met; if the first time difference is less than the first time difference threshold, and the second time difference is less than the second time difference threshold, it is determined that the abnormal time condition is not met. Among them, the expected execution time, the first time difference threshold and the second time difference threshold can be set by relevant business personnel according to experience. It can be understood that the expected execution time is the normal execution time for the terminal task under normal task scheduling of the task relationship diagram, so the actual start time should not differ too much from the expected start time, and the actual end time should not differ too much from the expected end time, that is, if the first time difference is greater than or equal to the first time difference threshold and / or the second time difference is greater than or equal to the second time difference threshold, it means that the actual execution time deviates too much from the expected execution time, and it is very likely that the terminal task has an abnormality.
[0061] S302: Determine at least one associated task of the kth key task according to the data flow between the N tasks.
[0062] In some embodiments, the electronic device can determine at least one associated task of the kth critical task from a non-end task set based on the data flow between N tasks. Specifically, it can be that according to the data flow between N tasks, a target task with a data flow pointing to the kth critical task is determined from the non-end task set, and at least one associated task is obtained based on the target task, that is, the target task is taken as at least one associated task.
[0063] It can be understood that since the process of determining the key tasks is an iterative process, one (specially, there may be multiple) key tasks are determined in each round. When the key task determined in this round is the starting task, that is, there is no preceding key task, it means that the key task set for the task to be detected is determined to be completed, and the final key task set is obtained, and the final key task set is used for anomaly detection.
[0064] S303: Obtain execution information of each associated task in at least one associated task.
[0065] S304: Determine a target associated task from at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task. The specific implementation of steps S303-S304 can refer to the relevant description of the above embodiment.
[0066] S305 : Generate a key task path for the task to be detected according to the key task set to which the target-related task is added.
[0067] In some embodiments, the electronic device can generate a critical task path for the task to be detected based on the data flow between each critical task included in the critical task set (i.e., the final critical task set, where the target associated task is the starting task) to which the target associated task is added. The critical task path is a subgraph containing critical tasks in the task relationship graph. The critical task path can be understood as the key reason for the abnormality of the task to be detected, so an abnormality analysis based on the critical task path can quickly locate the abnormal point.
[0068] For example, Figure 4a-4e As shown, Figure 4a-4e A schematic diagram of a scenario for determining a key task path provided in an embodiment of the present application; wherein, assuming that the task relationship diagram is as follows Figure 2 As shown, and assuming that the end task 1 is the task to be detected, that is:
[0069] (1) First round: At least one of the associated tasks of the end task 1 is intermediate task 6, intermediate task 7, and intermediate task 8. If the determined target associated task is intermediate task 7, then the intermediate task 7 is added to the key task set as the first key task, such as Figure 4a ;
[0070] (2) In the second round, at least one of the associated tasks of the first key task is intermediate task 1 and intermediate task 5. If the determined target associated task is intermediate task 5, then intermediate task 5 is added to the key task set as the second key task, such as Figure 4b ;
[0071] (3) In the third round, at least one of the associated tasks of the second key task is intermediate task 2 and intermediate task 3. If the determined target associated tasks are intermediate task 2 and intermediate task 3, then intermediate task 2 and intermediate task 3 are added to the key task set as the third key task (including two intermediate tasks). Figure 4c ;
[0072] (4) In the fourth round, at least one associated task of intermediate task 2 (starting task 2, starting task 3 and starting task 4) and at least one associated task of intermediate task 3 (starting task 1) are determined respectively. If the target associated task for intermediate task 2 is determined to be starting task 2, starting task 2 is added to the key task set as the fourth key task for intermediate task 2. If the target associated task for intermediate task 3 is determined to be starting task 1, starting task 1 is added to the key task set as the fourth key task for intermediate task 3. At this time, the fourth key task for intermediate task 2 and the fourth key task for intermediate task 3 are both starting tasks. The key task set with the target associated tasks added is the final key task set (intermediate task 7, intermediate task 5, intermediate task 2, intermediate task 3, starting task 2 and starting task 1). Figure 4d ;
[0073] (5) Based on the data flow between the key tasks included in the final key task set, generate the key task path for the task to be detected, such as Figure 4e .
[0074] S306 , performing abnormality detection on the tasks to be detected based on the execution information of each key task in the order of key tasks indicated by the key task path, obtaining detection results for the tasks to be detected, and sending the detection results to the terminal device.
[0075] In some embodiments, the electronic device performs anomaly detection on each key task based on the execution information of each key task in the order of key tasks indicated by the associated task path, obtains the detection result corresponding to each key task, and uses the detection result corresponding to each key task as the detection result for the task to be detected, and sends the detection result to the terminal device to repair the anomaly of the task to be detected based on the detection result.
[0076] In one possible implementation, the electronic device performs anomaly detection on each key task based on the execution information of each key task, and obtains the detection result corresponding to each key task. Specifically, the first key task whose execution status is in an unfinished state is detected for anomaly according to a first detection rule, and the detection result corresponding to the first key task is obtained; the second key task whose execution status is a completed state is detected for anomaly according to a second detection rule, and the detection result corresponding to the second key task is obtained.
[0077] In some embodiments, performing anomaly detection on a first key task whose execution status is in an unfinished state according to a first detection rule, and obtaining a detection result corresponding to the first key task may be generating a detection result indicating that an execution anomaly exists in the first key task.
[0078] In some embodiments, the target key task is assumed to be any key task in each key task (i.e., the second key task) whose execution status is completed; therefore, an abnormality detection is performed on the second key task in the execution status of the completed state according to the second detection rule, and the detection result corresponding to the second key task can be obtained by obtaining multiple historical execution times of the target key task, the historical execution time includes a historical start time and a historical end time, and the historical start time and the historical end time do not meet the abnormal time condition, that is, what is obtained is the historical execution time of the target key task in a normal execution state; a first time mapping image is generated according to the start time and multiple historical start times included in the execution information of the target key task, the first time mapping image includes a mapping point corresponding to the start time and mapping points corresponding to multiple historical start times; a second time mapping image is generated according to the end time and multiple historical end times included in the execution information of the target key task, the second time mapping image includes a mapping point corresponding to the end time and mapping points corresponding to multiple historical end times, and the detection result corresponding to the target key task is determined according to the first time mapping image and the second time mapping image.
[0079] In some embodiments, the electronic device determines the detection result corresponding to the target key task based on the first time mapping image and the second time mapping image. Specifically, the first reasonable area is determined in the first time mapping image based on the mapping points corresponding to the multiple historical start times, and the second reasonable area is determined in the second time mapping image based on the mapping points corresponding to the multiple historical end times. If the mapping point corresponding to the start time is in the first reasonable area, and the mapping point corresponding to the end time is in the second reasonable area, then a detection result indicating that there is no execution abnormality in the target key task is generated; if the mapping point corresponding to the start time is not in the first reasonable area, and / or the mapping point corresponding to the end time is not in the second reasonable area, then a detection result indicating that there is an execution abnormality in the target key task is generated. It can be understood that if the mapping point corresponding to the start time is not in the first reasonable area, it means that the start time is an abnormal start time, and if the mapping point corresponding to the end time is not in the second reasonable area, it means that the end time is an abnormal end time.
[0080] Among them, the electronic device can construct a mapping relationship between time and mapping image, and map the start time and multiple historical start times to the first mapping image according to the mapping relationship to obtain the first time mapping image, and map the end time to the multiple historical end times to the second mapping image according to the mapping relationship to obtain the second time mapping image. The electronic device can determine the first reasonable area in the first time mapping image by taking the mapping sub-area formed by the mapping points corresponding to the multiple historical start times as the first reasonable area, or predicting the first reasonable area according to the mapping points corresponding to the multiple historical start times, such as inputting the location information (such as coordinate data) of the mapping points corresponding to the multiple historical start times into the regional prediction model, and the regional prediction model outputs the mapping image including the first reasonable area, and then it can be determined whether the start time is in the first reasonable area according to the mapping image of the first reasonable area and the first time mapping image, and the regional prediction model is obtained by pre-training the regional prediction model to be trained using the sample set (the location information of the sample mapping points and the corresponding sample reasonable area). The specific method of determining the second reasonable area in the second time mapping image can be the same as the specific method of determining the first reasonable area.
[0081] For example, Figure 5 As shown, Figure 5 A schematic diagram of a first mapping image provided in an embodiment of the present application, wherein a solid black origin is a mapping point corresponding to a historical start time, a hollow black origin is a mapping point corresponding to a start time, and a first reasonable area is obtained by a mapping sub-area formed by the mapping points corresponding to the multiple historical start times. At this time, if the mapping point corresponding to the start time is not in the first reasonable area, it indicates that the start time is an abnormal start time.
[0082] In an embodiment of the present application, the electronic device can obtain a task relationship diagram and a key task set of the task to be detected, determine at least one associated task of the kth key task according to the data flow between N tasks, obtain the execution information of each associated task in the at least one associated task, determine the target associated task from at least one associated task according to the execution information of each associated task, add the target associated task to the key task set as the k+1th key task, generate a key task path for the task to be detected according to the key task set with the target key task added, perform anomaly detection on the task to be detected based on the execution information of each key task in the order of key tasks indicated by the key task path, obtain the detection result for the task to be detected, and send the detection result to the terminal device. By implementing the above method, the execution information of each associated task corresponding to the kth key task of the task to be detected can be obtained. Since the execution status of the associated tasks will affect the execution of the kth key task, the k+1th key task that has a greater impact on the execution of the kth key task can be determined through the execution information, and each key task has a connection relationship with the task to be detected, which will have an impact on the execution of the task to be detected. The key task set of the task to be detected thus obtained can be understood as a key task set that has a greater impact on the task to be detected, and the detection results obtained by using this key task set for anomaly detection are more accurate, which can improve the efficiency of anomaly analysis for tasks.
[0083] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a task data processing device provided in this application. It should be noted that: Figure 6 The task data processing device shown is used to execute the application Figure 1 and Figure 3 For the convenience of explanation, only the part related to the embodiment of the present application is shown, and the specific technical details are not disclosed. Please refer to the present application for details. Figure 1 and Figure 3 The task data processing device 600 may include: an acquisition module 601, a determination module 602, and a detection module 603. Among them:
[0084] The acquisition module 601 is used to acquire a task relationship diagram and a key task set of the task to be detected; the task relationship diagram includes N tasks and data flows between the N tasks, the key task set includes k key tasks, and the key tasks are connected to the task to be detected in the task relationship diagram, N is a positive integer, and k is a positive integer less than N;
[0085] A determination module 602, configured to determine at least one associated task of the kth key task according to the data flow between the N tasks;
[0086] The acquisition module 601 is further used to acquire execution information of each associated task in the at least one associated task;
[0087] The determining module 602 is further configured to determine a target associated task from the at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task;
[0088] The detection module 603 is used to perform anomaly detection on the task to be detected according to the key task set to which the target associated task is added, obtain a detection result for the task to be detected, and send the detection result to the terminal device.
[0089] In a possible implementation, the N tasks include an endpoint task set and a non-endpoint task set; the endpoint tasks included in the endpoint task set are tasks that do not have a data flow to any of the N tasks; the non-endpoint tasks included in the non-endpoint task set are tasks that have a data flow to any of the N tasks;
[0090] Before being used to obtain the task relationship diagram and the key task set of the task to be detected, the acquisition module 601 is also used to:
[0091] Determine the task to be detected from the endpoint task set;
[0092] When the determination module 602 is used to determine at least one associated task of the kth key task according to the data flow between the N tasks, it is specifically used to:
[0093] According to the data flow between the N tasks, at least one associated task of the kth critical task is determined from the non-endpoint task set.
[0094] In a possible implementation manner, when the acquisition module 601 is used to determine the task to be detected from the endpoint task set, it is specifically used to:
[0095] Acquire execution information of each endpoint task in the endpoint task set; the execution information includes execution time and execution status, the execution time includes start time and end time, and the execution status includes completion status or uncompleted status;
[0096] Determine the terminal task in the terminal task set whose execution status is in an unfinished state as the task to be detected; or,
[0097] The endpoint tasks in the endpoint task set whose execution status is completed and whose start time and end time meet the abnormal time condition are determined as the tasks to be detected.
[0098] In a possible implementation manner, when the determining module 602 is used to determine at least one associated task of the kth critical task from the non-endpoint task set according to the data flow between the N tasks, it is specifically used to:
[0099] According to the data flow between the N tasks, determining a target task having a data flow pointing to the kth key task from the non-endpoint task set;
[0100] The at least one associated task is obtained according to the target task.
[0101] In a possible implementation, the detection module 603 is used to perform anomaly detection on the task to be detected according to the key task set to which the target associated task is added, and obtain a detection result for the task to be detected, specifically for:
[0102] Generate a key task path for the task to be detected according to the data flow between each key task included in the key task set to which the target associated task is added;
[0103] According to the key task sequence indicated by the key task path, each key task is respectively subjected to anomaly detection based on the execution information of each key task in turn, to obtain a detection result corresponding to each key task;
[0104] The detection result corresponding to each key task is used as the detection result for the task to be detected.
[0105] In a possible implementation, the detection module 603 is used to perform anomaly detection on each key task based on the execution information of each key task to obtain the detection result corresponding to each key task, specifically for:
[0106] Acquire multiple historical execution times of the target key task; the historical execution time includes the historical start time and the historical end time, and the historical start time and the historical end time do not meet the abnormal time condition; the target key task is any key task whose execution status is completed among each key task;
[0107] Generate a first time mapping image according to the start time included in the execution information of the target key task and the multiple historical start times; the first time mapping image includes a mapping point corresponding to the start time and mapping points corresponding to the multiple historical start times;
[0108] Generate a second time mapping image according to the end time included in the execution information of the target key task and the multiple historical end times; the second time mapping image includes a mapping point corresponding to the end time and mapping points corresponding to the multiple historical end times;
[0109] A detection result corresponding to the target key task is determined according to the first time mapping image and the second time mapping image.
[0110] In a possible implementation manner, when the detection module 603 is used to determine the detection result corresponding to the target key task according to the first time mapping image and the second time mapping image, it is specifically used to:
[0111] Determining a first reasonable area in the first time mapping image according to the mapping points corresponding to the multiple historical start times;
[0112] Determining a second reasonable area in the second time mapping image according to the mapping points corresponding to the multiple historical end times;
[0113] If the mapping point corresponding to the start time is in the first reasonable area, and the mapping point corresponding to the end time is in the second reasonable area, a detection result indicating that there is no execution abnormality in the target key task is generated;
[0114] If the mapping point corresponding to the start time is not in the first reasonable area, and / or the mapping point corresponding to the end time is not in the second reasonable area, a detection result indicating that an execution abnormality exists in the target key task is generated.
[0115] In an embodiment of the present application, an acquisition module acquires a task relationship diagram and a key task set of a task to be detected; a determination module determines at least one associated task of the kth key task according to the data flow between N tasks; the acquisition module acquires execution information of each associated task in at least one associated task; the determination module determines a target associated task from at least one associated task according to the execution information of each associated task, and adds the target associated task to the key task set as the k+1th key task; the detection module performs an abnormality detection on the task to be detected according to the key task set to which the target associated task is added, obtains a detection result for the task to be detected, and sends the detection result to the terminal device. Through the above scheme, the execution information of each associated task corresponding to the kth key task of the task to be detected can be obtained. Since the execution status of the associated tasks will affect the execution of the kth key task, the k+1th key task that has a greater impact on the execution of the kth key task can be determined through the execution information, and each key task has a connection relationship with the task to be detected, which will have an impact on the execution of the task to be detected. The key task set of the task to be detected obtained in this way can be understood as a key task set that has a greater impact on the task to be detected, and the detection result obtained by using this key task set for anomaly detection is more accurate, which can improve the efficiency of anomaly analysis for tasks.
[0116] Each functional module in each embodiment of the present application can be integrated into one module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules, which is not limited in the present application.
[0117] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes: at least one processor 701 and a memory 702. Optionally, the electronic device may further include a network interface. The processor 701, the memory 702 and the network interface may exchange data, the network interface is controlled by the processor 701 to send and receive messages, the memory 702 is used to store a computer program, the computer program includes program instructions, and the processor 701 is used to execute the program instructions stored in the memory 702. The processor 701 is configured to call the program instructions to execute the above method.
[0118] The memory 702 may include a volatile memory, such as a random-access memory (RAM); the memory 702 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the memory 702 may also include a combination of the above-mentioned types of memory.
[0119] The processor 701 may be a central processing unit (CPU). In one embodiment, the processor 701 may also be a graphics processing unit (GPU). The processor 701 may also be a combination of a CPU and a GPU.
[0120] In a possible implementation, the memory 702 is used to store program instructions, and the processor 701 may call the program instructions to perform the following steps:
[0121] Obtain a task relationship diagram and a key task set of the task to be detected; the task relationship diagram includes N tasks and data flows between the N tasks, the key task set includes k key tasks, and the key tasks are connected to the task to be detected in the task relationship diagram, N is a positive integer, and k is a positive integer less than N;
[0122] Determine at least one associated task of the kth key task according to the data flow between the N tasks;
[0123] Acquire execution information of each associated task in the at least one associated task;
[0124] Determine a target associated task from the at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task;
[0125] According to the key task set to which the target associated task is added, anomaly detection is performed on the task to be detected, a detection result for the task to be detected is obtained, and the detection result is sent to the terminal device.
[0126] In a possible implementation, the N tasks include an endpoint task set and a non-endpoint task set; the endpoint tasks included in the endpoint task set are tasks that do not have a data flow to any of the N tasks; the non-endpoint tasks included in the non-endpoint task set are tasks that have a data flow to any of the N tasks;
[0127] Before being used for obtaining the task relationship graph and the key task set of the task to be detected, the processor 701 is further used for:
[0128] Determine the task to be detected from the endpoint task set;
[0129] When the processor 701 is used to determine at least one associated task of the kth key task according to the data flow between the N tasks, it is specifically used to:
[0130] According to the data flow between the N tasks, at least one associated task of the kth critical task is determined from the non-endpoint task set.
[0131] In a possible implementation manner, when the processor 701 is used to determine the task to be detected from the endpoint task set, it is specifically used to:
[0132] Acquire execution information of each endpoint task in the endpoint task set; the execution information includes execution time and execution status, the execution time includes start time and end time, and the execution status includes completion status or uncompleted status;
[0133] Determine the terminal task in the terminal task set whose execution status is in an unfinished state as the task to be detected; or,
[0134] The endpoint tasks in the endpoint task set whose execution status is completed and whose start time and end time meet the abnormal time condition are determined as the tasks to be detected.
[0135] In a possible implementation manner, when the processor 701 is used to determine at least one associated task of the kth critical task from the non-endpoint task set according to the data flow between the N tasks, it is specifically used to:
[0136] According to the data flow between the N tasks, determining a target task having a data flow pointing to the kth key task from the non-endpoint task set;
[0137] The at least one associated task is obtained according to the target task.
[0138] In a possible implementation manner, the processor 701, when used to perform anomaly detection on the task to be detected according to the key task set to which the target associated task is added, and obtain a detection result for the task to be detected, is specifically used to:
[0139] Generate a key task path for the task to be detected according to the data flow between each key task included in the key task set to which the target associated task is added;
[0140] According to the key task sequence indicated by the key task path, each key task is respectively subjected to anomaly detection based on the execution information of each key task in turn, to obtain a detection result corresponding to each key task;
[0141] The detection result corresponding to each key task is used as the detection result for the task to be detected.
[0142] In a possible implementation manner, the processor 701, when used to perform anomaly detection on each key task based on the execution information of each key task and obtain a detection result corresponding to each key task, is specifically used to:
[0143] Acquire multiple historical execution times of the target key task; the historical execution time includes the historical start time and the historical end time, and the historical start time and the historical end time do not meet the abnormal time condition; the target key task is any key task whose execution status is completed among each key task;
[0144] Generate a first time mapping image according to the start time included in the execution information of the target key task and the multiple historical start times; the first time mapping image includes a mapping point corresponding to the start time and mapping points corresponding to the multiple historical start times;
[0145] Generate a second time mapping image according to the end time included in the execution information of the target key task and the multiple historical end times; the second time mapping image includes a mapping point corresponding to the end time and mapping points corresponding to the multiple historical end times;
[0146] A detection result corresponding to the target key task is determined according to the first time mapping image and the second time mapping image.
[0147] In a possible implementation manner, when the processor 701 is used to determine the detection result corresponding to the target key task according to the first time mapping image and the second time mapping image, it is specifically used to:
[0148] Determining a first reasonable area in the first time mapping image according to the mapping points corresponding to the multiple historical start times;
[0149] Determining a second reasonable area in the second time mapping image according to the mapping points corresponding to the multiple historical end times;
[0150] If the mapping point corresponding to the start time is in the first reasonable area, and the mapping point corresponding to the end time is in the second reasonable area, a detection result indicating that there is no execution abnormality in the target key task is generated;
[0151] If the mapping point corresponding to the start time is not in the first reasonable area, and / or the mapping point corresponding to the end time is not in the second reasonable area, a detection result indicating that an execution abnormality exists in the target key task is generated.
[0152] In a specific implementation, the device, processor 701, memory 702, etc. described in the embodiments of the present application can execute the implementation described in the above method embodiments, and can also execute the implementation described in the embodiments of the present application, which will not be repeated here.
[0153] A computer (readable) storage medium is also provided in an embodiment of the present application, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor can execute some or all of the steps performed in the above method embodiment. Optionally, the computer storage medium may be volatile or non-volatile. The computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of blockchain nodes, etc.
[0154] Among them, the blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0155] An embodiment of the present application provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it can implement some or all of the steps in the above method, which are not described in detail here.
[0156] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0157] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer storage medium, which can be a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0158] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A task data processing method, characterized in that: The method comprises: Obtain a task relationship diagram and a key task set of the task to be detected; the task relationship diagram includes N tasks and data flows between the N tasks, the key task set includes k key tasks, and the key tasks are connected to the task to be detected in the task relationship diagram, N is a positive integer, and k is a positive integer less than N; Determine at least one associated task of the kth key task according to the data flow between the N tasks; Acquire execution information of each associated task in the at least one associated task; Determine a target associated task from the at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task; Generate a key task path for the task to be detected according to the data flow between each key task included in the key task set to which the target associated task is added; According to the key task sequence indicated by the key task path, each key task is respectively detected for anomalies based on the execution information of each key task, so as to obtain the detection result corresponding to each key task; wherein each key task is respectively detected for anomalies based on the execution information of each key task, so as to obtain the detection result corresponding to each key task, including: Acquire multiple historical execution times of the target key task; the historical execution time includes the historical start time and the historical end time, and the historical start time and the historical end time do not meet the abnormal time condition; the target key task is any key task whose execution status is completed among each key task; Generate a first time mapping image according to the start time included in the execution information of the target key task and the multiple historical start times; the first time mapping image includes a mapping point corresponding to the start time and mapping points corresponding to the multiple historical start times; Generate a second time mapping image according to the end time included in the execution information of the target key task and the multiple historical end times; the second time mapping image includes a mapping point corresponding to the end time and mapping points corresponding to the multiple historical end times; Determine a detection result corresponding to the target key task according to the first time mapping image and the second time mapping image; The detection result corresponding to each key task is used as the detection result for the task to be detected, and the detection result is sent to the terminal device.
2. The method according to claim 1, characterized in that The N tasks include an end task set and a non-end task set; the end tasks included in the end task set are tasks that do not have data flows pointing to any of the N tasks; the non-end tasks included in the non-end task set are tasks that have data flows pointing to any of the N tasks; Before obtaining the task relationship graph and the key task set of the task to be detected, the method further includes: Determine the task to be detected from the endpoint task set; The step of determining at least one associated task of the kth key task according to the data flow between the N tasks includes: According to the data flow between the N tasks, at least one associated task of the kth critical task is determined from the non-endpoint task set.
3. The method according to claim 2, characterized in that The step of determining the task to be detected from the endpoint task set includes: Acquire execution information of each endpoint task in the endpoint task set; the execution information includes execution time and execution status, the execution time includes start time and end time, and the execution status includes completion status or uncompleted status; Determine the terminal task in the terminal task set whose execution status is in an unfinished state as the task to be detected; or, The endpoint tasks in the endpoint task set whose execution status is completed and whose start time and end time meet the abnormal time condition are determined as the tasks to be detected.
4. The method according to claim 2, characterized in that: The step of determining at least one associated task of the kth key task from the non-endpoint task set according to the data flow between the N tasks includes: According to the data flow between the N tasks, determining a target task having a data flow pointing to the kth key task from the non-endpoint task set; The at least one associated task is obtained according to the target task.
5. The method according to claim 1, characterized in that The determining the detection result corresponding to the target key task according to the first time mapping image and the second time mapping image includes: Determining a first reasonable area in the first time mapping image according to the mapping points corresponding to the multiple historical start times; Determining a second reasonable area in the second time mapping image according to the mapping points corresponding to the multiple historical end times; If the mapping point corresponding to the start time is in the first reasonable area, and the mapping point corresponding to the end time is in the second reasonable area, a detection result indicating that there is no execution abnormality in the target key task is generated; If the mapping point corresponding to the start time is not in the first reasonable area, and / or the mapping point corresponding to the end time is not in the second reasonable area, a detection result indicating that an execution abnormality exists in the target key task is generated.
6. A task data processing device, characterized in that: The device is used to execute the task data processing method according to any one of claims 1 to 5, and the device comprises: An acquisition module is used to acquire a task relationship diagram and a key task set of the task to be detected; the task relationship diagram includes N tasks and data flows between the N tasks, the key task set includes k key tasks, and the key tasks are connected to the task to be detected in the task relationship diagram, N is a positive integer, and k is a positive integer less than N; A determination module, configured to determine at least one associated task of the kth key task according to the data flow between the N tasks; The acquisition module is further used to acquire execution information of each associated task in the at least one associated task; The determining module is further configured to determine a target associated task from the at least one associated task according to the execution information of each associated task, and add the target associated task to the key task set as the k+1th key task; The detection module is used to perform anomaly detection on the task to be detected according to the key task set to which the target associated task is added, obtain the detection result for the task to be detected, and send the detection result to the terminal device.
7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.
