Method and system for detecting abnormal job flow of cloud platform

Abstract

The invention discloses an abnormal job flow detection method and system for a cloud platform. The method comprises the following steps: acquiring job flow information and job flow operation information of the cloud platform; classifying the job flow information to obtain a plurality of types of job flows; for each type of job flow, whether the type of job flow is abnormal or not is judged according to the job flow operation information; if yes, generating abnormal job flow details according to the category job flow; inputting the abnormal job flow details into a pre-trained abnormal job flow classification model to obtain abnormal types corresponding to the abnormal job flow details; and feeding back the abnormal job flow details and the corresponding abnormal types to the front end. Therefore, abnormal detection is not manually carried out on the job flow any more, whether the job flow is abnormal or not is monitored through abnormal job flow monitoring, the abnormal detection efficiency is effectively improved, the job flow is classified according to the service logic requirement, and the difference early warning rule of the model is formulated so that the abnormal type of the job flow can be effectively obtained.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for detecting abnormal operation flow on a cloud platform. Background Art

[0002] With the continuous development of banking business, the number of offline tasks on the big data cloud platform is also increasing. In the process of big data job flow scheduling, some abnormal offline tasks often appear in the queue, resulting in excessive resource occupation, which not only wastes a lot of computing resources, but also may cause delays in data updates. Therefore, it is urgent to detect abnormal job flows on the cloud platform.

[0003] In the existing technology, the main method is to locate the abnormal operation flow when the business personnel find that the data is not updated in time when putting data into the cloud platform. Although the feedback from the business personnel can help locate the problem, its timeliness is poor, and it usually takes a long time to find the abnormality. In addition, on the big data cloud platform, the accumulation of historical data tracing tasks can easily lead to the blockage of the operation flow, which in turn affects the timeliness of data update. Summary of the invention

[0004] Based on the above-mentioned deficiencies of the prior art, the present application provides a method and system for detecting abnormal workflows on a cloud platform to solve the problem of untimely detection of abnormalities caused by the prior art.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] The first aspect of the present application provides a method for detecting abnormal operation flow of a cloud platform, comprising:

[0007] Obtain the job flow information and job flow operation information of the cloud platform;

[0008] Classifying the job flow information to obtain multiple categories of job flows;

[0009] For each of the category operation flows, judging whether the category operation flow is abnormal according to the operation flow running information;

[0010] If the category operation flow is abnormal, generating abnormal operation flow details according to the category operation flow;

[0011] Inputting the abnormal operation flow details into a pre-trained abnormal operation flow classification model to obtain the abnormal type corresponding to the abnormal operation flow details; wherein the abnormal operation flow classification model is pre-trained using historical abnormal operation flow details;

[0012] Feedback the abnormal operation flow details and the corresponding abnormal type to the front end.

[0013] Optionally, in the above-mentioned abnormal job flow detection method of the cloud platform, the step of determining, for each of the category job flows, whether the category job flow is abnormal according to the job flow running information, includes:

[0014] For each of the category work flows, extracting update information corresponding to the category work flow from the work flow running information;

[0015] According to the update information corresponding to the category job flow, determining whether the data in the category job flow is updated on time;

[0016] If the data in the category operation flow is updated on time, it is determined that there is no abnormality in the category operation flow;

[0017] If the data in the category workflow is not updated on time, it is determined that there is an abnormality in the category workflow.

[0018] Optionally, in the above-mentioned abnormal workflow detection method of the cloud platform, the training method of the abnormal workflow classification model includes:

[0019] Obtain details of multiple historical abnormal job flows;

[0020] Input each of the historical abnormal operation flow details into the abnormal operation flow classification model to obtain the abnormal type corresponding to each of the historical abnormal operation flow details;

[0021] For each of the historical abnormal operation flow details, determining whether the abnormality type corresponding to the historical abnormal operation flow detail is consistent with the actual abnormality type corresponding to the historical abnormal operation flow detail;

[0022] If the abnormal type corresponding to the historical abnormal operation flow details is consistent with the actual abnormal type corresponding to the historical abnormal operation flow details, determining that the abnormal operation flow classification model is a trained abnormal operation flow classification model;

[0023] If the exception type corresponding to the historical abnormal workflow details does not conform to the actual exception type corresponding to the historical abnormal workflow details, the parameters of the abnormal workflow classification model are adjusted, and the step of inputting each of the historical abnormal workflow details into the abnormal workflow classification model is returned to execute respectively, so as to obtain the exception type corresponding to each of the historical abnormal workflow details.

[0024] Optionally, in the above-mentioned abnormal operation flow detection method of the cloud platform, it also includes:

[0025] When the abnormal type corresponding to the abnormal operation flow details is a resource overlimit type, determining whether there is a dependent operation flow in the category operation flow corresponding to the abnormal operation flow details;

[0026] If there is no dependent workflow in the category workflow corresponding to the abnormal workflow detail, executing the restart strategy of the category workflow;

[0027] If there is a dependent workflow in the category workflow corresponding to the abnormal workflow detail, detecting whether the state of the dependent workflow is a completed state;

[0028] If the state of the dependent job flow is the completed state, executing the restart strategy of the category job flow;

[0029] If the state of the dependent job flow is an unfinished state, recursively perform the process on the category job flow and the dependent job flow.

[0030] Optionally, in the above-mentioned abnormal operation flow detection method of the cloud platform, it also includes:

[0031] When the abnormal type corresponding to the abnormal operation flow details is a restart abnormal type, writing the category operation flow corresponding to the abnormal operation flow details into the whitelist;

[0032] Determine whether there is a record of unsuccessful execution of the category of job flow within a preset time period;

[0033] If there is a record of unsuccessful execution of the category of job flows within a preset time period, an alarm mechanism is triggered.

[0034] The second aspect of the present application provides an abnormal operation flow detection system for a cloud platform, comprising:

[0035] An information acquisition unit, used to acquire the job flow information and job flow running information of the cloud platform;

[0036] A classification unit, used for classifying the job flow information to obtain a plurality of categories of job flows;

[0037] An abnormality judgment unit is used to judge whether there is an abnormality in each of the category operation flows according to the operation flow running information;

[0038] A generating unit, configured to generate abnormal operation flow details according to the category operation flow if there is an abnormality in the category operation flow;

[0039] A data input unit, used for inputting the abnormal operation flow details into a pre-trained abnormal operation flow classification model to obtain the abnormal type corresponding to the abnormal operation flow details; wherein the abnormal operation flow classification model is pre-trained using historical abnormal operation flow details;

[0040] The feedback unit is used to feed back the abnormal operation flow details and the corresponding abnormal type to the front end.

[0041] Optionally, in the abnormal operation flow detection system of the cloud platform, the abnormality judgment unit includes:

[0042] An extraction unit, configured to extract, for each of the category workflows, update information corresponding to the category workflow from the workflow running information;

[0043] A first judging unit, configured to judge whether the data in the category job flow is updated on time according to the update information corresponding to the category job flow;

[0044] A first determining unit, configured to determine that there is no abnormality in the category operation flow if the data in the category operation flow is updated on time;

[0045] The second determining unit is configured to determine that an abnormality exists in the category workflow if the data in the category workflow is not updated on time.

[0046] Optionally, in the above-mentioned abnormal operation flow detection system of the cloud platform, it also includes:

[0047] An acquisition unit, used to acquire details of multiple historical abnormal operation flows;

[0048] An input unit, used to input each of the historical abnormal operation flow details into the abnormal operation flow classification model to obtain the abnormal type corresponding to each of the historical abnormal operation flow details;

[0049] A second judgment unit is used to judge, for each of the historical abnormal operation flow details, whether the abnormality type corresponding to the historical abnormal operation flow detail is consistent with the actual abnormality type corresponding to the historical abnormal operation flow detail;

[0050] a third determining unit, configured to determine that the abnormal operation flow classification model is a trained abnormal operation flow classification model if the abnormal type corresponding to the historical abnormal operation flow details matches the actual abnormal type corresponding to the historical abnormal operation flow details;

[0051] An adjustment unit is used to adjust the parameters of the abnormal workflow classification model if the abnormal type corresponding to the historical abnormal workflow details does not conform to the actual abnormal type corresponding to the historical abnormal workflow details, and return to execute the step of inputting each of the historical abnormal workflow details into the abnormal workflow classification model to obtain the abnormal type corresponding to each of the historical abnormal workflow details.

[0052] Optionally, in the above-mentioned abnormal operation flow detection system of the cloud platform, it also includes:

[0053] A third judgment unit is used to judge whether there is a dependent workflow in the category workflow corresponding to the abnormal workflow detail when the abnormal type corresponding to the abnormal workflow detail is a resource overlimit type;

[0054] A first execution unit, configured to execute a restart strategy of the category workflow if there is no dependent workflow in the category workflow corresponding to the abnormal workflow detail;

[0055] a detection unit, configured to detect whether a state of a dependent workflow is a completed state if there is a dependent workflow in the category workflow corresponding to the abnormal workflow detail;

[0056] A second execution unit, configured to execute a restart strategy of the category workflow if the state of the dependent workflow is a completed state;

[0057] The recursive unit is used to recursively perform the process on the category workflow and the dependent workflow if the state of the dependent workflow is an unfinished state.

[0058] Optionally, the abnormal operation flow detection system of the cloud platform further includes:

[0059] A writing unit, configured to write the category of the operation flow corresponding to the abnormal operation flow details into the whitelist when the abnormal type corresponding to the abnormal operation flow details is a restart abnormal type;

[0060] A fourth determination unit, configured to determine whether there is a record of unsuccessful execution of the category operation flow within a preset time period;

[0061] The trigger unit is used to trigger an alarm mechanism if there is a record of unsuccessful execution of the category operation flow within a preset time period.

[0062] The present application provides a method for detecting abnormal workflows on a cloud platform. The method obtains the workflow information and workflow operation information of the cloud platform, and then classifies the workflow information to obtain multiple categories of workflows. Then, for each category of workflow, the method determines whether the category of workflow is abnormal based on the workflow operation information. If the category of workflow is abnormal, the abnormal workflow details are generated based on the category of workflow. Then, the abnormal workflow details are input into a pre-trained abnormal workflow classification model to obtain the abnormal type corresponding to the abnormal workflow details. The abnormal workflow classification model is pre-trained using historical abnormal workflow details, and finally the abnormal workflow details and their corresponding abnormal types are fed back to the front end. Therefore, the workflow is no longer detected manually for abnormalities, but the workflow is monitored for abnormalities through abnormal workflow monitoring, which effectively improves the efficiency of abnormal detection. The workflow is classified according to business logic requirements, and differential warning rules of the model are formulated to effectively obtain the abnormal type of the workflow. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0064] Figure 1 A flowchart of a method for detecting abnormal operation flow on a cloud platform provided in an embodiment of the present application;

[0065] Figure 2 A flowchart of a method for detecting anomalies in a category of job flows provided in an embodiment of the present application;

[0066] Figure 3 A flowchart of a method for training an abnormal workflow classification model provided in an embodiment of the present application;

[0067] Figure 4 A flowchart of a method for correcting abnormal operation flow details provided in another embodiment of the present application;

[0068] Figure 5 A flowchart of another method for correcting abnormal operation flow details provided in another embodiment of the present application;

[0069] Figure 6 A schematic diagram of the structure of an abnormal workflow detection system for a cloud platform provided in another embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0072] The present application embodiment provides a method for detecting abnormal operation flow of a cloud platform, such as Figure 1 As shown, the specific steps include:

[0073] S101, obtaining the job flow information and job flow running information of the cloud platform.

[0074] Specifically, the job flow information and the job flow operation information can be periodically obtained from the database of the cloud platform, so as to know whether there is an abnormal job flow in the cloud platform based on the job flow information and the job flow operation information, thereby affecting the stability of the cloud platform.

[0075] Among them, job flow information usually refers to metadata or configuration data related to the job flow itself, which mainly describes the overall structure, logical relationships, dependencies, etc. of the job flow. A job flow is a set of tasks composed of multiple jobs in a specific order and under specific conditions. These jobs usually have dependencies or need to be scheduled and executed at a predetermined time.

[0076] The operation flow running information refers to the dynamic status and related data recorded by the system during the actual execution of the operation flow. This information reflects the actual situation of the execution progress, status, exception handling, etc. of the operation flow.

[0077] S102: Classify the job flow information to obtain multiple categories of job flows.

[0078] It should be noted that according to the differentiated naming rules of the job flows of the data cloud platform and the maximum scheduling frequency of each job in the job flow, the job flows are classified into daily, monthly, quarterly, and annual categories. Therefore, in order to obtain whether the job flow information is not running according to the cycle and to determine whether the job flow is abnormal, the job flow information can be classified according to the scheduling frequency, thereby obtaining multiple categories of job flows such as daily, monthly, quarterly, and annual.

[0079] S103 , for each type of workflow, determine whether there is an abnormality in the type of workflow according to the workflow running information.

[0080] Specifically, it is possible to determine whether each category of workflow updates data according to the scheduling frequency based on the workflow running information, thereby knowing whether each category of workflow has an exception. Therefore, if a category of workflow has an exception, it means that the category of workflow has not updated data according to the scheduling frequency, so step S104 is executed at this time.

[0081] Optionally, if there is no anomaly in the category workflow, it means that the category workflow updates data according to the scheduling frequency. At this time, the anomaly detection of the category workflow is completed, and the next category workflow can be detected.

[0082] Optionally, in another embodiment of the present application, a specific implementation of step S103 is as follows: Figure 2 As shown, the following steps are included:

[0083] S201 . For each type of workflow, extract update information corresponding to the type of workflow from the workflow running information.

[0084] It is understandable that the job flow running information records the latest update information of each job flow. Therefore, the latest update information can be used to determine whether the data in the job flow is updated on time and whether there is an abnormality in the job flow. Therefore, the update information corresponding to each category of job flow can be extracted in advance from the job flow running information.

[0085] S202: Determine whether the data in the category workflow is updated on time based on the update information corresponding to the category workflow.

[0086] Specifically, according to the latest update record in the update information, it is determined whether the data in the category workflow is updated on time, if the data in the category workflow is updated on time, step S203 is executed, if the data in the category workflow is not updated on time, step S204 is executed.

[0087] For example, for daily updated work flow information, it is determined whether the data in the work flow information was updated yesterday. For monthly updated work flow information, it is determined whether the data in the work flow information was updated at the end of last month.

[0088] S203: Determine whether there is any abnormality in the category operation flow.

[0089] S204: Determine whether there is an abnormality in the category workflow.

[0090] When the data in the category workflow is not updated on time, it can be indicated that the category workflow has an abnormality, so step S104 can be executed at this time.

[0091] S104. Generate abnormal workflow details according to the category workflow.

[0092] Specifically, when there is an exception in a category workflow, abnormal data and data that are not updated on time, that is, abnormal workflows and workflows that are not updated on time, can be extracted from the category workflow, and abnormal workflow details can be generated based on the extracted workflows, so that the execution log of the workflow can be queried in a targeted manner based on the abnormal workflow details.

[0093] S105: Input the abnormal operation flow details into a pre-trained abnormal operation flow classification model to obtain the abnormal type corresponding to the abnormal operation flow details.

[0094] Among them, the abnormal operation flow classification model is pre-trained using historical abnormal operation flow details.

[0095] It is understandable that the abnormal workflow details may include the number of workflow dependencies, the maximum number of workflow table connections, the workflow operation cycle, the number of successful workflow restarts, and the number of historical dependency events that have not arrived. Therefore, by inputting these data into the abnormal workflow classification model, the abnormal type corresponding to the abnormal workflow details can be obtained.

[0096] In addition, the abnormal operation flow classification model is the LeNet-5 model, which uses a convolutional neural network (CNN) to train the historical abnormal operation flow details to obtain the abnormal types corresponding to the historical abnormal operation flow details.

[0097] Optionally, the present application embodiment provides a method for training an abnormal workflow classification model, such as Figure 3 As shown, the following steps are included:

[0098] S301. Obtain details of multiple historical abnormal operation flows.

[0099] Optionally, multiple historical abnormal operation flow details can be obtained from the database of the cloud platform for subsequent model training.

[0100] S302: Input each historical abnormal operation flow detail into the abnormal operation flow classification model to obtain the abnormal type corresponding to each historical abnormal operation flow detail.

[0101] Specifically, the historical abnormal operation flow details are classified through the neural network in the abnormal operation flow classification model.

[0102] S303: for each historical abnormal operation flow detail, determine whether the abnormality type corresponding to the historical abnormal operation flow detail is consistent with the actual abnormality type corresponding to the historical abnormal operation flow detail.

[0103] It should be noted that in order to know whether the abnormal type output by the abnormal workflow classification model is accurate, the abnormal type corresponding to the historical abnormal workflow details can be judged, that is, whether the abnormal type corresponding to the historical abnormal workflow details is consistent with the actual abnormal type corresponding to the historical abnormal workflow details. If the abnormal type corresponding to the historical abnormal workflow details is consistent with the actual abnormal type corresponding to the historical abnormal workflow details, it means that the abnormal type output by the abnormal workflow classification model is accurate, so step S304 is executed. If the abnormal type corresponding to the historical abnormal workflow details is not consistent with the actual abnormal type corresponding to the historical abnormal workflow details, it means that the abnormal type output by the abnormal workflow classification model is not accurate enough, so step S305 is executed.

[0104] S304: Determine that the abnormal operation flow classification model is a trained abnormal operation flow classification model.

[0105] S305: Adjust the parameters of the abnormal workflow classification model.

[0106] Specifically, when the exception type corresponding to the historical abnormal workflow details does not conform to the actual exception type corresponding to the historical abnormal workflow details, it is necessary to continuously adjust the parameters of the abnormal workflow classification model, and continue to train the abnormal workflow classification model based on the adjusted parameters. Therefore, when adjusting the parameters, it is necessary to return to execute step S302 until it is determined that the exception type corresponding to each historical abnormal workflow detail is consistent with the actual exception type corresponding to each historical abnormal workflow detail.

[0107] S106: Feedback abnormal operation flow details and corresponding abnormal types to the front end.

[0108] Specifically, the embodiment of the present application also constructs a visual intelligent monitoring system, which can build an abnormal workflow monitoring system through a cloud platform. The abnormal workflow monitoring system includes a workflow operation status module and an abnormal update workflow module. The workflow operation status module is responsible for monitoring the workflow operation status and updating the workflow details (you can filter by the workflow name, status, responsible person and other fields, and change the responsible person and view the change record), and visualizes the abnormal workflow details and their corresponding abnormal types, so that the front end can more clearly know the abnormal workflow.

[0109] In addition, the abnormal update workflow module is responsible for correcting the abnormal workflow detection results, manually correcting the erroneous data results or the workflow with known abnormal reasons, and adding the abnormal workflow whitelist.

[0110] Optionally, after obtaining the abnormal type corresponding to the abnormal operation flow details, the abnormal operation flow details may be modified. Therefore, in another embodiment of the present application, the abnormal operation flow details are also modified accordingly. Figure 4 As shown, another embodiment of the present application provides a method for correcting abnormal operation flow details, comprising the following steps:

[0111] S401. When the abnormal type corresponding to the abnormal operation flow details is a resource overlimit type, determine whether there is a dependent operation flow in the category operation flow corresponding to the abnormal operation flow details.

[0112] It should be noted that resource overrun will cause abnormal error reporting of the workflow, and the main reason is that there are too many workflows running at the same time. Therefore, the problem of resource overrun can be solved by first executing the dependent workflow corresponding to the category workflow. Therefore, it is necessary to determine whether there is a dependent workflow in the category workflow corresponding to the abnormal workflow details. If there is no dependent workflow in the category workflow corresponding to the abnormal workflow details, it means that the problem of resource overrun cannot be solved by first executing the dependent workflow corresponding to the category workflow, so step S402 is executed. If there is a dependent workflow in the category workflow corresponding to the abnormal workflow details, it means that the problem of resource overrun can be solved by first executing the dependent workflow corresponding to the category workflow, so step S403 is executed.

[0113] S402: Execute the restart strategy of the category job flow.

[0114] Specifically, when there is no dependent workflow in the category workflow corresponding to the abnormal workflow detail, it is necessary to execute the restart strategy of the category workflow to restart the workflow and resume its normal execution.

[0115] S403: Check whether the state of the dependent job flow is completed.

[0116] It should be noted that when there is a dependent workflow in the category workflow corresponding to the abnormal workflow details, it is also necessary to detect whether the status of the dependent workflow is completed to avoid the situation where the dependent workflow is repeatedly executed and wastes computing resources. Therefore, if the status of the dependent workflow is completed, it means that the dependent workflow cannot be executed, so step S404 is executed at this time. If the status of the dependent workflow is in an uncompleted state, it means that the dependent workflow can be executed, so step S405 is executed.

[0117] S404: Execute the restart strategy of the category job flow.

[0118] It is understandable that when the state of the dependent job flow is the completed state, the restart strategy of the category job flow needs to be executed to restart the job flow and resume its normal execution.

[0119] S405: recursively perform the category job flow and the dependent job flow.

[0120] Specifically, when the state of the dependent workflow is incomplete, the name of the abnormal workflow and its dependent workflows need to be input as key-value pairs into the recursive function until all data execution is completed, and then the recursive function is exited. This setting can solve the problem of abnormal errors in workflows A and B due to resource overload, avoiding the situation where downstream workflow B is repeatedly executed and wastes computing resources before upstream workflow A is successfully rerun. Without increasing computing resources, the normal operation of the upstream workflow of critical tasks is prioritized. When the restart fails, the relevant information is pushed to the responsible person in real time via SMS and email.

[0121] Optionally, in addition to Figure 4 In addition to the method for correcting abnormal operation flow details provided, there are other methods for correcting abnormal types. Therefore, another method for correcting abnormal operation flow details is also provided in another embodiment of the present application, such as Figure 5 As shown, the following steps are included:

[0122] S501. When the abnormal type corresponding to the abnormal operation flow details is a restart abnormal type, the category operation flow corresponding to the abnormal operation flow details is written into a whitelist.

[0123] It should be noted that when the exception type corresponding to the abnormal job flow details is a restart exception type, it is because the non-restartable upstream event did not arrive in time, causing the job flow to stagnate. This problem occurs because the upstream data or events are not updated. Therefore, in the case of unupdated upstream data, the job flow operation and maintenance personnel can manually add it to the whitelist to respond to the addition operation of the job flow operation and maintenance personnel. The category job flow corresponding to the abnormal job flow details will be written into the whitelist.

[0124] S502: Determine whether there is any record of unsuccessful execution of the category job flow within a preset time period.

[0125] It is understandable that after writing the category workflow corresponding to the abnormal workflow details into the whitelist, it is also necessary to set the maximum abnormal update days to know if the category workflow has no successful execution record after exceeding the maximum number of days, so as to trigger an early warning and avoid misjudging the normal workflow as abnormal. Therefore, it is necessary to determine whether there is a record of unsuccessful execution of the category workflow within the preset time period. If there is a record of unsuccessful execution of the category workflow within the preset time period, it means that the category workflow is an abnormal workflow, so step S503 is executed.

[0126] Optionally, if there is no record of unsuccessful execution of the category operation flow within a preset time period, it means that the category operation flow is a normal operation flow, and no warning is required.

[0127] S503: Trigger an alarm mechanism.

[0128] Specifically, when there are records of unsuccessful execution of a category operation flow within a preset time period, the relevant information is pushed to the responsible person in real time via SMS or email.

[0129] In addition, for some job flows that cannot be executed according to the established cycle due to the failure of upstream task events to arrive, the dependency relationship is automatically reissued and the job flow is started directly.

[0130] It should be noted that in addition to the restart exception type and resource limit type, there is also a syntax error type. Errors caused by program syntax errors cannot be solved by simply rerunning the program. In this case, the error log can be pushed to the corresponding responsible person via SMS or email, and the problem can be manually located and the program can be corrected.

[0131] The present application provides a method for detecting abnormal workflows on a cloud platform. The method obtains the workflow information and workflow operation information of the cloud platform, and then classifies the workflow information to obtain multiple categories of workflows. Then, for each category of workflow, the method determines whether the category of workflow is abnormal based on the workflow operation information. If the category of workflow is abnormal, the abnormal workflow details are generated based on the category of workflow. Then, the abnormal workflow details are input into a pre-trained abnormal workflow classification model to obtain the abnormal type corresponding to the abnormal workflow details. The abnormal workflow classification model is pre-trained using historical abnormal workflow details, and finally the abnormal workflow details and their corresponding abnormal types are fed back to the front end. Therefore, the workflow is no longer detected manually for abnormalities, but the workflow is monitored for abnormalities through abnormal workflow monitoring, which effectively improves the efficiency of abnormal detection. The workflow is classified according to business logic requirements, and differential warning rules of the model are formulated to effectively obtain the abnormal type of the workflow.

[0132] Another embodiment of the present application provides a cloud platform abnormal operation flow detection system, such as Figure 6 As shown, it includes the following units:

[0133] The information acquisition unit 601 is used to acquire the job flow information and job flow running information of the cloud platform.

[0134] The classification unit 602 is used to classify the job flow information to obtain multiple categories of job flows.

[0135] The abnormality determination unit 603 is used to determine whether there is an abnormality in each type of job flow according to the job flow running information.

[0136] The generating unit 604 is configured to generate abnormal operation flow details according to the category operation flow if there is an abnormality in the category operation flow.

[0137] The data input unit 605 is used to input the abnormal operation flow details into a pre-trained abnormal operation flow classification model to obtain the abnormal type corresponding to the abnormal operation flow details. The abnormal operation flow classification model is pre-trained using historical abnormal operation flow details.

[0138] The feedback unit 606 is used to feed back the abnormal operation flow details and the corresponding abnormal type to the front end.

[0139] It should be noted that the specific working process of the above-mentioned units in the embodiment of the present application can refer to steps S101 to S106 in the above-mentioned method embodiment, and will not be repeated here.

[0140] Optionally, in another embodiment of the present application, in a cloud platform abnormal operation flow detection system, the abnormality judgment unit 603 includes:

[0141] The extraction unit is used to extract the update information corresponding to each type of work flow from the work flow running information.

[0142] The first judging unit is used to judge whether the data in the category job flow is updated on time according to the updating information corresponding to the category job flow.

[0143] The first determining unit is used to determine that there is no abnormality in the category workflow if the data in the category workflow is updated on time.

[0144] The second determining unit is configured to determine that an abnormality exists in the category workflow if the data in the category workflow is not updated on time.

[0145] Optionally, another embodiment of the present application provides an abnormal operation flow detection system for a cloud platform, further comprising:

[0146] The acquisition unit is used to acquire details of multiple historical abnormal operation flows.

[0147] The input unit is used to input each historical abnormal operation flow detail into the abnormal operation flow classification model to obtain the abnormal type corresponding to each historical abnormal operation flow detail.

[0148] The second judgment unit is used to judge, for each historical abnormal operation flow detail, whether the abnormality type corresponding to the historical abnormal operation flow detail is consistent with the actual abnormality type corresponding to the historical abnormal operation flow detail.

[0149] The third determining unit is configured to determine that the abnormal operation flow classification model is a trained abnormal operation flow classification model if the abnormal type corresponding to the historical abnormal operation flow details matches the actual abnormal type corresponding to the historical abnormal operation flow details.

[0150] The adjustment unit is used to adjust the parameters of the abnormal workflow classification model if the abnormal type corresponding to the historical abnormal workflow details does not conform to the actual abnormal type corresponding to the historical abnormal workflow details, and return to execute to input each historical abnormal workflow detail into the abnormal workflow classification model respectively to obtain the abnormal type corresponding to each historical abnormal workflow detail.

[0151] Optionally, another embodiment of the present application provides an abnormal operation flow detection system for a cloud platform, further comprising:

[0152] The third judgment unit is configured to judge whether there is a dependent workflow in the category workflow corresponding to the abnormal workflow details when the abnormal type corresponding to the abnormal workflow details is a resource overlimit type.

[0153] The first execution unit is configured to execute a restart strategy of the category workflow if there is no dependent workflow in the category workflow corresponding to the abnormal workflow detail.

[0154] The detection unit is used to detect whether the state of the dependent workflow is a completed state if there is a dependent workflow in the category workflow corresponding to the abnormal workflow detail.

[0155] The second execution unit is configured to execute the restart strategy of the category workflow if the state of the dependent workflow is the completion state.

[0156] The recursive unit is used for recursively performing the classification job flow and the dependent job flow if the state of the dependent job flow is an unfinished state.

[0157] Optionally, another embodiment of the present application provides an abnormal operation flow detection system for a cloud platform, further comprising:

[0158] The writing unit is used to write the category operation flow corresponding to the abnormal operation flow details into the white list when the abnormal type corresponding to the abnormal operation flow details is a restart abnormal type.

[0159] The fourth determination unit is used to determine whether there is any record of unsuccessful execution of the category operation flow within a preset time period.

[0160] The trigger unit is used to trigger an alarm mechanism if there is a record of unsuccessful execution of the category operation flow within a preset time period.

[0161] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the corresponding steps in the above method embodiments, and will not be repeated here.

[0162] It should also be noted that the abnormal operation flow detection of a cloud platform provided in the embodiment of the present application has the technical effects of any of the above embodiments, and the embodiment of the present application will not be repeated here.

[0163] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting abnormal workflows on a cloud platform, characterized in that: include: Obtain the job flow information and job flow operation information of the cloud platform; Classifying the job flow information to obtain multiple categories of job flows; For each of the category operation flows, judging whether the category operation flow is abnormal according to the operation flow running information; If the category operation flow is abnormal, generating abnormal operation flow details according to the category operation flow; Inputting the abnormal operation flow details into a pre-trained abnormal operation flow classification model to obtain the abnormal type corresponding to the abnormal operation flow details; wherein the abnormal operation flow classification model is pre-trained using historical abnormal operation flow details; The abnormal operation flow details and the corresponding abnormal type are fed back to the front end.

2. The method according to claim 1, characterized in that: The step of determining, for each of the category of job flows, whether the category of job flows is abnormal according to the job flow running information, includes: For each of the category work flows, extracting update information corresponding to the category work flow from the work flow running information; According to the update information corresponding to the category job flow, determining whether the data in the category job flow is updated on time; If the data in the category operation flow is updated on time, it is determined that there is no abnormality in the category operation flow; If the data in the category workflow is not updated on time, it is determined that there is an abnormality in the category workflow.

3. The method according to claim 1, characterized in that The training method of the abnormal operation flow classification model includes: Get details of multiple historical abnormal job flows; Input each of the historical abnormal operation flow details into the abnormal operation flow classification model to obtain the abnormal type corresponding to each of the historical abnormal operation flow details; For each of the historical abnormal operation flow details, determining whether the abnormality type corresponding to the historical abnormal operation flow detail is consistent with the actual abnormality type corresponding to the historical abnormal operation flow detail; If the abnormal type corresponding to the historical abnormal operation flow details is consistent with the actual abnormal type corresponding to the historical abnormal operation flow details, determining that the abnormal operation flow classification model is a trained abnormal operation flow classification model; If the exception type corresponding to the historical abnormal workflow details does not conform to the actual exception type corresponding to the historical abnormal workflow details, the parameters of the abnormal workflow classification model are adjusted, and the step of inputting each of the historical abnormal workflow details into the abnormal workflow classification model is returned to execute respectively, so as to obtain the exception type corresponding to each of the historical abnormal workflow details.

4. The method according to claim 1, characterized in that Also includes: When the abnormal type corresponding to the abnormal operation flow details is a resource overlimit type, determining whether there is a dependent operation flow in the category operation flow corresponding to the abnormal operation flow details; If there is no dependent workflow in the category workflow corresponding to the abnormal workflow detail, executing the restart strategy of the category workflow; If there is a dependent workflow in the category workflow corresponding to the abnormal workflow details, detecting whether the state of the dependent workflow is a completed state; If the state of the dependent job flow is the completed state, executing the restart strategy of the category job flow; If the state of the dependent job flow is an unfinished state, recursively perform the process on the category job flow and the dependent job flow.

5. The method according to claim 1, characterized in that Also includes: When the abnormal type corresponding to the abnormal operation flow details is a restart abnormal type, writing the category operation flow corresponding to the abnormal operation flow details into the whitelist; Determine whether there is a record of unsuccessful execution of the category of job flow within a preset time period; If there is a record of unsuccessful execution of the category of job flows within a preset time period, an alarm mechanism is triggered.

6. An abnormal operation flow detection system for a cloud platform, characterized in that: include: An information acquisition unit, used to acquire the job flow information and job flow running information of the cloud platform; A classification unit, used for classifying the job flow information to obtain a plurality of categories of job flows; An abnormality judgment unit is used to judge whether there is an abnormality in each of the category operation flows according to the operation flow running information; A generating unit, configured to generate abnormal operation flow details according to the category operation flow if there is an abnormality in the category operation flow; A data input unit, used for inputting the abnormal operation flow details into a pre-trained abnormal operation flow classification model to obtain the abnormal type corresponding to the abnormal operation flow details; wherein the abnormal operation flow classification model is pre-trained using historical abnormal operation flow details; The feedback unit is used to feed back the abnormal operation flow details and the corresponding abnormal type to the front end.

7. The system according to claim 6, characterized in that The abnormality judgment unit comprises: An extraction unit, configured to extract, for each of the category workflows, update information corresponding to the category workflow from the workflow running information; A first judging unit, configured to judge whether the data in the category job flow is updated on time according to the update information corresponding to the category job flow; A first determining unit, configured to determine that there is no abnormality in the category operation flow if the data in the category operation flow is updated on time; The second determining unit is configured to determine that an abnormality exists in the category workflow if the data in the category workflow is not updated on time.

8. The system according to claim 6, characterized in that Also includes: An acquisition unit, used to acquire details of multiple historical abnormal operation flows; An input unit, used to input each of the historical abnormal operation flow details into the abnormal operation flow classification model to obtain the abnormal type corresponding to each of the historical abnormal operation flow details; A second judgment unit is used to judge, for each of the historical abnormal operation flow details, whether the abnormality type corresponding to the historical abnormal operation flow detail is consistent with the actual abnormality type corresponding to the historical abnormal operation flow detail; a third determining unit, configured to determine that the abnormal operation flow classification model is a trained abnormal operation flow classification model if the abnormal type corresponding to the historical abnormal operation flow details matches the actual abnormal type corresponding to the historical abnormal operation flow details; An adjustment unit is used to adjust the parameters of the abnormal workflow classification model if the abnormal type corresponding to the historical abnormal workflow details does not conform to the actual abnormal type corresponding to the historical abnormal workflow details, and return to execute the step of inputting each of the historical abnormal workflow details into the abnormal workflow classification model to obtain the abnormal type corresponding to each of the historical abnormal workflow details.

9. The system according to claim 6, characterized in that Also includes: A third judgment unit is used to judge whether there is a dependent workflow in the category workflow corresponding to the abnormal workflow detail when the abnormal type corresponding to the abnormal workflow detail is a resource overlimit type; A first execution unit, configured to execute a restart strategy of the category workflow if there is no dependent workflow in the category workflow corresponding to the abnormal workflow detail; a detection unit, configured to detect whether a state of a dependent workflow is a completed state if there is a dependent workflow in the category workflow corresponding to the abnormal workflow detail; A second execution unit, configured to execute a restart strategy of the category workflow if the state of the dependent workflow is a completed state; The recursive unit is used to recursively perform the process on the category workflow and the dependent workflow if the state of the dependent workflow is an unfinished state.

10. The system according to claim 6, characterized in that Also includes: A writing unit, configured to write the category of the operation flow corresponding to the abnormal operation flow details into the whitelist when the abnormal type corresponding to the abnormal operation flow details is a restart abnormal type; A fourth determination unit, configured to determine whether there is a record of unsuccessful execution of the category operation flow within a preset time period; The trigger unit is used to trigger an alarm mechanism if there is a record of unsuccessful execution of the category operation flow within a preset time period.

Images