Fusing detection method and device for abnormal operation of workflow and computer equipment

Abstract

The invention relates to a detection fusing method and device for workflow operation abnormity and computer equipment, and relates to the technical field of big data collection and analysis. The method comprises the following steps: acquiring exception announcement information corresponding to an exceptional workflow in a workflow processing system, and state information of a processing group; carrying out persistence on the abnormal announcement information, carrying out diagnostic analysis on the persisted abnormal announcement information, and screening out target announcement information from the abnormal announcement information based on a diagnostic analysis result; and under the condition that the target announcement information accords with a preset fusing rule, determining a fusing object associated with the abnormal workflow, and generating fusing indication information for the fusing object. By adopting the method, the workflow processing efficiency and stability can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of big data collection and analysis, and in particular to a method, device and computer equipment for detecting and fusing abnormal workflow operations. Background Art

[0002] The existing workflow operation anomaly detection and fusing solution for NiFi (a high-performance data integration workflow processing system for processing, routing and converting various data streams) only relies on the Bulletin (abnormal notification) information generated by NiFi and directly displays it to the user. Bulletin is a log that NiFi uses to record abnormalities during the runtime of various types of components, including but not limited to workflow processors. The content of the log is relatively simple, and the information is usually: component id + JAVA abnormal information. The Bulletin information exists in the memory of NiFi's JVM (Java virtual machine) and is valid for 5 minutes. It will be cleared after the expiration date.

[0003] The above solution has the following problems: (1) Bulletin information is too simple and technical, which is often not friendly to non-technical personnel. Often these prompts are directly displayed to users, which often seem difficult for users to understand; (2) Bulletin information exists in JVM memory and has a short validity period. In the event of application downtime, the Bulletin will be lost; (3) For those processors (also known as processors, a component used to execute workflows) with retries, or some components that are executed at a high frequency, a lot of bulletins will be generated in a short period of time, which will occupy more JVM memory and directly affect NiFi processing performance.

[0004] In general, the current anomaly detection and processing methods for workflows are not perfect, resulting in low workflow processing efficiency and poor stability. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for detecting and blowing workflow operation anomalies, which can improve workflow processing efficiency and stability, in response to the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting and fusing abnormal workflow operation. The method comprises:

[0007] Acquire the abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow;

[0008] The abnormal announcement information is persisted, and the abnormal announcement information after persistence is diagnosed and analyzed, and target announcement information is obtained from the abnormal announcement information based on the diagnosis and analysis result; the target announcement information is the abnormal announcement information whose occurrence frequency meets the preset frequency;

[0009] In the case where the target announcement information complies with a preset fusing rule, a fusing object associated with the abnormal workflow is determined, and fusing indication information for the fusing object is generated.

[0010] In one embodiment, after the step of filtering the target announcement information from the abnormal announcement information based on the diagnosis analysis result, the method further includes:

[0011] Identify the abnormal type corresponding to the target announcement information;

[0012] According to the exception type corresponding to the target announcement information, the exception announcement content in the target announcement information is converted to obtain converted announcement information; the converted announcement information includes exception description information and solution suggestion information matching the exception description information.

[0013] In one embodiment, the step of obtaining abnormal notification information corresponding to an abnormal workflow in the workflow processing system and status information of a processing group includes:

[0014] Configure and execute a first component to obtain abnormal announcement information corresponding to an abnormal workflow in the workflow processing system; the first component is a component for regularly pushing incremental abnormal announcement information;

[0015] A second component is configured and executed to obtain status information of the processing group; the second component is used to periodically push the status information of the processing group.

[0016] In one embodiment, the persisting the abnormal announcement information includes:

[0017] Obtaining persistence condition information; the persistence condition information includes a preset number threshold corresponding to the minimum number of persistence entries, and a preset time period corresponding to the interval time;

[0018] The current number of items to be persisted is obtained according to the preset time period. When the current number of items to be persisted is greater than the preset number threshold, the exception announcement information to be persisted is stored in the persistence database; and when the current number of items to be persisted is less than or equal to the preset number threshold, the exception announcement information to be persisted is stored in a preset memory queue, so that the exception announcement information to be persisted is stored in the persistence database according to the memory queue.

[0019] In one embodiment, the method further comprises:

[0020] Generate alarm notification information according to the fuse indication information for the fuse object and the converted announcement information;

[0021] Send the alarm notification information to the user terminal; wherein the alarm notification information includes the cluster address of the workflow processing system, the workflow name, the workflow access path, the exception cause recorded in the converted announcement information, the failure start time, the cumulative number of failed retries, and the solution.

[0022] In one embodiment, the fuse indication information includes a fuse task, and the method further includes:

[0023] When the HTTP service corresponding to the workflow processing system crashes and restarts, and the generated fuse task has not yet been executed, the fuse execution plan record recorded in the database queried after the restart;

[0024] The fusing task is regenerated and executed according to the fusing execution plan record obtained by querying.

[0025] In a second aspect, the present application also provides a device for detecting and fusing abnormal workflow operation. The device comprises:

[0026] The abnormal announcement acquisition module is used to obtain the abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow;

[0027] An abnormality analysis module, used to persist the abnormality announcement information, perform diagnostic analysis on the abnormality announcement information after persistence, and filter out target announcement information from the abnormality announcement information based on the diagnostic analysis result; the target announcement information is the abnormality announcement information whose occurrence frequency meets the preset frequency;

[0028] A fusing module is used to determine the fusing object associated with the abnormal workflow and generate fusing indication information for the fusing object when the target announcement information meets the preset fusing rules.

[0029] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0030] Acquire the abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow;

[0031] The abnormal announcement information is persisted, and the abnormal announcement information after persistence is diagnosed and analyzed, and target announcement information is obtained from the abnormal announcement information based on the diagnosis and analysis result; the target announcement information is the abnormal announcement information whose occurrence frequency meets the preset frequency;

[0032] In the case where the target announcement information complies with a preset fusing rule, a fusing object associated with the abnormal workflow is determined, and fusing indication information for the fusing object is generated.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0034] Acquire the abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow;

[0035] The abnormal announcement information is persisted, and the abnormal announcement information after persistence is diagnosed and analyzed, and target announcement information is obtained from the abnormal announcement information based on the diagnosis and analysis result; the target announcement information is the abnormal announcement information whose occurrence frequency meets the preset frequency;

[0036] In the case where the target announcement information complies with a preset fusing rule, a fusing object associated with the abnormal workflow is determined, and fusing indication information for the fusing object is generated.

[0037] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0038] Acquire the abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow;

[0039] The abnormal announcement information is persisted, and the abnormal announcement information after persistence is diagnosed and analyzed, and target announcement information is obtained from the abnormal announcement information based on the diagnosis and analysis result; the target announcement information is the abnormal announcement information whose occurrence frequency meets the preset frequency;

[0040] In the case where the target announcement information complies with a preset fusing rule, a fusing object associated with the abnormal workflow is determined, and fusing indication information for the fusing object is generated.

[0041] The above-mentioned workflow operation abnormality detection and fuse method, device, computer equipment, storage medium and computer program product first receive the abnormal announcement information generated in the workflow processing system through the HTTP end and store it persistently, then perform real-time diagnosis and analysis on the persisted abnormal announcement information, and then filter out the target announcement information whose frequency meets the preset standard; then, when the target announcement information meets the fuse rule, determine the fuse object associated with the abnormal workflow, and generate fuse indication information for the fuse object; finally, according to the configured fuse rule, create an automatic fuse countdown task to trigger the fuse operation mechanism so that the user can decide the final execution operation of the fuse in real time; this embodiment greatly improves the stability and fault tolerance of the workflow processing system through efficient abnormality monitoring and automatic fuse mechanism, reduces the risks and losses caused by system abnormalities, improves the abnormality detection and processing methods of the workflow, and improves the processing efficiency and stability of the workflow. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of a method for detecting and fusing abnormal workflow operation in an embodiment;

[0044] Figure 2 A schematic diagram of a flow chart of a method for detecting and fusing abnormal workflow operation in another embodiment;

[0045] Figure 3 is a software architecture diagram in one embodiment;

[0046] Figure 4 This is a flowchart of scheduled push and cleanup in one embodiment;

[0047] Figure 5 A flowchart of detecting a blown fuse alarm and a blown fuse trigger in one embodiment;

[0048] Figure 6 Creating a flow chart for an automatic fuse countdown task of a workflow in one embodiment;

[0049] Figure 7 A recovery flow chart of an automatic fuse countdown task in a workflow in one embodiment;

[0050] Figure 8 A flowchart of a fuse processing of a workflow in one embodiment;

[0051] Fig. 9 A flowchart for configuring a workflow exception fuse in one embodiment;

[0052] Fig.10 It is a structural block diagram of a device for detecting and fusing abnormalities in workflow operation in one embodiment;

[0053] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0056] In one embodiment, Figure 1 As shown, a method for detecting and fusing abnormal workflow operation is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0057] S101, obtaining abnormal announcement information corresponding to an abnormal workflow in a workflow processing system and status information of a processing group.

[0058] Among them, the workflow processing system can be NiFi, the exception announcement information can be Bulletin information; the processing group is a grouping of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow. Further, the status information of the processing group is used to determine the fuse object associated with the abnormal workflow.

[0059] S102, persist the abnormal announcement information, and perform diagnostic analysis on the persisted abnormal announcement information, and filter out target announcement information from the abnormal announcement information based on the diagnostic analysis result.

[0060] Among them, the target announcement information is abnormal announcement information whose occurrence frequency meets the preset frequency.

[0061] Exemplarily, the http end receives the Bulletin information, persists it, and then performs real-time diagnosis and analysis, and issues an alarm for the Bulletin information that needs to be analyzed.

[0062] S103, when the target announcement information complies with the preset fusing rule, determining the fusing object associated with the abnormal workflow, and generating fusing indication information for the fusing object.

[0063] Exemplarily, the workflow supports configuration of circuit breaking rules, supports circuit breaking of selected processors and circuit breaking of the entire workflow, and uses an abnormality detection circuit breaking algorithm to determine whether to perform circuit breaking according to the on-off rules of the workflow configuration if necessary.

[0064] More specifically, it is detected that the phenomenon meets the circuit breaking rules, the abnormal circuit breaking rules configured for the workflow are obtained, an automatic circuit breaking countdown task for the workflow is created, the circuit breaking operation mechanism is triggered, and after the alarm is sent, the user can decide the final execution operation of the circuit breaking in real time.

[0065] In the above-mentioned workflow operation abnormality detection and fuse method, first, the abnormal announcement information generated in the workflow processing system is received through the HTTP end and stored persistently, and then the persisted abnormal announcement information is diagnosed and analyzed in real time, and then the target announcement information whose frequency meets the preset standard is screened out; then, when the target announcement information meets the fuse rule, the fuse object associated with the abnormal workflow is determined, and the fuse indication information for the fuse object is generated; finally, according to the configured fuse rule, an automatic fuse countdown task is created to trigger the fuse operation mechanism so that the user can decide the final execution operation of the fuse in real time; this embodiment greatly improves the stability and fault tolerance of the workflow processing system through efficient exception monitoring and automatic fuse mechanism, reduces the risks and losses caused by system exceptions, improves the workflow exception detection and processing methods, and improves the workflow processing efficiency and stability.

[0066] In one embodiment, after the step of filtering target announcement information from abnormal announcement information based on the diagnostic analysis results, the method also includes: identifying the abnormal type corresponding to the target announcement information; according to the abnormal type corresponding to the target announcement information, converting the abnormal announcement content in the target announcement information to obtain the converted announcement information.

[0067] The converted announcement information includes exception description information and solution suggestion information matching the exception description information.

[0068] Among them, the exception type can be a processor type or a database type. The purpose of converting the exception announcement content in the target announcement information is to make the exception prompt information more friendly and easier for users or administrators to understand its meaning.

[0069] For example, according to the bulletin of each workflow detection thread, it is distinguished by type. For the bulletin of the processor type, it can be known which workflow and which processor it is generated by, so that the controller service type used by the processor can be queried. For example, for the database connection pool, the database type can be obtained through its configured URL. Through different database types, different exceptions are classified and the exception prompt information is friendly converted, and corresponding exception solutions are given. For example, for the TableNotFoundException of the MySQL database, the configured library table and database address are obtained in detail. The converted information is similar to: User <xxx>The MySQL table <database name.table name> at the access address 1.2.3.4:3306 does not exist. Please confirm whether the table exists. If not, please create the table in the correct format.

[0070] In this embodiment, after first filtering out the target announcement information from the abnormal announcement information based on the diagnostic analysis results, the abnormal type corresponding to the target announcement information is identified, and then the abnormal announcement content in the target announcement information is converted according to the abnormal type to obtain the converted announcement information; thereby making the abnormal prompt information more friendly and convenient for users or administrators to understand its meaning; finally, the different types of Bulletins are classified and processed, and corresponding solution prompts are provided according to the database type. This embodiment optimizes the expression of abnormal information so that the system can provide more intuitive guidance and suggestions in abnormal situations, improves the user's operating efficiency, and reduces the time cost of system troubleshooting.

[0071] In one embodiment, obtaining exception notification information corresponding to an abnormal workflow in a workflow processing system and status information of a processing group includes: configuring and executing a first component to obtain exception notification information corresponding to an abnormal workflow in the workflow processing system; configuring and executing a second component to obtain status information of the processing group.

[0072] Among them, the first component is a component used for regularly pushing incremental abnormal announcement information, such as the SiteToHttpBulletinReportingTask component; the second component is a SiteToHttpBulletinReportingTask component used for regularly pushing status information of the processing group.

[0073] For example, a custom SiteToHttpBulletinReportingTask component is used to regularly push incrementally generated Bulletins to the http server, and after a successful push, the pushed Bulletins are removed from the nifi memory to release memory resources in time; the SiteToHttpBulletinReportingTask component is configured and run, and the configuration is as follows: the component is scheduled every 3 seconds, and the number of Bulletin batches pushed each time is 2000. After a successful push, the Bulletins in the memory are cleaned up.

[0074] In this embodiment, the first component is first configured and executed to regularly push incremental exception announcement information, thereby obtaining the exception announcement information corresponding to the workflow in which the exception occurs in the workflow processing system; then the second component is configured and executed to regularly push the status information of the processing group, thereby obtaining the status information of the processing group, which facilitates cleaning up the Bulletin in the memory and ensuring real-time push of information, thereby improving the work efficiency and resource utilization of the system; this embodiment achieves the real-time and accuracy of system monitoring through an effective exception announcement information and status information acquisition mechanism.

[0075] In one embodiment, the exception announcement information is persisted, including: obtaining persistence condition information; obtaining the current number of items to be persisted according to a preset time period, and when the current number of items to be persisted is greater than a preset number threshold, storing the exception announcement information to be persisted in a persistent database; and, when the current number of items to be persisted is less than or equal to the preset number threshold, storing the exception announcement information to be persisted in a preset memory queue, so that the exception announcement information to be persisted is stored in the persistent database according to the memory queue.

[0076] The persistence condition information includes a preset number threshold corresponding to the minimum number of persistent items, such as the minimum number of persistent items num (2000 items), and a preset time period t (such as 10 seconds) corresponding to the interval time.

[0077] Exemplarily, the data batching algorithm is used to improve the performance of the storage database. Because the task push is relatively frequent, a timed thread pool is first designed with a core number of 1. The persistence conditions are set: the minimum number of persistent items num (for example, 2000 items) and the interval time t (for example, 10s); after HTTP receives the bulletin, if the number is greater than num, it is stored (processed according to the configured persistence engine); if the number is less than num, it is first added to the memory queue queue. The timed thread pool is executed once every interval to pull data from the queue for persistence, and the hook function of the analysis and detection application shutdown is configured to persist the bulletin in the queue.

[0078] In this embodiment, firstly, the conditional information of persistence is obtained, and then the current number of items to be persisted is obtained according to a preset time period; when the current number of items to be persisted is greater than a preset number threshold, the abnormal announcement information to be persisted is stored in a persistent database, otherwise the abnormal announcement information to be persisted is stored in a preset memory queue so that it can be stored in a persistent database according to the memory queue; then, the data batching algorithm is used to improve the performance of the storage database, and a timed thread pool is designed to regularly pull data from the memory queue for persistence; this embodiment achieves the technical effect of improving system performance and data security by optimizing the persistence strategy and memory management.

[0079] In one embodiment, the method further includes: generating alarm notification information according to the fuse indication information for the fuse object and the converted announcement information; and sending the alarm notification information to the user terminal.

[0080] Among them, the alarm notification information includes the cluster address of the workflow processing system, the workflow name, the workflow access path, the exception cause recorded in the converted announcement information, the failure start time, the cumulative number of failed retries, and the solution.

[0081] Exemplarily, the converted announcement information and fuse indication information are promptly notified to the user, and the alarm channels include but are not limited to: email, SMS, enterprise WeChat robot, etc. The key alarm information includes but is not limited to: NiFi cluster address, workflow name, workflow access path, exception reason (exception information after conversion), failure start time, cumulative number of failed retries, solution, etc. This patent takes email as an example to show the detailed content of an alarm prompt and some prompts for user operations.

[0082] In this embodiment, alarm notification information is first generated based on the fuse indication information for the fuse object and the converted announcement information; then the alarm notification information is sent to the user terminal to ensure that the user is aware of the system abnormality in a timely manner; and then multiple alarm channels are used to ensure that the information can be quickly conveyed to relevant users; this embodiment achieves the technical effect of improving system monitoring efficiency and user response speed by timely generating and sending alarm notifications.

[0083] In one embodiment, the circuit breaker indication information includes a circuit breaker task, and the method further includes: when the HTTP service corresponding to the workflow processing system crashes and restarts, and the generated circuit breaker task has not yet been executed, recording the circuit breaker execution plan record in the database queried after the restart; regenerating the circuit breaker task and executing it according to the circuit breaker execution plan record obtained by the query.

[0084] For example, when the http service crashes and restarts, the circuit breaker task is automatically restored. Specifically, during the detection and analysis, a circuit breaker execution plan is generated and persisted to the database, and a countdown circuit breaker task is generated in the memory. Assuming that the plan has not been executed and the application is restarted or crashes, after the application restarts, the database will be queried for the circuit breaker execution plan record (status is not executed), and then the corresponding countdown circuit breaker task will be regenerated according to the record for execution.

[0085] In this embodiment, a circuit breaker task is first generated during the restart of the HTTP service of the workflow processing system after a downtime, and the circuit breaker execution plan is persisted to the database; then, after the application is restarted, the circuit breaker execution plan recorded in the database is queried to identify the circuit breaker tasks with an unexecuted status; then, the corresponding circuit breaker task is regenerated and executed based on the queried circuit breaker execution plan record to ensure the continuity and effectiveness of the circuit breaker mechanism; finally, through this automatic recovery mechanism, it is ensured that even in the event of a service downtime, the circuit breaker task can still be executed after the system is restored, thereby reducing potential fault propagation and business impact; this embodiment achieves the technical effect of improving system reliability and stability by optimizing the recovery strategy of the circuit breaker mechanism.

[0086] In another embodiment, Figure 2 As shown, a method for detecting and fusing abnormal workflow operation is provided, comprising the following steps:

[0087] S201, configuring and executing a first component to obtain abnormal announcement information corresponding to an abnormal workflow in a workflow processing system;

[0088] S202, configuring and executing a second component to obtain status information of a processing group;

[0089] S203, obtaining persistent condition information;

[0090] S204, obtaining the current number to be persisted according to a preset time period, and persisting the abnormal announcement information;

[0091] S205, performing diagnosis and analysis on the persisted abnormal announcement information, and filtering target announcement information from the abnormal announcement information based on the diagnosis and analysis result;

[0092] S206, identifying the abnormal type corresponding to the target announcement information;

[0093] S207, transforming the abnormal announcement content in the target announcement information according to the abnormal type corresponding to the target announcement information to obtain transformed announcement information;

[0094] S208, when the target announcement information complies with the preset fusing rule, determine the fusing object associated with the abnormal workflow, and generate fusing indication information for the fusing object.

[0095] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a method for detecting and breaking a workflow operation abnormality mentioned above, which will not be repeated here.

[0096] The existing workflow detection process has the following problems: (1) The NiFi application Bulletin information cannot be persisted, expires, and data is lost due to application downtime or restart; (2) Bulletin information is generated frequently and cannot be cleaned up in time, resulting in a large amount of memory resources being occupied; (3) Bulletin alarm information is too simple and unfriendly, making it difficult for users to understand and providing no solution to the error; (4) The high frequency of operation of the retry processor in the workflow leads to unnecessary waste of resources (application system resources, disk space, IO operations).

[0097] Based on this, the present application provides a method for detecting and breaking workflow abnormalities, also known as a method for detecting and breaking NiFi workflow abnormalities. Bulletin information is pushed through real-time http, and persisted to some external storage (such as MySQL, etc.), and the Bulletin information in the JVM memory is cleaned up in time, and the JVM memory of the NiFi application is released in time; through the Bulletin real-time analysis and diagnosis algorithm, the information belongs to different types of components to map the prompt information into a more friendly display, and provide a solution for display, and inform the user of the detailed information through real-time alarm push (such as email, SMS, enterprise micro robot, etc.); the Bulletin real-time analysis and diagnosis algorithm calculates whether there is a processor with high-frequency failure retries in the workflow, and supports configuration The way to set the fuse mechanism for the processor is to trigger the processor to stop in time and avoid high-frequency retries, because the flow files are ordered in the queue. If the first flow file out of the queue fails to be processed and keeps retrying, it will not only waste system resources, but also generate Bulletin information at a high frequency, and will also cause the application log to be refreshed continuously, wasting log printing thread resources, and too much disk IO, and a large number of application logs will also cause a waste of disk space, and will also affect the performance of NiFi; through the processor anomaly detection and fuse algorithm, the abnormal processor or the entire workflow is stopped in time to avoid the high-frequency failure retries of the invalid processor, which leads to unnecessary waste of system and application resources. What's more, due to the existence of multiple workflows with the same abnormal situation at the same time, the job avalanche occurs, which has an adverse effect on the system.

[0098] The main process steps of the workflow operation abnormality detection and fuse method provided in this application are:

[0099] (1) Configure SiteToHttpBulletinReportingTask, which is a ReportingTask component based on NiFi extension. It runs directly in the NiFi process and is used to push incremental Bulletin to the http end at a fixed time. After the push is successful, the Bulletin in the JVM memory can be cleaned up;

[0100] (2) Configure SiteToHttpStatusReportingTask, which is a ReportingTask component based on NiFi extension. It runs directly in the NiFi process and is used to push ProcessGroupStatus to the http end at a regular interval to analyze the real-time information of the workflow (including: processing group information, processor status information, connection queue status information);

[0101] (3) The http end receives the Bulletin information, persists it, and performs real-time diagnostic analysis. For Bulletin information that needs to be alerted after analysis, it is processed by an algorithm and then an alert notification is issued (the alert information will be classified and mapped, converted into more optimized prompts, and solutions will be provided);

[0102] (4) Supports configuration of circuit breaking rules for workflows, supports circuit breaking of selected processors and circuit breaking of the entire workflow, and uses anomaly detection circuit breaking algorithms to determine whether to perform circuit breaking based on the on / off rules configured in the workflow, if necessary.

[0103] The present application provides a method for detecting and fusing abnormal workflow operations, which pushes Bulletin information in real time for persistence, cleans up and releases memory resources in a timely manner, and performs real-time analysis and diagnosis; Bulletin information is processed by an algorithm and converted into more friendly prompt information, and solutions and alarm notifications are provided; specific processors or entire workflows that meet the fusing rules are fused to improve system stability, release system resources in a timely manner, and alleviate the preemption of system resources between different workflows; the abnormal fusing configuration and detection of each NiFi workflow are isolated from each other and will not affect each other, and when the service is restarted, it supports the recovery operation of abnormal fusing tasks, the fusing configuration is simple and easy to implement, and the fusing supports user confirmation, cancellation and silent operation.

[0104] In this embodiment, the basic software architecture diagram designed is shown in Figure 3 The method for detecting and fusing abnormal workflow operation provided by the present application comprises the following steps:

[0105] Step 1.1, customize the SiteToHttpBulletinReportingTask component. The purpose of this component is to regularly push incrementally generated Bulletins to the http server, and after the push is successful, remove the pushed bulletin from the nifi memory to release memory resources in time. For details of the processing flow, see Figure 4 ;

[0106] 1.2. Configure the SiteToHttpBulletinReportingTask component and run it. The configuration is: the component is scheduled every 3 seconds, and the number of Bulletin batches pushed each time is 2000. After the push is successful, the Bulletin in the memory is cleared;

[0107] 1.3. After receiving the bulletin, the http end persists it: Use the data batching algorithm to improve the performance of the storage database. Because the task is pushed relatively frequently, first design a timed thread pool with a core number of 1 and set the persistence conditions: the minimum number of persistent items num (for example, 2000) and the interval time t (for example, 10s); after the http receives the bulletin, if the number is greater than num, it will be stored (processed according to the configured persistence engine); if the number is less than num, it will be added to the memory queue queue first. The timed thread pool executes once every interval to pull data from the queue for persistence, and configure the shutdown hook function of the analysis and detection application to persist the bulletin in the queue to avoid data loss;

[0108] 1.4、Configure workflow exception fuse rules;

[0109] 1.5. Bulletin analysis and detection algorithm, grouping received Bulletins by workflow, and designing thread pool for multi-threaded detection;

[0110] 1.5.1. Custom analysis and detection thread class BulletinAnalyzeDetectionTask implements the Runnable interface, which is used to analyze the grouped List <bulletin>Submit to the custom thread pool for execution, customize the Bulletin analysis and detection thread pool BulletinAnalyzeDetectionExecutor, design the core parameters of the thread pool, among which, customize the empty queue NoneOperationQueue of BlockingQueue type that does not actually add elements. Because NoneOperationQueue does not actually store elements, it can achieve the maximum number of core threads in the thread pool. It can immediately trigger the thread rejection strategy. Customize a thread rejection strategy QueueBulletinRejectedHandler to handle the processing when the thread refuses to submit. It will put the thread into the memory LinkedHashMap according to the processing group id + component id.<String,BulletinAnalyzeDetectionTask> In the cache, it is guaranteed that there is a mapping relationship between the processing group id and the processor id and the analysis and detection thread. The purpose is to enable the analysis and detection thread in execution to obtain the Bulletin in the Bulletin array of the corresponding key from the cache if the number of Bulletins does not reach the rule configuration within a fixed time window. Assuming that the processor B of workflow A is configured with 5 exceptions within 1 minute, then the Bulletin within 1 minute can be obtained from the cache and transferred to the analysis and detection thread being executed. During the execution, the custom thread pool BulletinAnalyzeDetectionExecutor can be used to process the process;

[0111] 1.5.2. Configure fuse information for NiFi workflow;

[0112] 1.5.3. According to the Bulletin of each workflow detection thread, obtain the workflow and processor to which it corresponds, as well as the processing error of which stream file, and whether the processor is configured with failure retry and triggers the retry of the stream file. The main judgment basis is: for the same workflow, the same processor, and the same stream file uuid, if the number of Bulletins that increase over time is continuously received, it can be determined that the processor has been retrying a stream file and has not been successful; see details. Figure 5 , Figure 5 A flowchart of detecting a blown fuse alarm and a blown fuse trigger in one embodiment.

[0113] 1.5.4. According to step 1.5.3, detect that the phenomenon meets the circuit breaker rules, obtain the abnormal circuit breaker rules configured for the workflow, and create an automatic circuit breaker countdown task for the workflow. For details, see Figure 6 , Figure 6 Create a flowchart for the automatic fuse countdown task of a workflow in an embodiment; trigger the fuse operation mechanism, send an alarm, and the user can decide the final execution operation of the fuse in real time; see Figure 7 and Figure 8 , Figure 7 A recovery flow chart of an automatic fuse countdown task in a workflow in one embodiment; Figure 8 The figure is a flow chart of the fuse processing of the workflow in one embodiment.

[0114] In addition, the process of configuring fuse information for NiFi workflow is detailed in Fig. 9 , Fig. 9 A flowchart for configuring workflow exception fuse in one embodiment.

[0115] 1.5.5. According to the bulletin of each workflow detection thread, it is distinguished by type. For the bulletin of the processor type, it can be known which workflow and which processor it is generated by, so that the controller service type used by the processor can be queried. For example, for the database connection pool, the database type can be obtained through its configured URL. According to different database types, different exceptions are classified and abnormal prompt information is converted in a friendly manner, and corresponding exception solutions are given. For example, for the TableNotFoundException of the MySQL database, the configured library table and database address are obtained in detail. The converted information is similar to: User <xxx>The MySQL table <database name.table name> at the access address 1.2.3.4:3306 does not exist. Please confirm whether the table exists. If not, please create the table in the correct format.

[0116] 1.5.6. For the error information converted in step 1.5.5 and the fuse operation, timely alarms are issued to notify the user. The alarm channels include but are not limited to: email, SMS, enterprise micro-bots, etc. The key alarm information includes but is not limited to: NiFi cluster address, workflow name, workflow access path, abnormal reason (abnormal information after conversion), failure start time, cumulative number of failed retries, solutions, etc. This patent takes email as an example to show a detailed alarm prompt and some prompts for user operations;

[0117] 1.6. Automatic recovery of the circuit breaker task when the http service crashes and restarts: During the detection and analysis, a circuit breaker execution plan is generated and persisted to the database, and a countdown circuit breaker task is generated in the memory. Assuming that the plan has not been executed and the application is restarted or crashes, after the application is restarted, the database will be queried for the circuit breaker execution plan record (status is not executed), and then the corresponding countdown circuit breaker task will be regenerated according to the record for execution;

[0118] The beneficial effects of steps 1.1, 1.2, and 1.3 are: Bulletin is pushed to the http server in real time and persisted, Bulletin of NiFi application is cleaned up in time, JVM memory is released, and batching is performed in memory to avoid frequent write operations of small amounts of data under high concurrency, thereby improving database storage performance.

[0119] The beneficial effects of step 1.4 are: multi-threaded detection and analysis, improved efficiency, the fuse detection algorithm automatically triggers the fuse, Bulletin information processing and conversion, and provision of exception solutions are more user-friendly, the alarm content is detailed and comprehensive, and the user is notified in real time. The automatic fuse operation is generated by the system and displayed to the user, and manual confirmation is provided. It also supports system default operations for flexibility.

[0120] The technical key points of the workflow operation abnormality detection and fusing method provided in this application include:

[0121] (1) Bulletin detection and diagnosis algorithm based on multi-threading;

[0122] The advantages are: real-time analysis, simple dependencies, and easy algorithm implementation. The technical means it adopts is to use a custom thread pool to group the real-time received Bulletin by workflow, and then perform multi-threaded analysis. It combines the workflow and processor configuration information with the characteristics of the Bulletin to analyze and diagnose whether the workflow and processor operation have any unrecoverable exceptions, thereby automatically triggering the fuse and notifying the user with a real-time alarm.

[0123] (2) Bulletin exception information conversion and solution algorithm;

[0124] The advantages are: more user-friendly exception information prompts and solutions. The technical means it adopts are to list various common exceptions, classify them, implement classes through different strategies, or configure some exception conversion mappings according to types (for example: database configuration mapping conversion), and convert the original simple Bulletin into a more user-friendly information display user according to the configuration information of the processor and related dependent components.

[0125] (3) Customize the SiteToHttpBulletinReportingTask component to push to external systems, persist the Bulletin in memory batches, and clean up and release the memory in time to improve database storage performance;

[0126] The advantages are: using the flexible expansion mechanism of the NiFi framework itself, custom component nar package, running in the NiFi process, no need to change any NiFi code and configuration, zero injection into the system, the technical means adopted is to implement the custom NiFi nar package, placed in the NiFi dependency directory can be loaded and used, the SiteToHttpBulletinReportingTask component used does not require a lot of resources, only a single thread can achieve the purpose, and clean up Bulletin in time to release memory.

[0127] (4) Workflow fuse configuration, creation and operation, recovery, manual cancellation / confirmation, and automatic triggering of multiple algorithms;

[0128] The advantages are: the circuit breaker configuration is simple and easy to implement, the algorithm design supports recovery / cancellation operations, and it supports multiple modes of silent + user confirmation to achieve a more friendly and automated protection mechanism.

[0129] Among them, NiFi is a high-performance data integration ETL system. Flow is a workflow. A simple NiFi workflow usually consists of one or several processors, a group of processors connected by a connection queue. The execution of a workflow is reflected in the generation of a flow file by the first processor, which flows through the connection queue to the downstream processor until the last processor completes the processing. ProcessGroup is a processing group, similar to a folder in an operating system, and can usually be composed of multiple sub-processing groups or several processors. Processor is a processor, a component that processes data (flow files). FlowFile is a flow file, a carrier that carries data. The behavior of each processor processing data is reflected in the process of processing a flow file. Connection is a connection queue, which is used to connect pipelines of different NiFi processors, which is used to transmit flow files. Retry queue is a connection queue for configuring failed retries, that is, the processor can connect to itself. ControllerService is a controller service, a component type used in a processor, used for common use between multiple workflows to achieve object resource sharing and reuse, generally used for resource class access, operations, etc., such as: database connection pool, file reader, etc. Bulletin is a bulletin board in NiFi, which is a log text describing the abnormal operation of NiFi components. For example, when a processor runs once, the abnormal information will be displayed as a message to the user. SiteToHttpBulletinReportingTask is a custom NiFi component, which is a scheduled component used to push incrementally generated Bulletins to the http server at a scheduled time. Circuit breaker refers to the means of stopping the operation of one or more processors in the workflow.

[0130] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a workflow operation abnormality detection and fuse device for implementing the workflow operation abnormality detection and fuse method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more workflow operation abnormality detection and fuse devices provided below can refer to the limitations of the workflow operation abnormality detection and fuse method above, and will not be repeated here.

[0132] In one embodiment, Fig.10 As shown, a detection and fusing device for abnormal operation of a workflow is provided, comprising: an abnormal announcement acquisition module 1001, an abnormal analysis module 1002, and a fusing module 1003, wherein:

[0133] The abnormal announcement acquisition module 1001 is used to obtain abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to represent the working status during the processing of the workflow;

[0134] The abnormality analysis module 1002 is used to persist the abnormality announcement information, perform diagnostic analysis on the persisted abnormality announcement information, and filter the target announcement information from the abnormality announcement information based on the diagnostic analysis result; the target announcement information is the abnormality announcement information whose occurrence frequency meets the preset frequency;

[0135] The fusing module 1003 is used to determine the fusing object associated with the abnormal workflow when the target announcement information meets the preset fusing rule, and generate fusing indication information for the fusing object.

[0136] Each module in the above-mentioned workflow operation abnormality detection and fuse device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

[0137] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a detection and fusing method for abnormal operation of a workflow is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0138] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0141] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0142] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0143] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.< / xxx> < / bulletin> < / xxx>

Claims

1. A method for detecting and fusing abnormal workflow operation, characterized in that: The method comprises: Acquire the abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow; The abnormal announcement information is persisted, and the abnormal announcement information after persistence is diagnosed and analyzed, and target announcement information is obtained from the abnormal announcement information based on the diagnosis and analysis result; the target announcement information is the abnormal announcement information whose occurrence frequency meets the preset frequency; In the case where the target announcement information complies with a preset fusing rule, a fusing object associated with the abnormal workflow is determined, and fusing indication information for the fusing object is generated.

2. The method according to claim 1, characterized in that After the step of filtering the target announcement information from the abnormal announcement information based on the diagnosis and analysis result, the method further includes: Identify the abnormal type corresponding to the target announcement information; According to the exception type corresponding to the target announcement information, the exception announcement content in the target announcement information is converted to obtain converted announcement information; the converted announcement information includes exception description information and solution suggestion information matching the exception description information.

3. The method according to claim 1, characterized in that The obtaining of abnormal announcement information corresponding to the abnormal workflow in the workflow processing system and the status information of the processing group includes: Configure and execute a first component to obtain abnormal announcement information corresponding to an abnormal workflow in the workflow processing system; the first component is a component for regularly pushing incremental abnormal announcement information; A second component is configured and executed to obtain status information of the processing group; the second component is used to periodically push the status information of the processing group.

4. The method according to claim 1, characterized in that: The persisting of the abnormal announcement information includes: Obtaining persistence condition information; the persistence condition information includes a preset number threshold corresponding to the minimum number of persistence entries, and a preset time period corresponding to the interval time; The current number of items to be persisted is obtained according to the preset time period. When the current number of items to be persisted is greater than the preset number threshold, the exception announcement information to be persisted is stored in the persistence database; and when the current number of items to be persisted is less than or equal to the preset number threshold, the exception announcement information to be persisted is stored in a preset memory queue, so that the exception announcement information to be persisted is stored in the persistence database according to the memory queue.

5. The method according to claim 2, characterized in that: The method further comprises: Generate alarm notification information according to the fuse indication information for the fuse object and the converted announcement information; Send the alarm notification information to the user terminal; wherein the alarm notification information includes the cluster address of the workflow processing system, the workflow name, the workflow access path, the exception cause recorded in the converted announcement information, the failure start time, the cumulative number of failed retries, and the solution.

6. The method according to claim 2, characterized in that The fuse indication information includes a fuse task, and the method further includes: When the HTTP service corresponding to the workflow processing system crashes and restarts, and the generated fuse task has not yet been executed, the fuse execution plan record recorded in the database queried after the restart; The fusing task is regenerated and executed according to the fusing execution plan record obtained by querying.

7. A device for detecting and fusing abnormalities in workflow operation, characterized in that: The device comprises: The abnormal announcement acquisition module is used to obtain the abnormal announcement information corresponding to the abnormal workflow in the workflow processing system, and the status information of the processing group; the processing group is a group of processors used to process the workflow, and the status information of the processing group is used to characterize the working status during the processing of the workflow; An abnormality analysis module, used to persist the abnormality announcement information, perform diagnostic analysis on the abnormality announcement information after persistence, and filter out target announcement information from the abnormality announcement information based on the diagnostic analysis result; the target announcement information is the abnormality announcement information whose occurrence frequency meets the preset frequency; A fusing module is used to determine the fusing object associated with the abnormal workflow and generate fusing indication information for the fusing object when the target announcement information meets the preset fusing rules.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images