Method and system for screening, discovering and intelligently predicting data flow and cluster exception

Abstract

The invention relates to the technical field of data processing, in particular to a method and system for screening, discovering and intelligently predicting data flow and cluster anomalies.The method comprises the steps that log data are obtained through a TailFile processor, and the obtained log data comprise flow log data and server data; a single machine or cluster NiFi is deployed, the NiFi is used as a data processing and distributing process, process log data are directly distributed and stored to be called and displayed by a front end, and server data are distributed to an InvokeHTTP processor for further diagnosis and prediction; the InvokeHTTP processor sends a POST request, calls a small sample learning anomaly diagnosis and prediction model combined with a crossform and an attention mechanism to perform anomaly diagnosis and prediction on server data, and stores the server data to be called and displayed by a front end if an anomaly diagnosis and prediction result is that anomaly exists; and the front end integrates and processes the log data stored in the database, and provides a one-key query function on the basis of displaying an exception process so as to provide an exception solution.

Description

Technical Field

[0001] The embodiments of this application relate to the field of data processing technology, and in particular to a method and system for screening, detecting and intelligently predicting data flow and cluster anomalies. Background Technology

[0002] With the rapid development of modern information technology, the world has entered the era of "Internet + Big Data," which has also spurred the rapid development of artificial intelligence technology. In recent years, various enterprise data governance application platforms have sprung up rapidly. These platforms provide one-stop services such as data integration, data development, data science, data cleaning, data analysis, data services, and data visualization. For example, the typical ETL tool Apache NiFi, based on a web graphical interface, allows users to complete process-based programming through drag-and-drop, connection, and configuration, realizing functions such as data collection, cleaning, verification, and transformation, bringing great convenience. As business scenarios increase, the processes of data processing and distribution systems like Apache NiFi are becoming increasingly complex, and the demand for computing resources is also growing, making cluster mode inevitable. However, in actual use, sudden anomalies are unavoidable.

[0003] Chinese invention patent CN118433646A proposes a method for implementing process SMS alarms based on NiFi. This method, through a bulletin board information acquisition module, a custom script parsing anomaly information module, and an SMS alarm sending module, can monitor numerous configured processes in a timely manner and notify specific operations and maintenance personnel via SMS alarms, ensuring timely response and handling of process anomalies. However, this method does not allow developers to intuitively obtain anomaly information and quickly resolve anomalies, nor does it provide an understanding of the overall process status. Furthermore, the anomaly collection method is limited to NiFi big data processing processes and lacks universality.

[0004] Chinese invention patent CN118211170A discloses a server fault diagnosis method. This method first aggregates acquired server fault logs into log groups based on server addresses. Then, it performs vectorization and clustering processing on the log groups to determine their labels. The log groups are stored in a first preset format, and a server fault diagnosis model is trained to assign labels to newly collected server fault logs. This achieves automatic, efficient, and accurate annotation of abnormal information data in server logs, reducing manual labor costs and improving the quality and efficiency of log annotation, thus enhancing the efficiency and accuracy of server fault diagnosis results. However, this method does not predict the time of fault occurrence, and the future state of the cluster servers is unknown.

[0005] Chinese invention patent CN118035061A proposes a deep learning-based method for server fault prediction and automatic processing. This method first acquires various server data prior to a potential fault to form a server dataset. This dataset is then preprocessed, and a server fault prediction model is created and trained using the preprocessed dataset. Furthermore, real-time data and log information from the server's operation are collected to form a real-time server dataset. This real-time dataset is then input into the server fault prediction model to predict whether a server fault will occur. If so, the predicted fault type is determined, and automatic processing is performed based on the determination result. This method achieves accurate fault prediction before a fault occurs and timely processing before it happens, effectively reducing maintenance costs. However, this method does not employ lightweight processing in the log data acquisition, cleaning, and standardization stages. For resource-constrained servers, log processing may be computationally intensive. Additionally, the method does not provide an integrated workflow from data acquisition to final result display. Summary of the Invention

[0006] In view of this, embodiments of this application propose a method and system for screening, discovering and intelligently predicting data flow and cluster anomalies. This system can acquire anomaly information of different clusters and processes anytime and anywhere, realize the diagnosis and prediction of the cluster status of the process, ensure the stable operation of the process, and has universality. It can realize an integrated process from data acquisition and processing to final result display, effectively improving the response efficiency of anomaly resolution and prediction.

[0007] To achieve the above objectives, embodiments of this application propose a method for screening, discovering, and intelligently predicting data flow and cluster anomalies, implemented based on NiFi. The method includes the following steps: using the TailFile processor, a component built into NiFi, to acquire log data in real time. The acquired log data includes process log data and server data; deploying a single-machine NiFi or clustered NiFi, using NiFi as a data processing and distribution process; directly distributing process log data into a database for front-end display; distributing server data to the InvokeHTTP processor for further diagnosis and prediction; the InvokeHTTP processor sending a POST request to invoke a small-sample learning anomaly diagnosis and prediction model combining crossformer and attention mechanisms to perform anomaly diagnosis and prediction on the server data; if the anomaly diagnosis and prediction result indicates the presence of anomalies, storing the server data in a database for front-end display; and the front-end integrating and further processing the stored log data, providing a one-click query function based on the displayed anomalies, thereby providing solutions to the anomalies.

[0008] To achieve the above objectives, embodiments of this application also propose a system for screening, discovering, and intelligently predicting data flow and cluster anomalies, implemented based on NiFi. The system includes: a data acquisition module, used to acquire log data in real time using the TailFile processor, a component built into NiFi; the acquired log data includes process log data and server data; a distribution module, used to deploy a single-machine NiFi or clustered NiFi, using NiFi as a data processing and distribution process; for process log data, it is directly distributed into a database for front-end display; for server data, it is distributed to the InvokeHTTP processor for further diagnosis and prediction; a prediction module, used to instruct the InvokeHTTP processor to send a POST request, calling a small-sample learning anomaly diagnosis and prediction model combining crossformer and attention mechanisms to perform anomaly diagnosis and prediction on the server data; if the anomaly diagnosis and prediction result indicates the presence of anomalies, the server data is stored in a database for front-end display; and a display module, used to integrate and further process the stored log data through the front-end, providing a one-click query function based on the displayed anomalies, thereby providing solutions to the anomalies.

[0009] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a method for screening and intelligently predicting data flow and cluster anomalies as described above.

[0010] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a method for screening and intelligently predicting data flow and cluster anomalies as described above.

[0011] Optionally, the TailFile processor, a component included with NiFi, is used to acquire log data in real time, including: If NiFi is a standalone NiFi, log data is directly obtained through the TailFile processor and sent to the next NiFi component; if NiFi is a clustered NiFi, the smaller, less resource-intensive edge data acquisition tool MiNiFi is used in conjunction with the TailFile processor as the log data acquisition tool. The TailFile processor is used to obtain log data in real time and the Remote Processor Group is used to send it to the remote NiFi. Specifically, TailFile processors are set up to collect process log data and server data respectively. Parameters including the file path where the data is located are configured. After obtaining the process log data, a RouteOnContent processor is added to set specific fields to perform preliminary filtering of the process log data, reducing the pressure of sending data.

[0012] Optionally, when using NiFi as the data processing and distribution process, the acquired log data is judged by NiFi's built-in component EvaluateJsonPath combined with RouteOnAttribute, and distributed to two ExecuteScript processors, and standardized using a custom script; if MiNiFi is used in conjunction with the TailFile processor as the log data acquisition tool, the log data acquired by MiNiFi is received by NiFi using NiFi's built-in component Input Port.

[0013] Optionally, the few-shot learning anomaly diagnosis prediction model combining crossformer and attention mechanisms consists of a crossformer model and a few-shot learning anomaly diagnosis prediction model. The few-shot learning anomaly diagnosis prediction model combining crossformer and attention mechanisms is called to perform anomaly diagnosis prediction on the server data. If the anomaly diagnosis prediction result indicates the presence of anomalies, the server data is stored in the database for front-end display, including: First, the crossformer model performs anomaly diagnosis and prediction on the server data for a period of time in the future. Then, the small sample anomaly diagnosis model combines the output of the crossformer model to diagnose the type of failure that will occur in the future. Finally, a JSON string with the time and type of the anomaly is output as the result of the anomaly diagnosis and prediction. The system uses NiFi's built-in component EvaluateJsonPath in conjunction with RouteOnAttribute to determine whether the anomaly diagnosis prediction results contain the anomaly occurrence time and anomaly type. If so, the server data is stored in the database for front-end display. If not, the data is sent to the component LogeMessage for disposal.

[0014] Optionally, the crossformer model consists of a two-stage attention layer, an encoder, and a decoder. Server data is embedded into a two-dimensional vector array by dimension to be input into the crossformer model. The two-stage attention layer is equipped with a multi-head attention mechanism across time stages and a routing attention mechanism across dimension stages. The few-shot learning anomaly diagnosis prediction model consists of an input layer, a feature fusion layer, an attention layer, a bidirectional GRU layer, a global average pooling layer, and an output layer. The input layer performs dual-path convolution on the output of the crossformer model, and the output layer consists of a fully connected layer and a Softmax function.

[0015] Optionally, while calling the few-shot learning anomaly diagnosis prediction model that combines crossformer and attention mechanisms to perform anomaly diagnosis prediction on the server data, the server data is saved separately to update the training data of the few-shot learning anomaly diagnosis prediction model that combines crossformer and attention mechanisms. During the save process, the database tables are first created, then the server data is converted into a standardized data format for database entry using JoltTransformJSON, then connected to the ConvertJSONToSQL processor with relevant parameters configured, then connected to the PutSQL processor with relevant parameters configured, and finally the server data is saved as a different file.

[0016] Optionally, displaying anomalies includes displaying the anomaly type, anomaly time, anomaly module ID, and specific anomaly content in the anomaly process, as well as displaying the predicted future anomaly time and anomaly type for the server; Common error handling solutions are pre-stored in the database, providing a one-click query function; Under the local area network, if the one-click query function is triggered, it will automatically search the database for matching solutions to the exception. If a matching solution is found, it will determine whether the matching solution can solve the current exception. If it can solve the current exception, the matching solution will be displayed. Otherwise, it will jump to the wide area network to search for solutions to the exception, display the searched solutions, and modify, supplement or delete the matching solutions. If no matching solution is found, the system will redirect to the wide area network to search for solutions, display the found solutions, add them as common solutions, and store them in the database.

[0017] This application proposes a method for screening, detecting, and intelligently predicting data flow and cluster anomalies. It utilizes the TailFile processor, a component built into NiFi, to acquire real-time log data, including process logs and server data. This lightweight log collection addresses the issue of low efficiency in anomaly detection. The smaller, less resource-intensive edge data acquisition tool, MiNiFi, allows for the acquisition of anomaly information from different processes across different clusters anytime, anywhere. After deploying a single-machine or clustered NiFi, NiFi is used as the data processing and distribution process. Process log data is directly distributed to the database for front-end display. Server data is distributed to the InvokeHTTP processor for further diagnosis and prediction. The InvokeHTTP processor sends a POST request to invoke a small-sample learning anomaly diagnosis and prediction model combining CrossFormer and attention mechanisms to perform anomaly diagnosis and prediction on the server data. If the anomaly diagnosis and prediction result indicates the presence of an anomaly, the server data is stored in the database for front-end display. This method enables the prediction of future anomaly times and types, overcoming the limitations of traditional methods that only provide post-event alerts and truly achieving proactive prevention. By integrating anomaly information at the front end and providing a one-click query function, non-professionals can quickly obtain solutions, significantly improving response efficiency and facilitating staff to quickly investigate anomalies. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0019] Figure 1 This is a flowchart of a data flow and cluster anomaly screening, detection, and intelligent prediction method provided in one embodiment of this application; Figure 2 This is a detailed schematic diagram of a method for screening and intelligently predicting data flow and cluster anomalies provided in one embodiment of this application; Figure 3 This is a schematic diagram illustrating the working principle of a few-shot learning anomaly diagnosis prediction model combining crossformer and attention mechanisms provided in one embodiment of this application. Figure 4 This is a schematic diagram illustrating the working principle of the one-click query function provided in one embodiment of this application; Figure 5This is a schematic diagram of the structure of a system for screening and detecting data flow and cluster anomalies and providing intelligent prediction, provided in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0021] With the widespread deployment of big data platforms in enterprise applications, the stability of data processes and cluster systems directly impacts business continuity. Typical data processing tools like Apache NiFi are used to build high-throughput, visualized data processing pipelines, but their operation is often accompanied by complex process anomalies (such as task blocking and data loss) and server resource anomalies (such as CPU overload and memory leaks). Traditional operations and maintenance rely on manual inspection of log files, resulting in low efficiency and delayed response. Most existing monitoring systems focus on single-dimensional alarm mechanisms, lacking the ability to collaboratively analyze process and cluster status, and are unable to predict potential future anomalies. Furthermore, anomaly handling typically relies on the experience and judgment of professional operations and maintenance personnel; non-technical personnel struggle to quickly locate and resolve problems, leading to prolonged fault recovery cycles. Therefore, there is an urgent need for an integrated anomaly screening and prediction method that can achieve full-process automation, proactive predictive capabilities, and intelligent decision support.

[0022] In view of this, one embodiment of this application proposes a method for screening and intelligently predicting data flow and cluster anomalies, implemented based on NiFi. The implementation details of the method for screening and intelligently predicting data flow and cluster anomalies proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0023] The specific process of the data flow and cluster anomaly screening, detection, and intelligent prediction method proposed in this embodiment can be as follows: Figure 1 As shown, specific details are as follows: Figure 2 As shown, it includes: Step 11: Use NiFi's built-in TailFile processor to acquire log data in real time. The acquired log data includes process log data and server data.

[0024] In practical implementation, the first step in anomaly screening and intelligent prediction is to collect log data. This embodiment uses NiFi's built-in TailFile processor to acquire log data in real time, including process log data and server data. The TailFile processor is a stream processing component specifically designed to monitor changes in text files and read newly added content line by line, making it particularly suitable for scenarios with continuously generated log files. This processor can be configured to monitor process log files (e.g., logs recording data process execution status and node operation information) and server operation indicator logs (e.g., system performance monitoring logs and resource utilization logs) under specific paths, thereby simultaneously acquiring two types of key data sources. During the acquisition process, features such as breakpoint resume and file scrolling recognition are supported to ensure that no data is missed and no duplicate readings occur. The acquired data is transmitted to subsequent processing stages via NiFi data streams, forming a stable and reliable data input foundation.

[0025] If the NiFi is a standalone NiFi instance, log data is directly acquired via the TailFile processor and sent to the next NiFi component. If the NiFi is a clustered NiFi instance, the smaller, less resource-intensive edge data acquisition tool MiNiFi is used in conjunction with the TailFile processor as the log data acquisition tool. The TailFile processor acquires log data in real time and sends it to the remote NiFi instance using the Remote Processor Group. Specifically, TailFile processors are configured for process log data and server data, with parameters including the file path where the data is located. After acquiring the process log data, a RouteOnContent processor is added, setting specific fields to perform preliminary filtering of the process log data, thereby reducing the pressure on data transmission.

[0026] This embodiment improves adaptability and operational efficiency in different scale scenarios by selecting different data acquisition strategies for single-machine and cluster environments and classifying and filtering the acquired data. This solution is particularly suitable for big data platform environments with multiple types of data sources, heterogeneous nodes, or resource-constrained edge devices. When NiFi is deployed in single-machine mode, it can directly call NiFi's built-in TailFile processor to read log data. The TailFile processor can continuously monitor the new content of the specified log file, capture it line by line and encapsulate it into NiFi FlowFile objects, and then pass the data to subsequent processors for processing. This method does not require additional proxy components, has a simple structure, and is easy to deploy, making it suitable for use in test environments or small production systems. In the scenario where NiFi is deployed in a cluster, considering the load pressure of the central node and the possible computing resource limitations of the edge nodes, this embodiment introduces MiNiFi as a lightweight edge acquisition agent. MiNiFi is a sub-project of NiFi, designed specifically for resource-constrained environments. Its core engine is small in size (usually less than 50MB of memory), has a fast startup speed, few dependencies, and can run stably in embedded devices, virtual machines, or containers. MiNiFi also supports the TailFile processor, which can be deployed on various service nodes for real-time collection of local log data. After collection, MiNiFi securely and reliably transmits the FlowFile to the remote NiFi node via the Remote Process Group mechanism.

[0027] For different types of data sources, process log data and server data are collected using separate TailFile processors in this embodiment. For example, process log data may come from NiFi's own operation audit logs, ETL task execution records, etc., while server data includes operating system performance indicators, JVM running status, disk I / O statistics, etc. By setting up a separate TailFile instance for each type of data and explicitly specifying parameters such as the monitoring file path, character encoding, and starting position (reading from the end or beginning of the file), logical isolation of data sources and accurate capture are achieved, avoiding parsing confusion and processing conflicts caused by mixed collection.

[0028] Furthermore, after acquiring the process log data, a RouteOnContent processor is introduced to perform preliminary content-level filtering. RouteOnContent can route the FlowFile to different output queues based on its content characteristics (such as the presence of specific keywords, regular expression matching results, JSON field values, etc.). For example, rules can be set to exclude DEBUG-level logs or health check-type heartbeat logs, retaining only WARN and higher-level exception information for subsequent processing. This effectively reduces the amount of invalid data transmitted over the network and the processing burden on the central node, especially in high-concurrency log writing scenarios, reducing bandwidth consumption and database write pressure.

[0029] This embodiment, when facing diverse deployment environments, can flexibly select the optimal log collection architecture based on actual resource conditions. In a single-machine environment, it simplifies the process and reduces complexity; in a cluster environment, it leverages MiniFi to achieve distributed edge collection, alleviating the pressure on the central node. By establishing independent collection channels for process logs and server data, it ensures clear data classification and orderly processing. Combined with the RouteOnContent content filtering mechanism, noise removal is completed at the data source, optimizing the overall data flow transmission efficiency and system response performance, and solving the problems of network congestion and resource overload that traditional centralized collection modes easily cause in large-scale clusters.

[0030] Step 12: Deploy a single-machine NiFi or a clustered NiFi. Use NiFi as the data processing and distribution process. For process log data, distribute it directly into the database for front-end display. For server data, distribute it to the InvokeHTTP processor for further diagnosis and prediction.

[0031] In its implementation, regarding the data processing architecture, this embodiment can select either a standalone NiFi instance or a clustered NiFi instance as the core scheduling hub, depending on the actual deployment environment. When the scale is small and the data volume is limited, deploying a standalone NiFi instance is sufficient. In large-scale cluster environments with multiple nodes and cross-regional operations, a NiFi cluster can be deployed to improve processing capacity and fault tolerance. NiFi's role in this data processing and distribution process is as follows: For process log data, since it is mainly used to trace historical execution status and locate error contexts, no complex modeling analysis is required; it can be directly stored in the database for the front-end to query and display on demand. For server data, because it contains time-series characteristics reflecting the system's health status, further in-depth analysis is required. Therefore, it is directed to the InvokeHTTP processor to prepare for initiating remote model calls.

[0032] In one example, when using NiFi as a data processing and distribution pipeline, the acquired log data is evaluated using NiFi's built-in component EvaluateJsonPath in conjunction with RouteOnAttribute. The data is then distributed to two ExecuteScript processors and standardized using a custom script. If MiNiFi is used with the TailFile processor as the log data acquisition tool, the Input Port component, built-in to NiFi, receives the log data acquired by MiNiFi. The Input Port is a standard component for establishing data transmission channels between NiFi clusters or between the edge and the center, typically used in conjunction with Remote Process Groups. The Input Port runs on the NiFi server, listens to the specified transmission endpoint, securely receives encrypted or authenticated data packets, and injects them into the local processing flow. This mechanism supports breakpoint resumption, flow control, and priority scheduling, making it suitable for industrial environments with limited bandwidth or unstable networks.

[0033] This design enables efficient classification and standardization of process logs and server runtime data in complex data environments. By employing an automated routing mechanism based on attribute extraction and conditional routing, it solves the manual configuration problems caused by inconsistent log formats in traditional methods. Furthermore, the introduction of a customizable script processing module allows the system to quickly adapt to new log types, enhancing its scalability. Simultaneously, by utilizing an Input Port to receive data from remote edge environments, it ensures data connectivity and transmission stability across network scenarios, effectively supporting the smooth execution of subsequent anomaly diagnosis and prediction functions.

[0034] Step 13: The InvokeHTTP processor sends a POST request to call the few-sample learning anomaly diagnosis and prediction model that combines crossformer and attention mechanisms to perform anomaly diagnosis and prediction on the server data. If the anomaly diagnosis and prediction result indicates that anomalies exist, the server data is stored in the database for the front-end to call and display.

[0035] In its implementation, the InvokeHTTP processor is a NiFi component used to initiate POST requests, supporting custom request headers, request body formats, and authentication mechanisms. In this embodiment, the InvokeHTTP processor is configured to send POST requests to the backend deployed machine learning service. The request body encapsulates the server runtime data for the current batch. The target server integrates a crossformer model and a few-shot learning anomaly diagnosis and prediction model. The crossformer model is an improved Transformer architecture designed for long-sequence, multivariate time series forecasting. It effectively captures cross-time and cross-dimensional dependencies between different server metrics, outputting trend predictions of the system state over a future period. Building upon this, the few-shot learning anomaly diagnosis model introduces an attention mechanism, accurately identifying potential anomaly patterns even with only a few labeled samples. It then combines the crossformer model's output to infer the specific anomaly type and expected occurrence time, ultimately returning a JSON string containing the anomaly occurrence time and anomaly type fields as the anomaly diagnosis and prediction result for the server data. A single crossformer model is primarily used for multivariate time series forecasting, effectively capturing time dependencies and correlations between different variables, and server-related data falls into this category. Attention-based few-shot learning-based anomaly diagnosis can effectively diagnose anomalies in situations with limited labeled samples and varying working environments. Combining crossformer multivariate time-series prediction with attention-based few-shot learning-based anomaly diagnosis can simultaneously achieve anomaly prediction.

[0036] In one example, the few-shot learning anomaly diagnosis prediction model combining crossformer and attention mechanisms consists of a crossformer model and a few-shot learning anomaly diagnosis prediction model. When the few-shot learning anomaly diagnosis prediction model combining crossformer and attention mechanisms is called to perform anomaly diagnosis prediction on server data, the crossformer model first performs anomaly diagnosis prediction on the server data for a period of time in the future. Then, the few-shot anomaly diagnosis model combines the output of the crossformer model to diagnose the type of failure that will occur in the future. Finally, a JSON string containing the time of occurrence of the anomaly and the type of the anomaly is output as the result of the anomaly diagnosis prediction.

[0037] In one example, the working principle of a few-shot learning anomaly diagnosis prediction model that combines crossformer and attention mechanisms can be described as follows: Figure 3 As shown.

[0038] The Crossformer model consists of a two-stage attention layer, an encoder, and a decoder. Server data is embedded into a two-dimensional vector array, segmented by dimension, and input into the Crossformer model. The two-stage attention layer incorporates a multi-head attention mechanism across time stages and a routing attention mechanism across dimensional stages. The Crossformer model is suitable for processing high-dimensional, long-period time series data. The encoder maps historical server monitoring data into a latent state representation, which the decoder uses to extrapolate trends for future time periods. The two-stage attention layer, as a core module, is embedded within the encoder, enabling cross-time and cross-dimensional information interaction modeling, respectively. The encoder can be composed of multiple stacked two-stage attention blocks, each followed by a feedforward neural network and a residual connection structure to enhance nonlinear expressiveness and gradient propagation stability. The decoder also contains a similar attention structure and introduces a masking mechanism to prevent information leakage and ensure the effectiveness of autoregressive predictions.

[0039] For the statement "server data is embedded into a two-dimensional vector array by dimension and input into the Crossformer model," "dimension-based segmentation" refers to dividing the raw, multivariate server data (such as CPU utilization, memory usage, disk I / O rate, network throughput, etc.) into independent data channels or subsequences according to different monitoring metrics. Each dimension is processed as an independent time series. These single-dimensional time series are then uniformly resampled to the same time granularity and concatenated to form a two-dimensional matrix data structure, i.e., a tensor with the behavior as the time step and the columns as the metric dimensions. This two-dimensional vector array serves as the input to the Crossformer model, preserving the spatial structure characteristics of the original data, which facilitates the subsequent attention mechanism to model separately along the two axes.

[0040] The key innovation of this embodiment lies in its "two-stage attention layer incorporating a multi-head attention mechanism across time stages and a routing attention mechanism across dimensions." The multi-head attention mechanism across time stages operates along the time axis, allowing the model to focus on key time points in the historical sequence in parallel across different representation subspaces. This effectively captures long-term dependencies and periodic patterns. Its calculation follows the Scaled Dot-Product Attention mechanism in the standard Transformer, but restricts attention weights to calculation only along the time dimension, processing each dimension independently before merging. The routing attention mechanism across dimensions operates along the indicator dimension, aiming to uncover the coupling relationships between different resource indicators. For example, high CPU load is often accompanied by increased memory usage or disk write latency. This mechanism employs a dynamic gating strategy similar to MoE, automatically allocating attention weights based on the importance of each dimension at the current time step, achieving sparsity and targeted feature selection. The two attention mechanisms are executed alternately or integrated in parallel, forming a progressive modeling process of "first looking at time changes, then looking at dimensional linkages," significantly improving the model's ability to understand the behavior of complex systems.

[0041] In one example, the few-shot learning anomaly diagnosis prediction model consists of an input layer, a feature fusion layer, an attention layer, a bidirectional GRU layer, a global average pooling layer, and an output layer. The input layer performs dual-path convolution on the output of the crossformer model, and the output layer consists of a fully connected layer and a Softmax function.

[0042] The few-shot learning anomaly diagnosis prediction model receives latent representations (usually future anomaly probability distributions or trend embedding vectors) from the output of a crossformer model, and then performs refined anomaly type identification based on these representations. The input layer first performs format adaptation and normalization on the received crossformer output, preparing it for feature extraction. The feature fusion layer integrates information from different time windows or prediction paths, achieving a unified representation of multi-source information through methods such as concatenation, weighted summation, or gated fusion. The attention layer introduces an attention mechanism again, focusing on the feature fragments most likely to indicate an anomaly and suppressing redundant noise interference. The bidirectional GRU layer performs forward and backward scanning of sequential features to fully capture contextual dependencies. The global average pooling layer compresses the temporal dimension into a fixed-length vector representation, removing the influence of time indices and facilitating the final classification decision. The output layer consists of a fully connected layer and a Softmax function. The former maps the pooled features to a predefined anomaly category space, while the latter outputs the probability distribution of each category, thereby determining the most likely anomaly type.

[0043] By combining a few-sample learning anomaly diagnosis and prediction model with crossformer and attention mechanisms, this embodiment achieves deep spatiotemporal modeling of server runtime data and accurate anomaly classification under low-resource conditions. Due to the adoption of a two-stage attention mechanism, crossformer can more effectively capture temporal evolution patterns and cross-indicator correlations, solving the problem that traditional models struggle to simultaneously consider temporal dynamics and variable interactions. Simultaneously, the few-sample anomaly diagnosis model enhances feature representation capabilities through dual-path convolution and attention mechanisms, enabling stable and reliable anomaly type judgments even in scenarios with scarce labeled data. Therefore, this composite model structure not only improves the accuracy of predicting the timing and type of future anomalies but also possesses good engineering applicability, continuously supporting intelligent operation and maintenance decisions in real-world production environments.

[0044] In one example, if the anomaly diagnosis prediction indicates the presence of an anomaly, the server data is stored in the database for front-end display. This includes using NiFi's built-in component EvaluateJsonPath in conjunction with RouteOnAttribute to determine whether the anomaly diagnosis prediction result contains the anomaly occurrence time and anomaly type. If so, the server data is stored in the database for front-end display; otherwise, it is sent to the component LogeMessage for discarding.

[0045] This processing mechanism ensures that only data with genuine early warning value enters the downstream display stage, improving response efficiency and reducing data redundancy pressure during long-term operation. Furthermore, since the entire decision-making process is completed entirely by the NiFi native component chain without the need for external scripts, the stability and maintainability of the process are guaranteed.

[0046] In one example, while calling a few-shot learning anomaly diagnosis prediction model combining crossformer and attention mechanisms to perform anomaly diagnosis prediction on server data, the server data is simultaneously saved to update the training data of the few-shot learning anomaly diagnosis prediction model combining crossformer and attention mechanisms. During the saving process, the database tables are first created, then the server data is converted to a standardized input data format using JoltTransformJSON, then connected to the ConvertJSONToSQL processor with relevant parameters configured, and finally connected to the PutSQL processor with relevant parameters configured, ultimately achieving the saving of the server data.

[0047] Each time the InvokeHTTP processor initiates a call to the "crossformer model" and the "few-shot learning anomaly diagnosis and prediction model," the original server data is routed in parallel to another independent data processing branch for persistent storage. This branch first determines whether a dedicated table structure for storing historical server data exists in the target database. If not, it creates a data table conforming to preset field specifications by executing DDL statements, including but not limited to key indicator columns such as timestamp, host identifier, CPU utilization, memory usage, disk I / O, and network throughput, ensuring that subsequent write operations have basic environmental support.

[0048] Subsequently, NiFi's built-in JoltTransformJSON processor was used to transform the format of the raw server data. JoltTransformJSON, based on predefined transformation rules, converts unstructured or semi-structured log output into a unified, standardized JSON format. This step ensures that all new data remains consistent with the existing training set in terms of type, units, and naming conventions, preventing model retraining failure due to data drift.

[0049] After format standardization, the data enters the ConvertJSONToSQL processor. This processor automatically converts the standard JSON object into the corresponding INSERT INTO SQL statement based on the target table name and field mapping relationships configured by the user. During this process, you can configure whether to enable batch insert mode to improve write efficiency, and you can also set conflict handling strategies (such as ignoring duplicate primary keys or performing updates).

[0050] Finally, the generated SQL statements are executed by the PutSQL processor, which communicates with the backend database (such as MySQL, PostgreSQL, or Apache Derby) through a pre-established JDBC connection pool to write the data into the physical tables. PutSQL supports transaction control, connection timeout retries, and error logging to ensure data integrity and system stability in high-concurrency scenarios. Furthermore, NiFi's Provenance feature allows tracking the flow path of each data entry, facilitating later auditing and troubleshooting.

[0051] The aforementioned components are asynchronously decoupled through NiFi's queue buffering mechanism. Even if the database is temporarily unavailable, the data will not be lost but will be temporarily stored in the transmission queue awaiting recovery. In addition, the entire data saving process can be flexibly adjusted in frequency according to the actual load. For example, it can be set to trigger archiving only when the prediction result for the day is determined to be "abnormal", or a fixed sampling rate can be used for periodic saving, thereby balancing the relationship between storage costs and model training data coverage.

[0052] Through the above design, this embodiment achieves the continuous accumulation of server status data in a real-world operating environment without affecting the performance of the main prediction process. Since all operations are completed within the NiFi native component chain, no additional middleware or scripts are required, significantly reducing system complexity and operational burden. More importantly, the accumulated data inherently possesses time-series characteristics and annotation context (such as whether a real failure subsequently occurred), providing a solid foundation for optimizing the time-series modeling capabilities of the crossformer model and improving the generalization ability of the few-shot learning model on rare anomaly categories.

[0053] Step 14 involves the front-end integrating and further processing the log data entering the database, providing a one-click query function based on the displayed exception process, thereby providing solutions to the exception.

[0054] In its implementation, after data processing and distribution are complete, the front-end can integrate and further process the imported log data. It provides a one-click query function based on the displayed abnormal process, thereby offering solutions to the anomalies. The front-end periodically polls the database to retrieve imported process anomaly logs and server-predicted anomaly information, presenting them centrally on a visual interface. The displayed content includes elements such as the anomaly occurrence time and anomaly type, allowing users to comprehensively understand the system's operational status. More importantly, the front-end integrates an anomaly solution knowledge base, supporting automatic matching and one-click query functions based on anomaly type, greatly lowering the barrier to problem handling and enabling developers or junior operations personnel to quickly take countermeasures.

[0055] In one example, displaying an exception includes showing the exception type, exception time, exception module ID, and exception details in the exception process, as well as the predicted future exception time and type for the server.

[0056] In one example, common exception handling methods are pre-stored in the database, providing a one-click query function. The working principle of the one-click query function can be described as follows: Figure 4 As shown.

[0057] The system pre-defines a structured knowledge base for common exception solutions, which is stored in a local database. It contains historically accumulated high-frequency exception cases and their corresponding standardized processing procedures. Each solution is stored in structured fields, including metadata such as applicable exception type, operation steps, expected results, risk warnings, and update records, which facilitates subsequent retrieval and management.

[0058] When a user discovers an anomaly on the front-end interface and clicks the one-click query function, the system automatically extracts the key features of the current anomaly (such as anomaly type, module ID, log keywords, etc.) and prioritizes initiating a matching query to the local knowledge base in a local area network environment. If an entry highly matches the current anomaly, the system further executes validity judgment logic. This judgment can be based on a rule engine or a lightweight classification model to analyze whether the operational conditions of the solution meet the current environmental state (such as whether the target service is online, whether configuration permissions are available, etc.). If it is determined to be executable, the solution is immediately displayed in a highlighted form on the anomaly details page for user reference or direct execution of the automated script.

[0059] If the local query fails to find a match, or if a match is found but deemed unsuitable for the current scenario (e.g., due to environmental differences or insufficient permissions), the system automatically triggers a redirection mechanism. This mechanism encapsulates the error information into a structured query string, guiding the user's browser to open a pre-defined WAN search engine page and automatically filling the search box with relevant keywords, thus improving the efficiency of external information retrieval. The displayed external search results are embedded in the front-end interface as a summary list, allowing users to quickly browse and select feasible solutions.

[0060] Furthermore, the system supports dynamic maintenance of existing solutions. Users can modify, supplement, or mark invalid and delete solutions obtained locally or externally. All changes are recorded with the operator and timestamp, ensuring the accuracy and timeliness of the knowledge base. For novel and effective solutions obtained through wide area network queries, users can choose to structure and reverse-engineer them for storage, adding them to the local knowledge base as new solutions to common anomalies, thereby achieving continuous evolution and closed-loop optimization of the knowledge system.

[0061] This design enables an integrated interactive process for displaying anomaly information and obtaining handling suggestions. Because the front-end not only presents the anomaly itself but also integrates intelligent recommendations for solution paths and cross-network query capabilities, both developers and junior operations personnel can quickly respond to system anomalies without requiring in-depth professional knowledge. Furthermore, the adoption of a local-first, remote-as-backup query strategy balances response speed and coverage breadth. The user-feedback-driven knowledge base update mechanism ensures the system's adaptability and usability during long-term operation, ultimately significantly improving the overall efficiency and service quality of anomaly handling.

[0062] This embodiment proposes a method for screening, detecting, and intelligently predicting data flow and cluster anomalies. It utilizes the TailFile processor, a component built into NiFi, to acquire real-time log data, including process logs and server data. This lightweight log collection solves the problem of low efficiency in anomaly detection. The smaller, less resource-intensive edge data acquisition tool, MiNiFi, allows for the acquisition of anomaly information for different processes across different clusters anytime, anywhere. After deploying a single-machine or clustered NiFi, NiFi is used as the data processing and distribution process. Process log data is directly distributed into the database for front-end display. Server data is distributed to the InvokeHTTP processor for further diagnosis and prediction. The InvokeHTTP processor sends a POST request to invoke a small-sample learning anomaly diagnosis and prediction model combining CrossFormer and attention mechanisms to perform anomaly diagnosis and prediction on the server data. If the anomaly diagnosis and prediction result indicates the presence of an anomaly, the server data is stored in the database for front-end display. This achieves prediction of future anomaly times and types, breaking through the limitations of traditional methods that only provide post-event alerts and truly enabling proactive prevention. By integrating anomaly information at the front end and providing a one-click query function, non-professionals can quickly obtain solutions, significantly improving response efficiency and facilitating staff to quickly investigate anomalies.

[0063] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0064] Another embodiment of this application proposes a system for screening, detecting, and intelligently predicting data flow and cluster anomalies, implemented based on NiFi. The details of this system are described below for ease of understanding and are not essential for implementing this example. The structure of the system for screening, detecting, and intelligently predicting data flow and cluster anomalies proposed in this embodiment is as follows: Figure 5 As shown, it includes: a data acquisition module 21, a distribution module 22, a prediction module 23, and a display module 24.

[0065] The acquisition module 21 is used to acquire log data in real time using the TailFile processor component that comes with NiFi. The acquired log data includes process log data and server data.

[0066] The distribution module 22 is used to deploy a single-machine NiFi or a clustered NiFi. It uses NiFi as a data processing and distribution process. For process log data, it directly distributes it into the database for front-end to call and display. For server data, it distributes it to the InvokeHTTP processor for further diagnosis and prediction.

[0067] Prediction module 23 is used to instruct the InvokeHTTP processor to send a POST request, call the few-sample learning anomaly diagnosis prediction model that combines crossformer and attention mechanism to perform anomaly diagnosis prediction on server data, and if the anomaly diagnosis prediction result indicates that there is an anomaly, store the server data in the database for front-end to call and display.

[0068] The display module 24 is used to integrate and further process the log data entered into the database through the front end, and provides a one-click query function on the basis of displaying anomalies, thereby providing solutions to anomalies.

[0069] It is worth noting that all modules involved in this embodiment are logical modules. In practical applications, a logical module can be a physical module, a part of a physical module, or an organic combination of multiple physical modules. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce modules that are not closely related to solving the technical problems proposed in this application. However, this does not mean that other modules are absent from this embodiment.

[0070] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0071] Another embodiment of this application provides an electronic device, such as Figure 6 As shown, it includes a processor 31 and a memory 32. The memory 32 stores instructions that the processor 31 can execute. When the processor 31 is configured to execute the instructions, the electronic device can realize a method for screening and discovering data flow and cluster anomalies and intelligent prediction as described in the above method embodiment.

[0072] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0073] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0074] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a method for screening and intelligently predicting data flow and cluster anomalies as described in the above method embodiments.

[0075] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0076] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for screening, detecting, and intelligently predicting data flow and cluster anomalies, characterized in that, Based on NiFi, the method includes: The TailFile processor, a component built into NiFi, is used to acquire log data in real time. The acquired log data includes process log data and server data. Deploy a single-machine NiFi or a clustered NiFi, and use NiFi as a data processing and distribution process. For process log data, distribute it directly into the database for front-end call and display. For server data, distribute it to the InvokeHTTP processor for further diagnosis and prediction. The InvokeHTTP processor sends a POST request, calls a few-sample learning anomaly diagnosis and prediction model that combines crossformer and attention mechanisms to perform anomaly diagnosis and prediction on the server data. If the anomaly diagnosis and prediction result indicates that an anomaly exists, the server data is stored in the database for the front-end to call and display. The front-end integrates and further processes the log data entering the database, and provides a one-click query function in addition to displaying anomalies, thereby providing solutions to anomalies.

2. The method according to claim 1, characterized in that, The NiFi built-in TailFile processor is used to acquire log data in real time, including: If NiFi is a standalone NiFi, log data is directly obtained through the TailFile processor and sent to the next NiFi component; if NiFi is a clustered NiFi, the smaller, less resource-intensive edge data acquisition tool MiNiFi is used in conjunction with the TailFile processor as the log data acquisition tool. The TailFile processor is used to obtain log data in real time and the Remote Processor Group is used to send it to the remote NiFi. Specifically, TailFile processors are set up to collect process log data and server data respectively. Parameters including the file path where the data is located are configured. After obtaining the process log data, a RouteOnContent processor is added to set specific fields to perform preliminary filtering of the process log data, reducing the pressure of sending data.

3. The method according to claim 2, characterized in that, When using NiFi as a data processing and distribution process, the acquired log data is evaluated using NiFi's built-in EvaluateJsonPath component in conjunction with RouteOnAttribute, and then distributed to two ExecuteScript processors. Custom scripts are used for standardization processing. If MiNiFi is used in conjunction with the TailFile processor as a log data acquisition tool, the log data acquired by MiNiFi is received using NiFi's built-in Input Port component.

4. The method according to claim 1, characterized in that, The few-shot learning anomaly diagnosis and prediction model, combining Crossformer and attention mechanisms, consists of a Crossformer model and a few-shot learning anomaly diagnosis and prediction model. This model is invoked to perform anomaly diagnosis and prediction on server data. If the anomaly diagnosis and prediction result indicates the presence of an anomaly, the server data is stored in the database for front-end display. This includes: First, the crossformer model performs anomaly diagnosis and prediction on the server data for a period of time in the future. Then, the small sample anomaly diagnosis model combines the output of the crossformer model to diagnose the type of failure that will occur in the future. Finally, a JSON string with the time and type of the anomaly is output as the result of the anomaly diagnosis and prediction. The system uses NiFi's built-in component EvaluateJsonPath in conjunction with RouteOnAttribute to determine whether the anomaly diagnosis prediction results contain the anomaly occurrence time and anomaly type. If so, the server data is stored in the database for front-end display. If not, the data is sent to the component LogeMessage for disposal.

5. The method according to claim 4, characterized in that, The crossformer model consists of a two-stage attention layer, an encoder, and a decoder. Server data is embedded into a two-dimensional vector array by dimension and input into the crossformer model. The two-stage attention layer is equipped with a multi-head attention mechanism across time stages and a routing attention mechanism across dimension stages. The few-shot learning anomaly diagnosis prediction model consists of an input layer, a feature fusion layer, an attention layer, a bidirectional GRU layer, a global average pooling layer, and an output layer. The input layer performs dual-path convolution on the output of the crossformer model, and the output layer consists of a fully connected layer and a Softmax function.

6. The method according to claim 5, characterized in that, While calling the few-shot learning anomaly diagnosis prediction model that combines crossformer and attention mechanisms to perform anomaly diagnosis prediction on server data, the server data is saved separately to update the training data of the few-shot learning anomaly diagnosis prediction model that combines crossformer and attention mechanisms. During the save process, the database tables are first created, then the server data is converted into a standardized data format for database entry using JoltTransformJSON, then connected to the ConvertJSONToSQL processor with relevant parameters configured, then connected to the PutSQL processor with relevant parameters configured, and finally the server data is saved as a different file.

7. The method according to any one of claims 1 to 6, characterized in that, The display of anomalies includes the anomaly type, anomaly time, anomaly module ID, and specific anomaly content in the anomaly process, as well as the predicted future anomaly time and anomaly type for the server. Common error handling solutions are pre-stored in the database, providing a one-click query function; Under the local area network, if the one-click query function is triggered, it will automatically search the database for matching solutions to the exception. If a matching solution is found, it will determine whether the matching solution can solve the current exception. If it can solve the current exception, the matching solution will be displayed. Otherwise, it will jump to the wide area network to search for solutions to the exception, display the searched solutions, and modify, supplement or delete the matching solutions. If no matching solution is found, the system will redirect to the wide area network to search for solutions, display the found solutions, add them as common solutions, and store them in the database.

8. A system for screening, detecting, and intelligently predicting data flow and cluster anomalies, characterized in that, Based on NiFi, the system includes: The acquisition module is used to acquire log data in real time using NiFi's built-in TailFile processor. The acquired log data includes process log data and server data. The distribution module is used to deploy single-machine NiFi or clustered NiFi. It uses NiFi as a data processing and distribution process. For process log data, it is directly distributed into the database for front-end call and display. For server data, it is distributed to the InvokeHTTP processor for further diagnosis and prediction. The prediction module is used to instruct the InvokeHTTP processor to send a POST request, call the few-sample learning anomaly diagnosis prediction model that combines crossformer and attention mechanism, perform anomaly diagnosis prediction on the server data, and if the anomaly diagnosis prediction result is that there is an anomaly, store the server data in the database for the front end to call and display. The display module is used to integrate and further process the log data entered into the database through the front end. It provides a one-click query function on the basis of displaying anomalies, thereby providing solutions to anomalies.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a method for screening, detecting, and intelligently predicting data flow and cluster anomalies as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a method for screening and intelligently predicting data flow and cluster anomalies as described in any one of claims 1 to 7.

Images