Abnormality detection and alarm method, device and equipment based on hierarchical funnel model and storage medium

Abstract

The application discloses a kind of based on layered funnel model exception detection and alarm method, device and equipment and storage medium, it is related to data processing technical field, it discloses based on layered funnel model exception detection and alarm method, comprising: to initial log data is carried out data processing, obtains structured log data;According to the structured log data and layered funnel model is carried out layered monitoring, obtains layered monitoring data, wherein the layered monitoring data includes aggregation monitoring layer data, convergence monitoring layer data and node monitoring layer data;According to the layered monitoring data is carried out exception detection, obtains exception detection result;According to the exception detection result generates hierarchical alarm information.Application scheme can improve the accuracy of exception detection and fault emergency response efficiency, while the dynamic adjustment of organization structure of adaptive mechanism can reduce the operation and maintenance management cost of monitoring system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to anomaly detection and alarm methods, apparatus, devices and storage media based on a hierarchical funnel model. Background Technology

[0002] As banks gradually complete the mainframe-free transformation of their core business architecture, the processing chain for counter services has been broken down into multiple service nodes, making business call paths more complex, and transaction volume and access patterns exhibit significant volatility and variability. Meanwhile, bank organizational structures typically present a multi-tiered system of branches and outlets, with significant differences between different levels in terms of business scale, transaction frequency, peak and off-peak periods, and customer structure.

[0003] Against this backdrop, traditional monitoring methods typically rely on operations and maintenance management components to collect microservice operational metrics and determine alerts based on unified metric definitions and pre-set fixed thresholds. Simultaneously, they centrally aggregate and uniformly analyze metrics across the entire organization. In practical applications, this approach often lacks adaptability to differences across different organizational levels, leading to the need for manual analysis to determine the scope of impact when anomalies occur, resulting in low overall response efficiency. Summary of the Invention

[0004] The main purpose of this application is to provide an anomaly detection and alarm method, device, equipment and storage medium based on a hierarchical funnel model, which aims to solve the technical problem of how to accurately identify and warn of abnormal states in the operation of bank counter channels, taking into account different institutional levels and business differences.

[0005] To achieve the above objectives, this application proposes an anomaly detection and alarm method based on a hierarchical funnel model, the method comprising: The initial log data is processed to obtain structured log data; Based on the structured log data and the hierarchical funnel model, hierarchical monitoring is performed to obtain hierarchical monitoring data, which includes aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data. Anomaly detection is performed based on the hierarchical monitoring data to obtain anomaly detection results; Based on the anomaly detection results, a tiered alarm information is generated.

[0006] In one embodiment, the hierarchical funnel model includes an aggregation monitoring layer, a convergence monitoring layer, and a node monitoring layer connected in sequence. The step of performing hierarchical monitoring based on the structured log data and the hierarchical funnel model to obtain hierarchical monitoring data includes: Data stratification is performed based on the structured log data to obtain the data stratification results; Based on the data layering results, the target data processing layer is matched in the aggregation monitoring layer, convergence monitoring layer, and node monitoring layer. The structured log data is monitored in layers according to the target data processing layer to obtain layered monitoring data.

[0007] In one embodiment, the step of performing layered monitoring on the structured log data according to the target data processing layer to obtain layered monitoring data includes: The time sliding window and indicator aggregation strategy are determined based on the target data processing layer; Based on the time sliding window and the indicator aggregation strategy, the structured log data is monitored in layers to generate initial monitoring indicator data; The initial monitoring index data is converged according to the preset funnel convergence order to obtain hierarchical monitoring data. The preset funnel convergence order is to input the initial monitoring index data into the node monitoring layer, the convergence monitoring layer, and the aggregation monitoring layer in sequence for convergence.

[0008] In one embodiment, the step of determining the time sliding window and indicator aggregation strategy based on the target data processing layer includes: When the target data processing layer is an aggregation monitoring layer, the time sliding window is determined to be a global time sliding window, and the indicator aggregation strategy is determined to be a full-dimensional aggregation strategy. When the target data processing layer is a convergence monitoring layer, the time sliding window is determined to be a regional time sliding window, and the indicator aggregation strategy is determined to be a regional dimension aggregation strategy. When the target data processing layer is a node monitoring layer, the time sliding window is determined to be a terminal-level time sliding window, and the indicator aggregation strategy is determined to be a single business node dimension aggregation strategy.

[0009] In one embodiment, the step of generating graded alarm information based on the anomaly detection results includes: Analyze the anomaly detection result to determine the anomaly level; When the anomaly belongs to the aggregated monitoring layer, hierarchical alarm information is generated based on the global impact scope identifier; When the anomaly belongs to the convergence monitoring layer, hierarchical alarm information is generated based on the regional impact range identifier. When the level to which the anomaly belongs is the node monitoring layer, hierarchical alarm information is generated based on the single node's impact range identifier.

[0010] In one embodiment, the step of performing anomaly detection based on the hierarchical monitoring data to obtain anomaly detection results includes: Analyze the anomaly level corresponding to the hierarchical monitoring data; Load the target dynamic threshold according to the level to which the anomaly belongs; Anomaly detection is performed based on the target dynamic threshold and the hierarchical monitoring data to obtain anomaly detection results.

[0011] In one embodiment, the step of processing the initial log data to obtain structured log data includes: Retrieve context metadata; The initial log data is enhanced by performing business context enhancement processing based on the context metadata to obtain enhanced log data; The enhanced log data is cleaned to obtain cleaned log data; The cleaning log data is processed into structured log data.

[0012] In addition, to achieve the above objectives, this application also proposes an anomaly detection and alarm device based on a hierarchical funnel model. The anomaly detection and alarm device based on a hierarchical funnel model includes: a data cleaning module, used to process the initial log data to obtain structured log data. The hierarchical monitoring module is used to perform hierarchical monitoring based on the structured log data and the hierarchical funnel model to obtain hierarchical monitoring data, wherein the hierarchical monitoring data includes aggregated monitoring layer data, convergent monitoring layer data and node monitoring layer data. Anomaly detection module is used to perform anomaly detection based on the hierarchical monitoring data and obtain anomaly detection results; The anomaly alarm module is used to generate tiered alarm information based on the anomaly detection results.

[0013] Furthermore, to achieve the above objectives, this application also proposes an anomaly detection and alarm device based on a hierarchical funnel model. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the anomaly detection and alarm method based on the hierarchical funnel model described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the anomaly detection and alarm method based on the hierarchical funnel model described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the anomaly detection and alarm method based on the hierarchical funnel model described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: By processing initial log data to obtain structured log data, and then using a hierarchical funnel model to perform layered monitoring, resulting in layered monitoring data including aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data, anomaly detection is performed based on the layered monitoring data to obtain anomaly detection results. Based on these results, tiered alarm information is generated, achieving precise matching between anomaly detection and monitoring levels. This solves the problems of missed detection of minor anomalies and false alarms of normal fluctuations caused by the one-size-fits-all judgment rules in traditional anomaly detection models. By generating tiered alarm information based on anomaly detection results, hierarchical identification of abnormal events and corresponding alarm output are achieved. Compared with existing technologies, the anomaly's level and impact range can be clearly identified without manual drill-down analysis, significantly shortening the anomaly emergency response cycle and significantly improving the accuracy of anomaly detection and the relevance of alarm information. Furthermore, the hierarchical monitoring architecture can directly adapt to the hierarchical characteristics of an organization's organizational structure, eliminating the need for large-scale reconstruction of monitoring rules with organizational restructuring, effectively reducing the operation and maintenance costs of the monitoring system. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the anomaly detection and alarm method based on a hierarchical funnel model provided in this application. Figure 2 This is a schematic diagram of a bank's hierarchical funnel model provided in Embodiment 1 of the anomaly detection and alarm method based on the hierarchical funnel model of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the anomaly detection and alarm method based on the hierarchical funnel model of this application. Figure 4 This is a simplified flowchart of the anomaly detection and alarm method based on a hierarchical funnel model provided in Embodiment 2 of this application; Figure 5 This is a schematic diagram of the module structure of the anomaly detection and alarm device based on the hierarchical funnel model in an embodiment of this application. Figure 6This is a schematic diagram of the device structure of the hardware operating environment involved in the anomaly detection and alarm method based on the hierarchical funnel model in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: data processing is performed on the initial log data to obtain structured log data; hierarchical monitoring is performed based on the structured log data and the hierarchical funnel model to obtain hierarchical monitoring data, wherein the hierarchical monitoring data includes aggregated monitoring layer data, convergent monitoring layer data and node monitoring layer data; anomaly detection is performed based on the hierarchical monitoring data to obtain anomaly detection results; and hierarchical alarm information is generated based on the anomaly detection results.

[0024] In this embodiment, for ease of description, the following description focuses on identifying anomaly detection and alarm devices based on a hierarchical funnel model.

[0025] Due to the limitations of existing technologies in accurately identifying and issuing early warnings for abnormal states during bank counter operations, considering different institutional levels and business differences, this application provides a solution. This solution involves processing initial log data to obtain structured log data. Based on this structured log data and a hierarchical funnel model, hierarchical monitoring is performed to obtain hierarchical monitoring data, including aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data. Anomaly detection is then performed based on this hierarchical monitoring data to obtain anomaly detection results. Based on these results, tiered alarm information is generated, achieving precise matching between anomaly detection and monitoring levels, thus solving the problem of traditional... The traditional one-size-fits-all approach to anomaly detection often leads to missed detections of minor anomalies and false alarms of normal fluctuations. By generating tiered alarm information based on anomaly detection results, this approach enables tiered identification of anomalies and corresponding alarm output. Compared to existing technologies, it eliminates the need for manual drill-down analysis to clearly identify the anomaly's level and scope of impact, significantly shortening the anomaly emergency response cycle and greatly improving the accuracy of anomaly detection and the relevance of alarm information. Furthermore, the hierarchical monitoring architecture can be directly adapted to the hierarchical characteristics of an organization's structure, eliminating the need for large-scale reconstruction of monitoring rules as the organizational structure changes, thus effectively reducing the operation and maintenance costs of the monitoring system.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, or an anomaly detection and alarm device based on a hierarchical funnel model, etc. The following description uses an anomaly detection and alarm device based on a hierarchical funnel model as an example to illustrate this embodiment and the subsequent embodiments.

[0027] Based on this, embodiments of this application provide an anomaly detection and alarm method based on a hierarchical funnel model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the anomaly detection and alarm method based on the hierarchical funnel model of this application.

[0028] In this embodiment, the anomaly detection and alarm method based on the hierarchical funnel model includes steps S10 to S40: Step S10: Process the initial log data to obtain structured log data.

[0029] It should be noted that the initial log data is the raw record data of interface calls captured in real time by the counter client during the execution of business interactions. It contains all the interaction information during the business execution process and is the basic data source for all subsequent monitoring and detection processes.

[0030] In this embodiment, the initial log data is captured in real time by the lightweight software development kit (SDK) integrated into the counter client, which fully records the entire process information of each business interface call.

[0031] In addition, data processing involves performing standardized processing operations on the raw, non-standardized log data. This includes core steps such as data cleaning, format standardization, outlier filtering, and business metadata injection. The goal is to transform the scattered and disordered raw logs into standardized data that can be directly called by subsequent calculation processes.

[0032] In this embodiment, the data processing flow is executed by the Apache Flink (Flink) stream processing engine, ensuring the real-time performance and throughput of data processing.

[0033] In addition, structured log data is standardized log data that has been processed through the data processing stage. It has a unified format, complete business context metadata, and can be accurately parsed and classified for calculation. It is the core intermediate data that connects the data collection and hierarchical monitoring processes.

[0034] Understandably, the initial log data generated by the counter client's business interaction is first captured using a lightweight software development kit. Before the data is reported, rich business context metadata is injected into the initial log data, including key fields such as terminal IP, affiliated branch, superior branch, and specific business scenario code. Then, the initial log data with injected metadata is pushed to the Apache Kafka (Kafka) message queue to complete data buffering.

[0035] Furthermore, the initial log data in the message queue is consumed through the Flink stream computing engine, and real-time cleaning, format standardization, and outlier filtering are performed to remove invalid and interfering data, ultimately obtaining structured log data that meets the requirements of subsequent calculations.

[0036] It should be understood that by standardizing and enhancing the metadata of the initial log data, a standardized data foundation that can be accurately stratified, traceable, and aggregated for subsequent tiered monitoring processes is provided. This solves the problems of missing dimensions, inconsistent formats, and inability to adapt to tiered aggregation calculations in the original log data from the source, effectively improving the accuracy of subsequent tiered monitoring and anomaly detection.

[0037] In one feasible implementation, step S10 may include steps S11 to S14: Step S11: Obtain context metadata.

[0038] It should be noted that context metadata is the core attribute data used to identify the business affiliation, organizational level, terminal information, and business scenario of log data. It is the key identifier for realizing the hierarchical distribution and hierarchical aggregation calculation of log data, and can completely restore the full-link background information of the business corresponding to a single log.

[0039] In this embodiment, the context metadata includes core fields such as terminal IP, affiliated branch, superior branch, and specific business scenario code, providing core content for subsequent business context enhancement.

[0040] Understandably, by using the lightweight software development kit integrated into the counter client, while capturing initial log data, the system simultaneously collects all attribute information corresponding to the terminal and the business, extracts core attribute fields that conform to preset specifications, and completes the collection and aggregation of context metadata.

[0041] It should be understood that by collecting complete contextual metadata in advance, accurate attribute support is provided for the subsequent enhanced processing of log data, ensuring that the generated log data can be accurately identified in terms of hierarchical classification, thus laying the foundation for hierarchical calculation of the hierarchical funnel model.

[0042] Step S12: Perform business context enhancement processing on the initial log data based on the context metadata to obtain enhanced log data.

[0043] It should be noted that business context enhancement processing is a processing operation that injects the collected context metadata into the corresponding initial log data according to a preset format specification. The purpose is to supplement the original logs, which originally only contained interface call information, with complete business and organizational attributes, so that a single log has the ability to be layered, traceable, and categorized for statistical analysis.

[0044] In addition, enhanced log data is log data that has undergone business context enhancement processing and contains both original interface call information and complete context metadata. Compared with the initial log data, it has richer business dimension information and can be directly adapted to subsequent cleaning and structuring processes.

[0045] Understandably, the process involves first establishing a unique mapping between a single initial log data entry and its corresponding context metadata, and then, according to predefined field specifications, injecting the core fields from the context metadata into the specified locations of the corresponding initial log data to complete the binding and fusion of metadata and the original log, thereby generating enhanced log data.

[0046] It should be understood that by enhancing the business context, complete hierarchical and business scenario information is added to the initial log data, which solves the problem of missing dimensions in the original logs and inability to adapt to hierarchical aggregation calculations from the source, thereby improving the accuracy and efficiency of subsequent data processing.

[0047] Step S13: Perform data cleaning on the enhanced log data to obtain cleaned log data.

[0048] It should be noted that data cleaning is a process of removing invalid data, filtering interfering data, and verifying the format compliance of log data. The purpose is to remove redundant content, outliers, and invalid records from the log data, ensuring the validity and accuracy of subsequent data processing and avoiding interference from invalid data with subsequent monitoring and calculation results.

[0049] In addition, cleaned log data is log data that has undergone data cleaning operations to remove invalid and interfering data, and whose format meets the specifications. It has the characteristics of data integrity, format compliance and content validity, and can be directly entered into the subsequent structured processing flow.

[0050] Understandably, the enhanced log data is first checked for format compliance according to the preset verification rules, and invalid log data that does not conform to the format is removed. Then, duplicate records, null records and abnormal value data that are outside the normal business scope are filtered out. The compliance verification and invalid content removal of the entire data are completed to obtain clean log data.

[0051] It should be understood that data cleaning effectively removes invalid and interfering content from log data, ensuring the accuracy and validity of the data used in subsequent calculations, avoiding interference from invalid data with the calculation results of monitoring indicators, and improving the accuracy of subsequent hierarchical monitoring and anomaly detection.

[0052] Step S14: Perform structured processing on the cleaning log data to obtain structured log data.

[0053] It should be noted that structured processing is the process of transforming non-standardized cleaning log data into standardized data with fixed format and unified fields, which can be directly recognized and called by the computing engine, according to a preset unified data model and field specifications.

[0054] Understandably, according to the preset unified data model, all fields in the cleaned log data are standardized and mapped and converted in format. Field content in different formats is transformed into standardized content with unified specifications. All fields are classified and organized to generate a standardized data structure that meets the reading requirements of the computing engine, thus obtaining structured log data.

[0055] It should be understood that by structuring the data, the scattered log data is transformed into standardized data that can be directly called by the computing engine. This achieves the unification of log data format and the standardization of fields, providing standardized data input for the hierarchical aggregation calculation of the subsequent hierarchical funnel model, and greatly improving the execution efficiency and accuracy of the subsequent calculation process.

[0056] Step S20: Perform hierarchical monitoring based on the structured log data and hierarchical funnel model to obtain hierarchical monitoring data, wherein the hierarchical monitoring data includes aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data.

[0057] It should be noted that the hierarchical funnel model is a three-layer monitoring and computing architecture that is cascaded from top to bottom, including an aggregation monitoring layer, a convergence monitoring layer, and a node monitoring layer connected in sequence.

[0058] In addition, layered monitoring is a hierarchical architecture based on a layered funnel model. It performs hierarchical distribution, differentiated aggregation calculation and step-by-step convergence full-process monitoring operations on structured log data. The purpose is to break the limitations of traditional flat aggregation monitoring and realize independent monitoring and overall coordination of the business operation status of different management levels.

[0059] In addition, the hierarchical monitoring data is the standardized monitoring indicator data of each level of the hierarchical funnel model generated after the full-process calculation of hierarchical monitoring. It is the core input data of the subsequent anomaly detection process and includes three types of core data: aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data.

[0060] In addition, the aggregated monitoring layer data is the top-level monitoring indicator data in the hierarchical funnel model. It corresponds to the aggregated results of the full business operation status within the global jurisdiction and can fully reflect the business operation health across the entire scope.

[0061] In addition, the convergence monitoring layer data is the monitoring indicator data of the middle level in the hierarchical funnel model. It corresponds to the aggregated results of the business operation status of each region's jurisdiction and can fully reflect the health of business operations within the corresponding jurisdiction.

[0062] In addition, the node monitoring layer data is the monitoring indicator data at the lowest level of the hierarchical funnel model. It corresponds to the calculation result of the business operation status of a single business node and can accurately reflect the business operation health of a single business node.

[0063] In this embodiment, the aggregation monitoring layer can monitor data for the entire bank; the convergence monitoring layer can monitor data for the bank's branches; and the node monitoring layer can monitor data for the bank's outlets. (Refer to...) Figure 2 , Figure 2 This is a schematic diagram of a bank's hierarchical funnel model, which is the first embodiment of the anomaly detection and alarm method based on the hierarchical funnel model of this application.

[0064] like Figure 2As shown, the hierarchical funnel model deployed in a bank can be divided into three levels: the bank-wide level, the branch level, and the outlet level. The core design logic is to set different data collection windows and monitoring thresholds for different levels, thereby greatly improving fault coverage. The top layer of the model is the bank-wide level, which monitors APIs from a bank-wide perspective. The shortest time window is set to 1 minute, enabling rapid detection of bank-wide issues and issuing alerts. The middle layer is the branch level, which monitors APIs and network conditions from a branch perspective. The time window is set to 5 minutes, enabling the detection of issues affecting only a few branches. The bottom layer is the outlet level, which monitors APIs and network conditions from an outlet perspective. The time window is set to 5 minutes, enabling the detection of issues affecting only a few outlets. By implementing a top-down hierarchical structure, the system achieves progressively convergent monitoring from the entire bank to branches and then to outlets. Different time windows and monitoring strategies are matched to different levels. This allows for the rapid identification of large-scale major faults through short time windows at the bank-wide level, as well as the precise location of abnormal issues affecting only local institutions through adaptive windows at the branch and outlet levels. It comprehensively covers fault scenarios with different impact ranges, providing a hierarchical and precise monitoring system for bank counter channel fault monitoring. This effectively solves the problem of false alarms and missed alarms caused by the one-size-fits-all threshold of traditional monitoring, while clearly defining the impact level and scope of faults, supporting subsequent hierarchical alarms and emergency response.

[0065] Understandably, the organizational hierarchy metadata in the structured log data is first parsed to complete the hierarchical affiliation matching and automatic routing of each log data, and the corresponding log data is distributed to the matching hierarchical calculation unit in the hierarchical funnel model.

[0066] Furthermore, differentiated time sliding windows and indicator aggregation strategies are matched to each level of the hierarchical funnel model; based on the matched time sliding windows and indicator aggregation strategies, parallel aggregation calculations are performed on the split structured log data within each level's computing unit to generate the initial monitoring indicator data for the corresponding level.

[0067] Furthermore, following the preset funnel convergence sequence, the initial monitoring index data of the node monitoring layer are aggregated upwards to the corresponding convergence monitoring layer according to the jurisdiction relationship, and the initial monitoring index data of all convergence monitoring layers are uniformly aggregated upwards to the aggregate monitoring layer, completing the cascade verification and feature solidification of data at each level.

[0068] Furthermore, corresponding level identifiers and time dimension identifiers are added to the initial monitoring indicator data of each level after the cascading verification is completed, generating aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data respectively, which are then summarized to form complete hierarchical monitoring data.

[0069] It should be understood that the hierarchical funnel model enables the independent calculation and progressive convergence of monitoring data at different levels. It can simultaneously achieve overall monitoring of the global business status and precise tracking of the business status of a single node. This solves the core problems of traditional flat monitoring, which cannot distinguish the scope of anomaly impact and cannot adapt to the differences in business scale at different levels. It provides accurate hierarchical data support for subsequent hierarchical anomaly detection and graded alarms.

[0070] In one feasible implementation, step S20 may include steps S21 to S23: Step S21: Perform data layering based on the structured log data to obtain the data layering result.

[0071] It should be noted that data stratification is an operation that identifies and classifies the hierarchical affiliation of the full set of structured log data based on the organizational hierarchy metadata in the structured log data. The core is to match a single log data to the corresponding monitoring level according to the preset hierarchical division rules, so as to provide a data source with a complete classification for subsequent hierarchical monitoring.

[0072] In this embodiment, the hierarchical division rules of the data stratification are completely consistent with the three-layer architecture of the hierarchical funnel model, ensuring that the classification results can be directly adapted to the subsequent hierarchical matching process.

[0073] In this embodiment, the hierarchical funnel model is a three-layer monitoring and computing architecture that is cascaded from top to bottom. It includes an aggregation monitoring layer, a convergence monitoring layer, and a node monitoring layer connected in sequence. Each layer corresponds one-to-one with the administrative jurisdiction topology of the target monitoring object. The core is to realize the hierarchical independent calculation and step-by-step convergence of monitoring data.

[0074] Understandably, the process involves first parsing the organizational hierarchy metadata in the structured log data one by one, extracting the organizational jurisdiction information corresponding to each log data, then matching the extracted jurisdiction information with the three-layer architecture of the hierarchical funnel model to complete the hierarchical labeling of a single log data, and finally classifying and aggregating the entire log data according to the hierarchical labels to obtain the data stratification results.

[0075] It should be understood that by accurately stratifying the structured log data, the hierarchical classification and aggregation of the entire log data is achieved, providing accurate data source support for subsequent differentiated hierarchical monitoring calculations, and ensuring the hierarchical correspondence and calculation accuracy of the hierarchical monitoring from the data source.

[0076] Step S22: Match the target data processing layer in the aggregation monitoring layer, convergence monitoring layer, and node monitoring layer according to the data layering results.

[0077] It should be noted that the target data processing layer is a monitoring and computing layer in the hierarchical funnel model that perfectly matches the hierarchical affiliation of a certain type of log data subset in the data hierarchical results. It is the dedicated execution unit for subsequent aggregation calculations of this type of log data, which can ensure that the log data completes accurate monitoring and calculations at the corresponding level.

[0078] In this embodiment, the target data processing layer is the corresponding level among the aggregation monitoring layer, convergence monitoring layer, and node monitoring layer, which corresponds to the hierarchical affiliation of the log data.

[0079] Understandably, the process involves first extracting the hierarchical labels corresponding to each subset of log data in the data stratification results, then comparing the hierarchical labels with the three monitoring levels of the hierarchical funnel model, matching each subset of log data with a monitoring level that perfectly corresponds to the hierarchical level, and finally determining that level as the target data processing layer for the corresponding subset of log data.

[0080] It should be understood that by accurately matching the data stratification results with the monitoring level, it is ensured that log data at different levels can enter the corresponding computing units to perform differentiated monitoring calculations, thus guaranteeing the implementation of the hierarchical monitoring logic of the stratified funnel model from a process perspective.

[0081] Step S23: Perform layered monitoring on the structured log data according to the target data processing layer to obtain layered monitoring data.

[0082] Understandably, by configuring differentiated time sliding windows and indicator aggregation strategies for each matched target data processing layer, parallel aggregation calculations are performed on the corresponding structured log data within each target data processing layer to generate initial monitoring indicator data for the corresponding level. The data at each level is then aggregated and verified step by step according to the preset funnel convergence rules to obtain hierarchical monitoring data.

[0083] It should be understood that by using differentiated monitoring and calculation at the target data processing layer, independent monitoring and overall coordination of the operational status of different levels of business are achieved, solving the problem that traditional flat monitoring cannot adapt to the characteristics of different levels of business, and providing accurate hierarchical indicator data for subsequent anomaly detection.

[0084] In one feasible implementation, step S23 may include steps S231 to S233: Step S231: Determine the time sliding window and indicator aggregation strategy based on the target data processing layer.

[0085] It should be noted that the time sliding window is a rule for defining the time interval for segmented aggregation calculation of real-time data streams in stream computing scenarios. It can continuously update the calculation range at fixed time intervals to ensure the continuity and timeliness of real-time data stream aggregation calculation.

[0086] In this embodiment, different levels of monitoring scenarios will be matched with time sliding windows of different durations to adapt to different levels of anomaly identification needs.

[0087] In addition, the indicator aggregation strategy is a set of statistical calculation rules for structured log data. It clarifies the core indicator dimensions, calculation logic and statistical scope that need to be counted, and is the core basis for transforming scattered log data into standardized indicators that can be used for monitoring.

[0088] In this embodiment, the indicator aggregation strategy will be configured differently according to the business characteristics of the corresponding monitoring level to ensure the relevance of the aggregation results.

[0089] Understandably, the process involves first identifying the layer type corresponding to the target data processing layer, then matching a time sliding window of the corresponding duration based on the business monitoring requirements corresponding to the layer type, and simultaneously matching the indicator aggregation strategy with the corresponding statistical range and calculation logic to complete the determination of the time sliding window and indicator aggregation strategy.

[0090] It should be understood that by matching differentiated time sliding windows and indicator aggregation strategies to different target data processing layers, the limitations of traditional unified monitoring calculation rules are broken. This can simultaneously meet the dual needs of rapid detection of major global faults and accurate identification of minor local anomalies, thereby improving the adaptability and accuracy of monitoring calculations.

[0091] Step S232: Based on the time sliding window and the indicator aggregation strategy, perform hierarchical monitoring on the structured log data to generate initial monitoring indicator data.

[0092] It should be noted that the initial monitoring metric data is a preliminary monitoring statistical data generated by aggregating and calculating structured log data based on the corresponding time sliding window and metric aggregation strategy. It fully records the statistical results of business operation status within the corresponding level and time window, and serves as the basic data source for subsequent convergence processing.

[0093] In this embodiment, the initial monitoring metrics data includes core business monitoring metrics such as total transaction volume, number of successful transactions, number of failed transactions, transaction failure rate, and average response time.

[0094] Understandably, the calculation range of real-time data is first defined according to a certain time sliding window, and then parallel aggregation calculation is performed on the structured log data within the corresponding range according to the matching indicator aggregation strategy to obtain the values ​​of various monitoring indicators at the corresponding level and generate initial monitoring indicator data.

[0095] It should be understood that by using differentiated windows and strategies to complete hierarchical aggregation calculations, business status indicators corresponding to each monitoring level can be accurately generated, and the business operation characteristics of each level can be fully preserved, providing accurate basic data support for subsequent funnel convergence and anomaly detection.

[0096] Step S233: The initial monitoring index data is converged according to the preset funnel convergence order to obtain hierarchical monitoring data. The preset funnel convergence order is to input the initial monitoring index data into the node monitoring layer, the convergence monitoring layer and the aggregation monitoring layer in sequence for convergence.

[0097] It should be noted that the preset funnel convergence order is a pre-defined rule for the aggregation of indicator data that conforms to the top-down cascade architecture of the hierarchical funnel model. It clarifies the aggregation direction and cascade logic of the initial monitoring indicator data at each level and is the core rule for achieving funnel-style step-by-step convergence.

[0098] In this embodiment, the preset funnel convergence order follows the convergence direction from the end node to the intermediate level, and then to the top global level.

[0099] Additionally, convergence involves aggregating initial monitoring indicator data from lower-level monitoring layers upwards according to their jurisdictional affiliation to the corresponding higher-level monitoring layers, completing a secondary verification and summary statistical operation of the higher-level indicator data. It should be understood that this allows for the step-by-step aggregation of lower-level data to higher-level data while fully preserving the independent indicator data of each level.

[0100] In addition, the hierarchical monitoring data is standardized monitoring indicator data corresponding to each level of the hierarchical funnel model after convergence processing, cascading verification and feature solidification. It includes aggregated monitoring layer data, converged monitoring layer data and node monitoring layer data.

[0101] In this embodiment, the hierarchical monitoring data will fully retain the independent calculation results and hierarchical correlation of each level, which can support subsequent determination of the scope of influence and root cause analysis.

[0102] Understandably, following the preset funnel convergence order, the initial monitoring indicator data generated by the node monitoring layer is first aggregated upwards to the corresponding convergence monitoring layer according to the jurisdiction relationship, completing the summary verification of the convergence monitoring layer indicator data. Then, the initial monitoring indicator data of all convergence monitoring layers are uniformly aggregated upwards to the aggregation monitoring layer, completing the summary verification of the aggregation monitoring layer indicator data. Finally, the corresponding level identifier is attached to the verified indicator data of each level to obtain the hierarchical monitoring data.

[0103] It should be understood that by completing the hierarchical aggregation of indicator data through the preset funnel convergence sequence, it not only achieves the overall summary of the global business status, but also fully preserves the independent monitoring indicators of each level. In subsequent processes, the impact level and scope of anomalies can be directly determined through indicator data without the need for manual drill-down analysis, which greatly improves the efficiency of fault emergency response.

[0104] In one feasible implementation, step S231 may include: When the target data processing layer is an aggregation monitoring layer, the time sliding window is determined to be a global time sliding window, and the indicator aggregation strategy is determined to be a full-dimensional aggregation strategy.

[0105] When the target data processing layer is a convergence monitoring layer, the time sliding window is determined to be a regional time sliding window, and the indicator aggregation strategy is determined to be a regional dimension aggregation strategy.

[0106] When the target data processing layer is a node monitoring layer, the time sliding window is determined to be a terminal-level time sliding window, and the indicator aggregation strategy is determined to be a single business node dimension aggregation strategy.

[0107] It should be noted that the global time sliding window is the shortest time sliding window set to adapt to the top-level full business monitoring requirements, and is used to perform high-frequency real-time aggregation calculations on all business data within the full scope of management.

[0108] In this embodiment, the duration of the global time sliding window can be set to 1 minute to meet the core requirements of bank-wide business monitoring.

[0109] In addition, the full-dimensional aggregation strategy is a comprehensive statistical calculation rule that adapts to the top-level global monitoring scenario settings, covering the aggregation statistical dimensions of all governing agencies, all business scenarios, and all interface types.

[0110] In this embodiment, the statistical scope of the full-dimensional aggregation strategy covers all counter service data of all branches and outlets of the bank.

[0111] Additionally, the regional-level time sliding window is a medium-duration time sliding window adapted to the business monitoring needs of intermediate-level regions, used to perform real-time aggregation calculations on business data within the corresponding jurisdiction at a fixed frequency.

[0112] In this embodiment, the duration of the regional time sliding window can be set to 5 minutes to meet the core requirements of branch-level business monitoring.

[0113] In addition, the regional dimension aggregation strategy is a regional scope statistical calculation rule adapted to the intermediate-level regional monitoring scenario settings, covering all subordinate institutions within the corresponding jurisdiction, and the aggregation statistical dimension of all business scenarios and interface types within the corresponding region.

[0114] In this embodiment, the statistical scope of the regional dimension aggregation strategy covers the counter service data of all branches within the jurisdiction of the corresponding branch.

[0115] Additionally, the terminal-level time sliding window is a medium-length time sliding window adapted to the business monitoring needs of end nodes, used to perform real-time aggregation calculations on the business data of a single business node at a fixed frequency.

[0116] In this embodiment, the duration of the terminal-level time sliding window is set to 5 minutes to meet the core requirements of branch-level business monitoring.

[0117] In addition, the single-business-node-dimensional aggregation strategy is a single-node statistical calculation rule adapted to the end-node monitoring scenario, which only covers the aggregation statistical dimensions of the full range of business scenarios and interface types of a single business node.

[0118] In this embodiment, the statistical scope of the single-service node dimension aggregation strategy covers the full range of counter service data for the corresponding single outlet.

[0119] Understandably, the first step is to identify the layer type corresponding to the target data processing layer, clarify the monitoring scope and core requirements of that layer, and then match the corresponding time sliding window and indicator aggregation strategy according to the layer type. When the target data processing layer is an aggregation monitoring layer, a global time sliding window and a full-dimensional aggregation strategy are determined; when it is a convergence monitoring layer, a regional time sliding window and a regional dimension aggregation strategy are determined; and when it is a node monitoring layer, a terminal-level time sliding window and a single business node dimension aggregation strategy are determined.

[0120] It should be understood that by matching differentiated time sliding windows and indicator aggregation strategies to different levels, the limitations of the traditional one-size-fits-all fixed rules in monitoring are broken. This not only ensures the rapid detection of major global faults, but also achieves the accurate identification of minor local anomalies. It fundamentally solves the persistent problem of false alarms and missed alarms in traditional monitoring, and greatly improves the adaptability and accuracy of monitoring.

[0121] In one feasible implementation, in the scenario of monitoring bank counter channels, Flink can be used to read bank counter business log data that has been processed into a structured form, parse the pre-injected institutional hierarchy metadata in the structured log data one by one, identify the branch institution, superior branch and head office attribution information corresponding to each log, and complete the hierarchical attribution matching and automatic diversion of the entire log data.

[0122] Furthermore, the log data of the corresponding branch is distributed to the node monitoring layer, the log data of the corresponding branch jurisdiction is distributed to the convergence monitoring layer, and the log data of the corresponding bank-wide scope is distributed to the aggregation monitoring layer. Then, differentiated time sliding windows and indicator aggregation granularity are matched for the three monitoring layers. The aggregation monitoring layer is matched with the shortest time sliding window adapted to the rapid discovery of major faults at the bank-wide level and the full-institution, full-dimensional aggregation rules. The convergence monitoring layer is matched with the time sliding window adapted to the identification of regional anomalies at the branch level and the corresponding jurisdictional dimension aggregation rules. The node monitoring layer is matched with the time sliding window adapted to the accurate identification of single-point anomalies at the branch level and the single-branch dimension aggregation rules. Based on the matched time sliding windows and aggregation rules...

[0123] Furthermore, parallel aggregation calculations are performed on the diverted log data within each level to generate initial monitoring indicator data such as total transaction volume, transaction failure rate, and average response time for the corresponding level. Then, following the top-down funnel convergence logic, the initial monitoring indicator data of all node monitoring layers (i.e., branch level) are aggregated upwards to the corresponding convergence monitoring layer (i.e., branch level) according to their jurisdictional affiliation. The initial monitoring indicator data of all convergence monitoring layers (i.e., branch level) are then uniformly aggregated upwards to the aggregation monitoring layer (i.e., head office level). This completes the cascading verification and feature solidification of data at each level. The initial monitoring indicator data of each level that has completed verification is attached with the corresponding level identifier, unique institutional code, and time window start and end stamps, generating aggregation monitoring layer data corresponding to the entire bank, convergence monitoring layer data corresponding to the branch jurisdiction, and node monitoring layer data corresponding to a single branch, which are then summarized to form the hierarchical monitoring data of the bank counter channel.

[0124] Step S30: Perform anomaly detection based on the hierarchical monitoring data to obtain anomaly detection results.

[0125] It should be noted that anomaly detection is a compliance verification operation performed on the hierarchical monitoring data to identify abnormal business states that deviate from the normal operating range by comparing the monitoring indicator data of each level with the detection benchmark of the corresponding level.

[0126] In addition, the anomaly detection result is the verification result data output after the anomaly detection process, which contains core information such as the anomaly level, anomaly indicator type, and anomaly deviation degree. It is the core basis for generating hierarchical alarm information.

[0127] Understandably, the process involves first parsing the hierarchical monitoring data to determine its corresponding level, then matching the pre-defined anomaly detection thresholds for each level; next, comparing the monitoring data at the corresponding level with the matched anomaly detection thresholds to determine if the monitoring data deviates from the normal business operation range; and finally, when the monitoring data exceeds the limit of the corresponding anomaly detection threshold, it is determined that there is a business anomaly, and the corresponding level, anomaly indicator type, and degree of deviation are recorded simultaneously to generate a complete anomaly detection result.

[0128] It should be understood that by matching differentiated anomaly detection benchmarks to monitoring data at different levels, the anomaly detection is accurately adapted to the business characteristics of the corresponding level, solving the persistent problems of missed detection of minor anomalies and false alarms of normal fluctuations under the traditional unified threshold detection mode, and greatly improving the accuracy and adaptability of anomaly detection.

[0129] Step S40: Generate graded alarm information based on the anomaly detection results.

[0130] It should be noted that the tiered alarm information is a notification message generated based on the level and scope of impact of the anomaly in the anomaly detection results. It includes key information such as the scope of impact of the anomaly, the core content of the anomaly, and the preliminary diagnostic conclusions, which can directly support the rapid activation of emergency response procedures at different levels.

[0131] Understandably, the process involves first analyzing the anomaly detection result to determine its corresponding anomaly level, then simultaneously retrieving the corresponding layered monitoring data to clarify the anomaly's impact scope and core characteristics. Next, the corresponding alarm level and alarm merging rules are determined based on the anomaly's level. When the anomaly's level is the aggregated monitoring layer, the alarm level is determined to be a global alarm, and similar anomaly alarms from the converged monitoring layer and node monitoring layer within the corresponding jurisdiction are simultaneously merged to generate hierarchical alarm information with a global impact scope identifier.

[0132] Furthermore, when the anomaly belongs to the convergence monitoring layer, the alarm level is determined to be a regional alarm, and the same type of anomaly alarms in the corresponding node monitoring layer within the jurisdiction are merged synchronously to generate hierarchical alarm information with regional impact range identifiers; when the anomaly belongs to the node monitoring layer, the alarm level is determined to be a terminal node alarm, and after verifying that there are no similar anomalies in the corresponding convergence monitoring layer and aggregation monitoring layer above, hierarchical alarm information with single node impact range identifiers is generated.

[0133] In one feasible implementation, step S40 may include steps S41 to S44: Step S41: Analyze the level to which the anomaly belongs corresponding to the anomaly detection result.

[0134] It should be noted that the level to which the anomaly belongs is the monitoring level corresponding to the anomaly detection result and the level at which the business anomaly occurred. This corresponds one-to-one with the three-layer architecture of the hierarchical funnel model, which includes three types: aggregation monitoring layer, convergence monitoring layer, and node monitoring layer.

[0135] Understandably, the core feature fields in the anomaly detection results are extracted first, and the monitoring level labels corresponding to the anomaly indicators are identified from the feature fields. The monitoring level corresponding to the anomaly is determined by matching the level labels, thus completing the parsing of the level to which the anomaly belongs.

[0136] Step S42: When the level to which the anomaly belongs is the aggregated monitoring layer, generate hierarchical alarm information based on the global impact scope identifier.

[0137] It should be noted that the global impact range identifier is a feature identifier used to mark the scope of an abnormal event that covers the entire jurisdiction. It can clearly indicate that the abnormal event affects all subordinate regions and business nodes, and belongs to the highest level of impact range labeling.

[0138] Understandably, when an anomaly is determined to belong to the aggregated monitoring layer, a global impact scope identifier is first matched for the anomaly event, then the core anomaly information in the anomaly detection results is extracted, and the global impact scope identifier and the core anomaly information are merged to generate a hierarchical alarm information corresponding to the highest alarm level.

[0139] Step S43: When the level to which the anomaly belongs is the convergence monitoring layer, generate hierarchical alarm information based on the regional impact range identifier.

[0140] It should be noted that the regional impact range identifier is a feature identifier used to mark the abnormal event covering a specific jurisdictional area. It can be clearly stated that the abnormal event only affects the subordinate business nodes within the corresponding area and will not affect the entire jurisdictional area.

[0141] Understandably, when an anomaly is determined to belong to the convergence monitoring layer, the regional impact range identifier of the corresponding jurisdiction is first matched for the anomaly event, and then the core anomaly information in the anomaly detection results is extracted. The regional impact range identifier and the core anomaly information are then merged to generate hierarchical alarm information corresponding to the intermediate alarm level.

[0142] It should be understood that alarm information with regional impact range identifiers for anomalies generated at the convergence monitoring layer can accurately locate anomalies as branch-level regional events, avoiding misjudging regional anomalies as global events. At the same time, it can directly identify the affected branch range, improving the accuracy and efficiency of regional anomaly handling.

[0143] Step S44: When the level to which the anomaly belongs is the node monitoring layer, generate hierarchical alarm information based on the single node impact range identifier.

[0144] It should be noted that the single-node impact range identifier is a feature identifier used to mark that an abnormal event only covers a single business node. It can be clearly stated that the abnormal event is a single-point occasional event and will not affect the superior jurisdiction area or other business nodes.

[0145] Understandably, when the level of an anomaly is determined to be the node monitoring layer, the single-node impact range identifier of the corresponding business node is first matched for the anomaly event, and then the core anomaly information in the anomaly detection results is extracted. The single-node impact range identifier and the core anomaly information are then merged to generate hierarchical alarm information corresponding to the basic alarm level.

[0146] It should be understood that generating alarm information with a single node impact range identifier for anomalies at the node monitoring layer can accurately locate anomalies as single-point events at the network level, enabling precise identification and individual alarms for minor anomalies, preventing single-point anomaly alarms from being submerged in a massive amount of alarm information, and improving the efficiency of handling and responding to single-point anomalies.

[0147] In one feasible implementation, in the bank counter channel monitoring scenario, the anomaly detection result can be read first, the anomaly level corresponding to the result can be analyzed, and the hierarchical monitoring data of the corresponding level can be retrieved simultaneously to clarify the core monitoring indicators, occurrence time and business scenario corresponding to the anomaly.

[0148] Furthermore, when the anomaly is determined to belong to the aggregated monitoring layer (i.e., the entire bank level), it is identified as a global major anomaly. Simultaneously, a preset top-down alarm merging strategy is executed, automatically severing all similar alarm outputs from the converged monitoring layer (i.e., the branch level) and the node monitoring layer (i.e., the network point level) within the jurisdiction of the anomaly, marking them as suppressed. At the same time, a global impact scope identifier is matched for the anomaly, clarifying the impact scope of the anomaly covering all institutions in the entire bank. Combining the core information of the anomaly with the preliminary root cause diagnosis conclusion, a hierarchical alarm information with the highest alarm level and the label of the entire bank's impact scope is generated.

[0149] Additionally, when the anomaly is determined to belong to the convergence monitoring layer, i.e., the branch level, it is identified as a regional anomaly. The corresponding alarm merging strategy is executed synchronously, automatically suspending the output of similar alarms from all nodes at the monitoring layer (i.e., the branch level) within the branch's jurisdiction. The regional impact range identifier of the corresponding jurisdiction is matched for the anomaly, and the specific branch and subordinate branch list covered by the anomaly is clarified. Based on the core information of the anomaly, hierarchical alarm information with the corresponding intermediate alarm level and the regional impact range label of the branch level is generated.

[0150] Additionally, when the anomaly is determined to belong to the node monitoring layer (i.e., the branch level), it is first verified that there are no similar anomaly detection results in the upper-level convergence monitoring layer (i.e., the branch) and the aggregation monitoring layer (i.e., the entire line). After confirming that the anomaly is an occasional independent anomaly in a single branch, the single-node impact range identifier of the corresponding single branch is matched for the anomaly, clarifying that the anomaly only covers the single branch. Based on the core information of the anomaly, a hierarchical alarm information with the branch-level single-node impact range label is generated at the corresponding basic alarm level.

[0151] This embodiment provides an anomaly detection and alarm method based on a hierarchical funnel model. By enhancing and standardizing the business context metadata of the initial log data of counter business, and combining it with a three-level hierarchical funnel model of the whole bank, branches, and outlets adapted to the bank's administrative architecture, it performs hierarchical differentiated indicator aggregation calculation, and is equipped with a hierarchical adapted anomaly detection and cascading alarm merging mechanism. This solves the core problems of traditional counter channel monitoring, such as false alarms and missed alarms due to static thresholds, inability to automatically determine the scope of fault impact, and inability of monitoring configuration to adapt to dynamic changes in architecture. It significantly reduces alarm noise, shortens the fault emergency response cycle, and enables rapid discovery of global major faults and accurate identification of local minor anomalies.

[0152] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The anomaly detection and alarm method based on the hierarchical funnel model includes steps S31 to S33 in step S30: Step S31: parse the level to which the anomaly belongs corresponding to the hierarchical monitoring data.

[0153] It should be noted that the level to which the exception belongs is the monitoring level in the hierarchical funnel model corresponding to the business exception event, which corresponds one-to-one with the three-layer architecture of the hierarchical funnel model, including three types: aggregation monitoring layer, convergence monitoring layer, and node monitoring layer.

[0154] Understandably, the process involves first extracting the hierarchical identifiers and organizational affiliation information from the hierarchical monitoring data, then matching and verifying the extracted identifiers with the three-layer monitoring architecture of the hierarchical funnel model to accurately determine the monitoring level corresponding to the data and complete the parsing of the level to which the anomaly belongs.

[0155] Step S32: Load the target dynamic threshold according to the level to which the anomaly belongs.

[0156] It should be noted that the target dynamic threshold is an adaptive anomaly detection benchmark value that precisely matches the level to which the anomaly belongs. It is a dynamic fluctuation benchmark generated based on historical business operation data of the corresponding level, rather than a fixed static value. This threshold can be adapted to the business scale, transaction period, and business load characteristics of the corresponding level, conforming to the business operation rules of different levels.

[0157] In addition, the dynamic baseline is a normal business fluctuation range that is independently modeled and generated for the business characteristics of different levels and institutions. It can effectively adapt to the business heterogeneity and business volume tidal effect of different institutions and avoid false alarms and false negatives caused by fixed thresholds.

[0158] Understandably, the process begins by matching the threshold model and dynamic baseline of the corresponding level based on the parsed anomaly level, retrieving the adaptive detection benchmark value corresponding to that level from the time series database, and determining it as the target dynamic threshold used for this anomaly detection, thus completing the threshold loading.

[0159] It should be understood that by loading matching target dynamic thresholds for different anomaly levels, the anomaly detection benchmark and the corresponding business characteristics of the level are accurately adapted, solving the core problem that the traditional static threshold one-size-fits-all strategy cannot adapt to the differences in business at different levels.

[0160] Step S33: Perform anomaly detection based on the target dynamic threshold and the hierarchical monitoring data to obtain anomaly detection results.

[0161] It should be noted that anomaly detection is a verification operation that compares and verifies hierarchical monitoring data with target dynamic thresholds to identify whether the business operation status deviates from the normal fluctuation range.

[0162] Understandably, the values ​​of each monitoring indicator in the hierarchical monitoring data of the corresponding level are first compared with the normal fluctuation range defined by the target dynamic threshold to determine whether the indicator exceeds the normal range. When the indicator exceeds the threshold limit, it is determined that there is an anomaly, and the full feature information of the anomaly is recorded simultaneously to generate an anomaly detection result.

[0163] It should be understood that by matching the target dynamic threshold of the corresponding level to complete the hierarchical anomaly detection, it can effectively adapt to the business operation characteristics of different levels, which not only avoids the missed detection of minor anomalies in small branches, but also solves the problem of false alarms of normal business fluctuations in large branches, and significantly improves the accuracy of anomaly detection.

[0164] This embodiment provides an anomaly detection and alarm method based on a hierarchical funnel model. By parsing the hierarchical monitoring data to which the anomaly belongs, it matches and loads appropriate target dynamic thresholds for different monitoring levels. Based on the hierarchical differentiated adaptive dynamic baseline thresholds, it completes hierarchical anomaly detection. This solves the core problem that the traditional static threshold one-size-fits-all strategy cannot adapt to the business characteristics and business volume tidal effects of different levels, thus leading to false alarms and missed alarms in anomaly detection. It accurately fits the business operation rules of each level, takes into account the identification of minor anomalies and the filtering of normal business fluctuations, and significantly improves the accuracy of anomaly detection.

[0165] For example, to help understand the implementation process of the anomaly detection and alarm method based on the hierarchical funnel model obtained by combining this embodiment with the above embodiment one, please refer to... Figure 4 , Figure 4 A simplified flowchart of an anomaly detection and alarm method based on a hierarchical funnel model is provided, specifically: The log collection SDK is responsible for uniformly collecting all business logs and API call data generated from various business scenarios, realizing the full injection and collection of counter business context metadata. The client's log collection SDK interfaces with the OAM cloud cluster. The OAM cloud cluster contains multiple scalable service modules such as service1, service2, and service3. Each service module receives corresponding business data reported by the log collection SDK, providing stable backend service support for front-end business scenarios and also providing a complete source of business metadata for log collection. After completing log data collection, the log collection SDK uniformly reports all log data to the KAFKA message queue. The KAFKA message queue, as the system's high-throughput data buffer layer, receives the raw log stream reported by the client, achieving peak smoothing and orderly distribution of log data, providing stable data stream support for subsequent parallel data processing. The log stream output from the KAFKA message queue is divided into two parallel processing paths. The first path is the monitoring metric aggregation path. The log stream first enters the Flink metric aggregation module. Based on the core requirements of the hierarchical funnel monitoring model, this module performs real-time metric aggregation calculations on the log data according to the three-level organizational dimensions (entire bank, branch, and network) and the business scenario dimension. After aggregation, the multi-dimensional monitoring metric data is written to the corresponding database for persistent storage. The real-time metric data stored in the database is synchronously pushed to the monitoring and alarm service, providing core metric data support for the monitoring and alarm service and driving anomaly threshold determination and graded alarm triggering. The second path is the raw log processing path. The log stream synchronously enters the Flink cleaning module. This module performs real-time data cleaning, format standardization, and outlier filtering on the raw logs. The structured raw log data after processing is written to the Elasticsearch database for full retention. The full raw log data stored in the Elasticsearch database is synchronously provided to the monitoring and alarm service for root cause analysis and business impact analysis after alarm triggering, providing complete raw data support for accurate fault location and graded determination of impact scope. The overall architecture enables the collection of client-side business logs, cloud message queue buffering, and dual-link parallel stream computing processing to complete monitoring and alarm triggering and deep root cause analysis.

[0166] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the anomaly detection and alarm method based on the hierarchical funnel model of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0167] This application also provides an anomaly detection and alarm device based on a hierarchical funnel model; please refer to [reference needed]. Figure 5 The anomaly detection and alarm device based on the hierarchical funnel model includes: Data cleaning module 10 is used to process the initial log data to obtain structured log data; The hierarchical monitoring module 20 is used to perform hierarchical monitoring based on the structured log data and the hierarchical funnel model to obtain hierarchical monitoring data, wherein the hierarchical monitoring data includes aggregated monitoring layer data, convergent monitoring layer data and node monitoring layer data. Anomaly detection module 30 is used to perform anomaly detection based on the hierarchical monitoring data and obtain anomaly detection results; The anomaly alarm module 40 is used to generate hierarchical alarm information based on the anomaly detection results.

[0168] The anomaly detection and alarm device based on a hierarchical funnel model provided in this application, employing the anomaly detection and alarm method based on a hierarchical funnel model in the above embodiments, can solve the technical problem of how to accurately identify and warn of abnormal states during the operation of bank counter channels, taking into account different institutional levels and business differences. Compared with the prior art, the beneficial effects of the anomaly detection and alarm device based on a hierarchical funnel model provided in this application are the same as those of the anomaly detection and alarm method based on a hierarchical funnel model provided in the above embodiments, and other technical features in the anomaly detection and alarm device based on a hierarchical funnel model are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0169] In one embodiment, the hierarchical funnel model includes an aggregation monitoring layer, a convergence monitoring layer, and a node monitoring layer connected in sequence; the hierarchical monitoring module 20 is further configured to perform data hierarchical analysis based on the structured log data to obtain data hierarchical results; Based on the data layering results, the target data processing layer is matched in the aggregation monitoring layer, convergence monitoring layer, and node monitoring layer. The structured log data is monitored in layers according to the target data processing layer to obtain layered monitoring data.

[0170] In one embodiment, the hierarchical monitoring module 20 is further configured to determine a time sliding window and an indicator aggregation strategy based on the target data processing layer; Based on the time sliding window and the indicator aggregation strategy, the structured log data is monitored in layers to generate initial monitoring indicator data; The initial monitoring index data is converged according to the preset funnel convergence order to obtain hierarchical monitoring data. The preset funnel convergence order is to input the initial monitoring index data into the node monitoring layer, the convergence monitoring layer, and the aggregation monitoring layer in sequence for convergence.

[0171] In one embodiment, the hierarchical monitoring module 20 is further configured to determine that the time sliding window is a global time sliding window and the indicator aggregation strategy is a full-dimensional aggregation strategy when the target data processing layer is an aggregation monitoring layer; When the target data processing layer is a convergence monitoring layer, the time sliding window is determined to be a regional time sliding window, and the indicator aggregation strategy is determined to be a regional dimension aggregation strategy. When the target data processing layer is a node monitoring layer, the time sliding window is determined to be a terminal-level time sliding window, and the indicator aggregation strategy is determined to be a single business node dimension aggregation strategy.

[0172] In one embodiment, the hierarchical monitoring module 20 is further configured to parse the level to which the anomaly belongs corresponding to the anomaly detection result; When the anomaly belongs to the aggregated monitoring layer, hierarchical alarm information is generated based on the global impact scope identifier; When the anomaly belongs to the convergence monitoring layer, hierarchical alarm information is generated based on the regional impact range identifier. When the level to which the anomaly belongs is the node monitoring layer, hierarchical alarm information is generated based on the single node's impact range identifier.

[0173] In one embodiment, the anomaly detection module 30 is further configured to parse the anomaly level corresponding to the hierarchical monitoring data; Load the target dynamic threshold according to the level to which the anomaly belongs; Anomaly detection is performed based on the target dynamic threshold and the hierarchical monitoring data to obtain anomaly detection results.

[0174] In one embodiment, the data cleaning module 10 is further configured to acquire context metadata; The initial log data is enhanced by performing business context enhancement processing based on the context metadata to obtain enhanced log data; The enhanced log data is cleaned to obtain cleaned log data; The cleaning log data is processed into structured log data.

[0175] This application provides an anomaly detection and alarm device based on a hierarchical funnel model. The anomaly detection and alarm device based on a hierarchical funnel model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the anomaly detection and alarm method based on the hierarchical funnel model in the above embodiment 1.

[0176] The following is for reference. Figure 6This document illustrates a structural schematic diagram of an anomaly detection and alarm device based on a hierarchical funnel model suitable for implementing embodiments of this application. The anomaly detection and alarm device based on a hierarchical funnel model in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The anomaly detection and alarm device based on the hierarchical funnel model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0177] like Figure 6 As shown, the anomaly detection and alarm device based on the hierarchical funnel model may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the anomaly detection and alarm device based on the hierarchical funnel model. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the hierarchical funnel model-based anomaly detection and alarm device to exchange data wirelessly or via wired communication with other devices. Although the figure shows hierarchical funnel model-based anomaly detection and alarm devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems can be implemented alternatively.

[0178] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0179] The anomaly detection and alarm device based on a hierarchical funnel model provided in this application, employing the anomaly detection and alarm method based on a hierarchical funnel model in the above embodiments, can solve the technical problem of how to accurately identify and warn of abnormal states during the operation of bank counter channels, taking into account different institutional levels and business differences. Compared with the prior art, the beneficial effects of the anomaly detection and alarm device based on a hierarchical funnel model provided in this application are the same as those of the anomaly detection and alarm method based on a hierarchical funnel model provided in the above embodiments, and other technical features in this anomaly detection and alarm device based on a hierarchical funnel model are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0180] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0181] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0182] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the anomaly detection and alarm method based on the hierarchical funnel model in the above embodiments.

[0183] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), Erasable Programmable Read Only Memory (EPROM), optical fiber, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0184] The aforementioned computer-readable storage medium may be included in an anomaly detection and alarm device based on a hierarchical funnel model; or it may exist independently and not be assembled into an anomaly detection and alarm device based on a hierarchical funnel model.

[0185] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the anomaly detection and alarm device based on a hierarchical funnel model, the device performs the following actions: processes initial log data to obtain structured log data; performs hierarchical monitoring based on the structured log data and the hierarchical funnel model to obtain hierarchical monitoring data, wherein the hierarchical monitoring data includes aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data; performs anomaly detection based on the hierarchical monitoring data to obtain anomaly detection results; and generates hierarchical alarm information based on the anomaly detection results.

[0186] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0188] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0189] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described anomaly detection and alarm method based on a hierarchical funnel model. This solves the technical problem of accurately identifying and issuing early warnings of abnormal states during the operation of bank counter channels, taking into account different institutional levels and business differences. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the anomaly detection and alarm method based on a hierarchical funnel model provided in the above embodiments, and will not be repeated here.

[0190] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described anomaly detection and alarm method based on a hierarchical funnel model.

[0191] The computer program product provided in this application can solve the technical problem of accurately identifying and issuing early warnings of abnormal states during the operation of bank counter channels, taking into account different institutional levels and business differences. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the anomaly detection and alarm method based on the hierarchical funnel model provided in the above embodiments, and will not be repeated here.

[0192] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An anomaly detection and alarm method based on a hierarchical funnel model, characterized in that, The method includes: The initial log data is processed to obtain structured log data; Based on the structured log data and the hierarchical funnel model, hierarchical monitoring is performed to obtain hierarchical monitoring data, which includes aggregated monitoring layer data, convergent monitoring layer data, and node monitoring layer data. Anomaly detection is performed based on the hierarchical monitoring data to obtain anomaly detection results; Based on the anomaly detection results, a tiered alarm information is generated.

2. The method as described in claim 1, characterized in that, The hierarchical funnel model consists of an aggregation monitoring layer, a convergence monitoring layer, and a node monitoring layer connected in sequence. The step of performing hierarchical monitoring based on the structured log data and the hierarchical funnel model to obtain hierarchical monitoring data includes: Data stratification is performed based on the structured log data to obtain the data stratification results; Based on the data layering results, the target data processing layer is matched in the aggregation monitoring layer, convergence monitoring layer, and node monitoring layer. The structured log data is monitored in layers according to the target data processing layer to obtain layered monitoring data.

3. The method as described in claim 2, characterized in that, The step of performing layered monitoring on the structured log data according to the target data processing layer to obtain layered monitoring data includes: The time sliding window and indicator aggregation strategy are determined based on the target data processing layer; Based on the time sliding window and the indicator aggregation strategy, the structured log data is monitored in layers to generate initial monitoring indicator data; The initial monitoring index data is converged according to the preset funnel convergence order to obtain hierarchical monitoring data. The preset funnel convergence order is to input the initial monitoring index data into the node monitoring layer, the convergence monitoring layer, and the aggregation monitoring layer in sequence for convergence.

4. The method as described in claim 3, characterized in that, The step of determining the time sliding window and indicator aggregation strategy based on the target data processing layer includes: When the target data processing layer is an aggregation monitoring layer, the time sliding window is determined to be a global time sliding window, and the indicator aggregation strategy is determined to be a full-dimensional aggregation strategy. When the target data processing layer is a convergence monitoring layer, the time sliding window is determined to be a regional time sliding window, and the indicator aggregation strategy is determined to be a regional dimension aggregation strategy. When the target data processing layer is a node monitoring layer, the time sliding window is determined to be a terminal-level time sliding window, and the indicator aggregation strategy is determined to be a single business node dimension aggregation strategy.

5. The method as described in claim 1, characterized in that, The step of generating graded alarm information based on the anomaly detection results includes: Analyze the anomaly detection result to determine the anomaly level; When the anomaly belongs to the aggregated monitoring layer, hierarchical alarm information is generated based on the global impact scope identifier; When the anomaly belongs to the convergence monitoring layer, hierarchical alarm information is generated based on the regional impact range identifier. When the level to which the anomaly belongs is the node monitoring layer, hierarchical alarm information is generated based on the single node's impact range identifier.

6. The method as described in claim 1, characterized in that, The step of performing anomaly detection based on the hierarchical monitoring data and obtaining the anomaly detection result includes: Analyze the anomaly level corresponding to the hierarchical monitoring data; Load the target dynamic threshold according to the level to which the anomaly belongs; Anomaly detection is performed based on the target dynamic threshold and the hierarchical monitoring data to obtain anomaly detection results.

7. The method as described in claim 1, characterized in that, The steps of processing the initial log data to obtain structured log data include: Retrieve context metadata; The initial log data is enhanced by performing business context enhancement processing based on the context metadata to obtain enhanced log data; The enhanced log data is cleaned to obtain cleaned log data; The cleaning log data is processed into structured log data.

8. An anomaly detection and alarm device based on a hierarchical funnel model, characterized in that, The device includes: The data cleaning module is used to process the initial log data to obtain structured log data; The hierarchical monitoring module is used to perform hierarchical monitoring based on the structured log data and the hierarchical funnel model to obtain hierarchical monitoring data, wherein the hierarchical monitoring data includes aggregated monitoring layer data, convergent monitoring layer data and node monitoring layer data. Anomaly detection module is used to perform anomaly detection based on the hierarchical monitoring data and obtain anomaly detection results; The anomaly alarm module is used to generate tiered alarm information based on the anomaly detection results.

9. An anomaly detection and alarm device based on a hierarchical funnel model, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the anomaly detection and alarm method based on the hierarchical funnel model as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the anomaly detection and alarm method based on the hierarchical funnel model as described in any one of claims 1 to 7.

Images