Real-time analysis and abnormity early warning system and method for Java back-end logs

Abstract

The invention relates to the technical field of computers, in particular to a Java back-end log real-time analysis and abnormity early warning system and method.The system comprises a plurality of modules for cooperative work: a log collection module obtains and structuralizes Java application log data in real time and stores the data in a distributed system; the trend analysis module constructs a trend chart based on a time sequence; the threshold management module dynamically calculates early warning, abnormal and serious abnormal thresholds of each interface; the anomaly detection module determines an anomaly level by comparing the trend change value and the ratio with a threshold value; the feature management module extracts abnormal log feature values and compares the abnormal log feature values with a blacklist library to filter false alarms; the alarm notification module executes graded alarm according to the abnormal grade and the preset authority, and collects and feeds back to the optimization system; according to the system, through multi-dimensional trend analysis and dynamic threshold management, the false alarm rate and the missing report rate are effectively reduced, the anomaly detection accuracy is improved, and stable operation of a Java back-end application is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a Java backend log real-time analysis and anomaly warning system and method, which is suitable for real-time analysis and anomaly detection of log data generated during the operation of Java applications to provide early warning of possible system failures. Background Art

[0002] As enterprise informatization progresses, more and more business systems are developed and deployed using Java technology. These systems generate a large amount of log data during operation, which records important information such as system operation status, error messages, and business operations.

[0003] Existing log analysis systems typically use offline analysis, collecting log data for batch processing and analysis, making them unable to detect system anomalies in real time. While other systems can process logs in real time, they often use fixed thresholds for anomaly detection, making them unable to adapt to business changes and system load fluctuations, resulting in high false positive and false negative rates. Furthermore, existing systems often use a simple push method for anomaly notifications, failing to consider the severity and business importance of the anomaly. This can easily lead to notification bombardment or important anomalies going unnoticed.

[0004] Therefore, there is an urgent need for a system that can analyze Java backend logs in real time, intelligently detect anomalies and provide accurate early warnings to improve system operation and maintenance efficiency and service quality. Summary of the Invention

[0005] The purpose of the present invention is to provide a Java backend log real-time analysis and anomaly warning system and method, which can collect and analyze Java application logs in real time, accurately detect anomalies through trend analysis and dynamic threshold technology, and promptly feedback anomalies to relevant personnel through an intelligent notification mechanism.

[0006] The present invention proposes a Java backend log real-time analysis and abnormality warning system, including:

[0007] The log collection module is used to obtain log data from Java applications in real time and store the log data in a distributed storage system after structured processing;

[0008] A trend analysis module, communicatively connected to the log collection module, configured to receive structured log data sent by the log collection module and construct a trend graph based on a time series sharding strategy, wherein the trend graph includes a plurality of time point nodes, each of which includes an interface identifier, a count value, and a status mark;

[0009] A threshold management module, in communication with the trend analysis module, for dynamically calculating a warning threshold, an abnormality threshold, and a severe abnormality threshold for each interface identifier based on the interface's historical call pattern and the importance of the service;

[0010] an anomaly detection module, in communication with the trend analysis module and the threshold management module, configured to calculate the trend change value and the ratio of the time point node, and determine the anomaly level and fluctuation pattern based on the comparison result of the trend change value, the ratio and the corresponding threshold;

[0011] A feature management module, in communication with the anomaly detection module, is configured to extract feature values ​​from the anomaly log, compare the feature values ​​with feature records in a blacklist library, filter known false positive features, and dynamically update the blacklist library;

[0012] The alarm notification module is in communication with the anomaly detection module and the feature management module, and is used to determine the notification object and notification channel according to the anomaly level and the preset alarm authority configuration, perform hierarchical alarms, and collect user feedback to optimize system parameters.

[0013] Preferably, the trend analysis module includes:

[0014] The time slicing unit is used to split the continuous time series into discrete time points according to preset rules;

[0015] A node generation unit, configured to create time point nodes based on the discrete time points, and calculate a count value of each of the time point nodes;

[0016] A data smoothing unit, configured to perform interpolation processing on the sparse data in the time point node sequence to smooth abnormal fluctuations;

[0017] The trend graph construction unit is used to associate the time point nodes in chronological order to form a complete trend graph, and perform index management on the trend graph based on the interface identifier.

[0018] Preferably, the threshold management module includes:

[0019] A reference value calculation unit, configured to calculate a reference value for each interface based on the interface historical call data;

[0020] an adjustment factor generating unit, configured to generate a threshold adjustment factor according to a fluctuation coefficient, an importance factor, and a time pattern of the interface;

[0021] a threshold calculation unit, configured to calculate the warning threshold, the abnormality threshold, and the severe abnormality threshold by combining the reference value and the adjustment factor;

[0022] The threshold evaluation unit is used to regularly evaluate the effectiveness of the threshold and optimize the threshold parameters based on anomaly detection results and user feedback.

[0023] Preferably, the anomaly detection module includes:

[0024] The trend calculation unit is used to calculate the difference in count values ​​between nodes at adjacent time points and generate a trend change value;

[0025] A ratio calculation unit, used to calculate the ratio of the node count value at the current time point to the warning threshold;

[0026] a progressive detection unit, configured to first determine whether the ratio exceeds a warning condition, and if so, further determine whether the trend change value exceeds an abnormal threshold or a severe abnormal threshold;

[0027] The fluctuation pattern recognition unit is used to identify abnormal patterns such as sudden increase, sudden decrease, continuous increase, continuous decrease and increased fluctuation based on the trend change values ​​of multiple consecutive time points.

[0028] Preferably, the feature management module includes:

[0029] A feature extraction unit is used to extract structured features and unstructured features from the abnormal log and generate feature values;

[0030] A similarity calculation unit, configured to calculate the similarity between the feature value and the feature record in the blacklist database;

[0031] Blacklist management unit, used to maintain the blacklist database and dynamically update blacklist features based on similarity comparison results and user feedback;

[0032] The false alarm learning unit is used to record features marked as false alarms, analyze the causes and patterns of false alarms, and optimize the blacklist strategy.

[0033] Preferably, the alarm notification module includes:

[0034] Alarm authority unit, used to determine the alarm authority level based on the abnormality level, business importance and impact scope;

[0035] A notification routing unit, configured to select notification objects and notification channels based on the alarm authority level, time factors, and preset configurations;

[0036] Notification aggregation unit, used to intelligently aggregate similar exceptions that occur within a short period of time to prevent notification bombardment;

[0037] An escalation processing unit, used to trigger the notification escalation process when there is no response to the initial notification or the problem worsens;

[0038] The feedback processing unit is used to collect user responses to notifications and process results to provide a basis for system optimization.

[0039] Preferably, the log collection module includes:

[0040] Log receiving unit, used to obtain raw log files from Java containers or applications;

[0041] A log parsing unit, configured to parse the original log file into structured data and extract interface call information, timestamps, status codes, and error information;

[0042] A data distribution unit, configured to send the structured data to a distributed message queue;

[0043] The data storage unit is used to persistently store the structured data in a distributed storage system and provide data indexing and retrieval functions.

[0044] Preferably, the adjustment factor generating unit is further configured to:

[0045] Adjust threshold parameters based on working day / non-working day identification;

[0046] Adjust threshold sensitivity based on peak / valley times throughout the day;

[0047] Adjust threshold elasticity based on seasonal fluctuations in business.

[0048] Preferably, the feedback processing unit is further configured to:

[0049] Record false positive / false negative flag data;

[0050] Statistics notification effectiveness indicators;

[0051] Generate threshold adjustment recommendations;

[0052] Update blacklist feature weights;

[0053] Optimize notification policy parameters.

[0054] A Java backend log real-time analysis and exception warning method, including:

[0055] Collect Java application log data, structure the log data, and store it in a distributed storage system;

[0056] Receive the structured log data, and construct a trend graph based on a time series sharding strategy, wherein the trend graph includes a plurality of time point nodes, each of the time point nodes includes an interface identifier, a count value, and a status mark;

[0057] Dynamically calculate the warning threshold, abnormality threshold and severe abnormality threshold for each interface identifier based on the interface historical call pattern and business importance;

[0058] Calculating the trend change value and the ratio of the time point node, and determining the abnormality level and the fluctuation pattern based on the comparison result of the trend change value, the ratio and the corresponding threshold value;

[0059] Extracting feature values ​​from the anomaly log, comparing the feature values ​​with feature records in the blacklist database, filtering known false positive features, and dynamically updating the blacklist database;

[0060] According to the abnormality level and the preset alarm authority configuration, the notification object and notification channel are determined, the graded alarm is executed, and user feedback is collected to optimize the system parameters.

[0061] The beneficial effects of the present invention include:

[0062] 1. Improve anomaly detection accuracy: Through multi-dimensional trend analysis and dynamic threshold management, the false alarm rate and missed alarm rate are significantly reduced. Experimental data shows that compared with traditional static threshold systems, the false alarm rate is reduced by about 75% and the missed alarm rate is reduced by about 60%.

[0063] 2. Early warning of system anomalies: Through real-time analysis of trend changes, abnormal signs can be discovered in advance before serious problems occur in the system. The average advance warning time is 15 minutes, buying valuable time for problem resolution.

[0064] 3. Reduce the burden of operation and maintenance: Through precise notification and a graded alarm mechanism, invalid alarms are reduced by more than 50%, 80% of general anomalies do not require manual intervention, and the average time to locate anomalies is shortened by 65%.

[0065] 4. Improve system response speed: Using a streaming processing architecture and distributed design, the average time from log generation to anomaly detection is less than 2 seconds, supporting the processing of tens of thousands of log records per second.

[0066] 5. Enhanced business insight: Through trend analysis and pattern recognition, it can predict 70% of potential system problems, provide data support for business optimization, and help identify performance bottlenecks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the system architecture of the present invention;

[0068] Figure 2 Schematic diagram of the structure of the trend analysis module of the present invention;

[0069] Figure 3 This is a schematic diagram of the structure of the threshold management module of the present invention;

[0070] Figure 4 Schematic diagram of the structure of the anomaly detection module of the present invention;

[0071] Figure 5 This is a schematic diagram of the structure of the feature management module of the present invention;

[0072] Figure 6 This is a structural diagram of the alarm notification module of the present invention;

[0073] Figure 7 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0074] Please refer to the attached Figure 1-7 The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] like Figure 1 As shown, the Java backend log real-time analysis and anomaly warning system provided by the present invention includes: a log collection module 1, a trend analysis module 2, a threshold management module 3, an anomaly detection module 4, a feature management module 5, and an alarm notification module 6. The modules are connected through communication to achieve data exchange and collaborative work.

[0076] In one embodiment of the present invention, the log collection module 1 is used to obtain log data from a Java application in real time and store the log data in a distributed storage system after structured processing. The trend analysis module 2 is in communication with the log collection module 1 and is used to receive the structured log data sent by the log collection module 1 and construct a trend chart based on the time series sharding strategy. The trend chart includes multiple time point nodes, each of which contains an interface identifier, a count value, and a status mark. The threshold management module 3 is in communication with the trend analysis module 2 and is used to dynamically calculate the warning threshold, anomaly threshold, and severe anomaly threshold for each interface identifier based on the interface's historical call pattern and the importance of the business. The anomaly detection module 4 is in communication with the trend analysis module 2 and the threshold management module 3 and is used to calculate the trend change value and ratio of the time point node and determine the anomaly level and fluctuation pattern based on the comparison results of the trend change value, the ratio, and the corresponding threshold. The feature management module 5 is in communication with the anomaly detection module 4 and is used to extract feature values ​​from the anomaly log, compare the feature values ​​with the feature records in the blacklist library, filter out known false positive features, and dynamically update the blacklist library. The alarm notification module 6 is in communication with the anomaly detection module 4 and the feature management module 5, and is used to determine the notification object and notification channel according to the anomaly level and the preset alarm authority configuration, perform hierarchical alarms, and collect user feedback to optimize system parameters.

[0077] like Figure 1As shown, the log collection module 1 includes a log receiving unit 11 , a log parsing unit 12 , a data distribution unit 13 and a data storage unit 14 .

[0078] The log receiving unit 11 is used to obtain raw log files from Java containers or applications. In a preferred embodiment of the present invention, the log receiving unit 11 supports multiple acquisition methods, including but not limited to directly reading log files output by Java applications, monitoring the standard output stream of the Java container, and obtaining logs through custom appenders in logging frameworks (such as Log4j and Logback). This support for multiple acquisition methods enables the system to adapt to different application deployment environments and can be integrated without modifying application code.

[0079] Preferably, the log receiving unit 11 adopts different access strategies for different types of Java applications. For example, for traditional monolithic applications, the log files of the application server (such as Tomcat or JBoss) can be directly read; for microservice applications based on Spring Boot, logs can be obtained by monitoring the standard output stream; and for systems using the ELK (Elasticsearch, Logstash, Kibana) stack, structured log data can be obtained through the Logstash interface.

[0080] The log parsing unit 12 is used to parse the original log file into structured data and extract interface call information, timestamps, status codes, and error information. Preferably, the log parsing unit 12 supports parsing of multiple log formats, including common log formats such as JSON format and custom delimiter formats. For logs in non-standard formats, the log parsing unit 12 uses regular expressions and pattern matching techniques for parsing. For example, for common Java exception stack information, the exception type, exception message, and key call stack information can be extracted.

[0081] In practical applications, the log parsing unit 12 needs to handle various complex situations, such as merging multi-line logs, converting timestamp formats, and standardizing log levels. For example, a typical Java exception log might contain the exception type (such as NullPointerException), the location where the exception occurred (class name, method name, line number), the exception message, and the complete call stack. The log parsing unit 12 extracts and structures this information for subsequent analysis.

[0082] Data distribution unit 13 is used to send structured data to a distributed message queue. In one embodiment of the present invention, the distributed message queue is implemented using Kafka, with an appropriate number of topics and partitions configured to support high-throughput log data processing. Data distribution unit 13 performs preliminary screening and aggregation of log data to reduce the amount of data required for subsequent processing.

[0083] Preferably, the data distribution unit 13 sets different priorities and processing strategies based on the type and importance of different logs. For example, error-level logs will be given a higher processing priority to ensure rapid identification and response to system anomalies; debug-level logs may only be processed in specific scenarios to reduce system burden.

[0084] Data storage unit 14 is used to persistently store structured data in a distributed storage system and provide data indexing and retrieval capabilities. Preferably, the distributed storage system utilizes Elasticsearch, establishing a suitable index structure to support fast retrieval based on time range, interface identifier, error type, and other dimensions. Data storage unit 14 also implements data lifecycle management, automatically clearing expired data according to configured retention policies to control storage space usage.

[0085] In practical applications, data storage unit 14 is typically configured with multiple indexes, such as creating daily indexes. This improves write performance and facilitates the management and querying of historical data. For example, transaction system logs may need to retain at least 90 days of data for auditing and analysis, while general system logs may only need to be retained for 30 days. Data storage unit 14 manages different types of logs differently based on their retention policies.

[0086] like Figure 2 As shown, the trend analysis module 2 includes a time slicing unit 21 , a node generating unit 22 , a data smoothing unit 23 and a trend graph constructing unit 24 .

[0087] The time slicing unit 21 is used to divide the continuous time series into discrete time points according to a preset rule. In a preferred embodiment of the present invention, the time slicing strategy adopts the method of time factor multiplication + time offset, and the specific formula is:

[0088] ,

[0089] Among them, TimePoint is the final generated time point identifier; TimeWeight is the time weight factor, which is usually 100; DateFactor is the date factor, which contains the numerical representation of year, month, and day, for example, 20250523 (representing May 23, 2025); TimeOffset is the time offset, which contains the numerical representation of hours and minutes, for example, 1430 (representing 14:30).

[0090] For example, for the time point 14:30 on May 23, 2025, the calculated TimePoint value is:

[0091] ,

[0092] This method of representing time points not only retains complete time information, but also facilitates comparison and sorting of time points.

[0093] Preferably, the time slicing unit 21 automatically adjusts the time granularity based on the frequency of interface calls. For high-frequency interfaces (e.g., >10 calls per second), a minute-level granularity is used; for medium-frequency interfaces (between 1 and 10 calls per second), a ten-minute granularity is used; and for low-frequency interfaces (<1 call per second), an hour-level granularity is used. This adaptive time granularity design enables the system to control data processing volume and storage space while ensuring analysis accuracy.

[0094] In practice, the time slicing unit 21 is optimized based on business characteristics. For example, during promotional periods, the order interface of an e-commerce system might require a higher time granularity (e.g., using second-level slicing) to promptly capture system pressure caused by traffic surges. During off-peak periods, the time granularity can be lowered (e.g., using hour-level slicing) to reduce data processing volume.

[0095] The node generation unit 22 is used to create time point nodes based on discrete time points and calculate the count value of each time point node.

[0096] At each time point, the node generation unit 22 aggregates all interface calls within that time point and calculates the call count of each interface. The counting method can be a simple call count or a weighted count, such as a weighted calculation based on the interface response time or error rate.

[0097] In actual applications, the node generation unit 22 adopts different counting strategies for different interfaces. For example, for the login interface, the login failure rate may be more important, so the failed requests will be given a higher weight when counting; for the data query interface, the response time may be more important, so the response time will be weighted when counting, and the longer the response time, the higher the weight.

[0098] The data smoothing unit 23 is used to interpolate the sparse data in the time point node sequence to smooth abnormal fluctuations. In a preferred embodiment of the present invention, data smoothing is performed using a moving average method combined with outlier identification. For abnormal fluctuations in the time series, the following smoothing process is applied:

[0099] ,

[0100] in, is the smoothed value at the i-th time point; is the original count value at the i-th time point; is the smoothed value at the i-1th time point; is the smoothing factor, the value range is [0,1], and it is usually 0.3.

[0101] For example, suppose the raw count values ​​of an interface at three consecutive time points are 100, 300, and 120 respectively, and the smoothing factor is Taking 0.3, the smoothed value is calculated as follows:

[0102] The first time point: (The initial value directly uses the original value);

[0103] The second time point: ;

[0104] The third time point: ;

[0105] Through this smoothing process, the abnormally high value (300) at the second time point is appropriately reduced, avoiding false alarms caused by sudden fluctuations.

[0106] In the case of sparse data (there is no data at a time point), the data smoothing unit 23 uses linear interpolation to fill in the missing values:

[0107] ,

[0108] in, is the interpolation result; is the value of the previous time point with data; is the value of the next time point with data; The time point at which interpolation is currently required; is the previous time point with data; The next time point with data.

[0109] For example, if the count value of an interface is 50 at time 1000 and 80 at time 1030, and there is no data at time 1010 and 1020, linear interpolation can be used to calculate the following:

[0110] Interpolation at time point 1010: ;

[0111] Interpolation at time point 1020: ;

[0112] This interpolation process ensures the continuity of the trend graph and facilitates subsequent trend analysis.

[0113] The trend graph construction unit 24 is used to associate the time point nodes in chronological order to form a complete trend graph, and perform index management on the trend graph based on the interface identifier.

[0114] The trend chart construction unit 24 links all time point nodes of the same interface in chronological order to form a complete trend chart. Simultaneously, to improve query efficiency, an index mapping is established from the interface code to the trend chart. Preferably, the trend chart retains historical data within a certain time range (e.g., the last seven days). Historical data outside this range is archived or cleared to control memory usage and improve processing performance.

[0115] In actual applications, the trend graph construction unit 24 constructs different types of trend graphs for different types of interfaces. For example, for the payment interface of a transaction system, trend graphs of multiple dimensions may be constructed, including a call volume trend graph, a success rate trend graph, and a response time trend graph, to comprehensively monitor the health of the interface. For the user login interface, the focus may be on the failure rate trend graph to promptly detect abnormal login behavior.

[0116] like Figure 3 As shown, the threshold management module 3 includes a reference value calculation unit 31 , an adjustment factor generation unit 32 , a threshold calculation unit 33 and a threshold evaluation unit 34 .

[0117] The reference value calculation unit 31 is used to calculate the reference value of each interface based on the interface historical call data. In a preferred embodiment of the present invention, the calculation of the reference value takes into account the historical call pattern and fluctuation characteristics of the interface and adopts a weighted average method:

[0118] ,

[0119] Among them, BaseValue is the base value of the interface; is the average call volume in the i-th historical time period; is the weight of the i-th time period, with the recent time period having a higher weight; The number of historical time periods for reference, usually 24 (for 24 hours) or 168 (for 7 days); Represents the sum operation from i=1 to n.

[0120] For example, if the average call volume of an interface in the past three time periods is 100, 120, and 90 respectively, the corresponding weights are 0.5, :

[0121] ,

[0122] To adapt to different interface characteristics, the benchmark calculation unit 31 automatically selects an appropriate historical time window and weight distribution based on the interface's call frequency and fluctuation pattern. For interfaces with large fluctuations, the reference range of historical data is increased; for stable interfaces with small fluctuations, the reference range is mainly based on recent call data.

[0123] In practical applications, the benchmark calculation unit 31 also considers time periodicity. For example, the order interface of an e-commerce system experiences significant traffic variations at different times of the day, with peak traffic times at 10:00 AM and 8:00 PM, and low traffic times at 2:00 AM. Therefore, benchmark calculations are performed using a two-dimensional matrix of hour and week, calculating independent benchmark values ​​for each hour of each workday to accommodate cyclical business fluctuations.

[0124] The adjustment factor generation unit 32 is used to generate a threshold adjustment factor based on the fluctuation coefficient, importance factor and time pattern of the interface. In one embodiment of the present invention, the threshold adjustment factor is composed of the following parts:

[0125] ,

[0126] Among them, AdjustFactor is the final adjustment factor; FluctCoef is the fluctuation coefficient, which reflects the natural fluctuation degree of the interface, and its value range is usually [0.1, 0.5]. ImportFactor is the importance factor, which reflects the business importance of the interface, and its value range is usually [1, 10]. TimePatternAdj is the time pattern adjustment coefficient, which takes into account the influence of time characteristics, and its value range is [0.8, 1.2].

[0127] For example, for a core payment interface, its fluctuation coefficient FluctCoef is 0.2, the importance factor ImportFactor is 8, and the time pattern adjustment coefficient TimePatternAdj on weekday mornings is 1.1. The adjustment factor is calculated as:

[0128] ,

[0129] This means that the threshold of this interface on weekday mornings will be approximately 10 times higher than the baseline value, reflecting the high importance of this interface and the busy business characteristics of weekday mornings.

[0130] Preferably, as described in claim 8, the adjustment factor generation unit 32 further adjusts the threshold parameters based on time characteristics. Specifically, for working days and non-working days, the system adopts different adjustment strategies: during working days, due to high business activity, the TimePatternAdj value is close to 1.2; during non-working days, due to low business activity, the TimePatternAdj value is close to 0.8. Similarly, for peak periods within a day (such as 10 am to 11 am, 3 pm to 4 pm), the TimePatternAdj value is relatively high, the threshold is increased, and false alarms are reduced; for low periods (such as 2 am to 5 am), the TimePatternAdj value is relatively low, the threshold is lowered, and sensitivity is increased. In addition, the system will also take into account seasonal fluctuations in business. For example, during promotional activities, the e-commerce system will appropriately increase the elasticity of the threshold.

[0131] In practical applications, time pattern adjustments also need to consider the impact of special periods. For example, in financial systems, centralized settlements typically occur at the end of the month and quarter, leading to a significant increase in transaction volume. In government systems, centralized business processing peaks may occur at the beginning and end of the year. Adjustment factor generation unit 32 configures adjustment coefficients for special periods to accommodate these business characteristics.

[0132] The threshold calculation unit 33 is used to calculate the warning threshold, abnormality threshold and severe abnormality threshold by combining the reference value and the adjustment factor. In one embodiment of the present invention, the three-level threshold is calculated as follows:

[0133] ,

[0134] ,

[0135] ,

[0136] Among them, WarningThreshold is the warning threshold, which is used to preliminarily determine whether attention is needed; AnomalyThreshold is the anomaly threshold, which is used to confirm anomalies; SeriousAnomalyThreshold is the severe anomaly threshold, which is used to identify serious problems that require urgent treatment.

[0137] For example, continuing with the above example, assuming that the base value BaseValue of a core payment interface is 100 and the adjustment factor AdjustFactor is 10.56, the three-level threshold calculation is:

[0138] Warning threshold: WarningThreshold = 100 × 10.56 = 1056;

[0139] Anomaly threshold: AnomalyThreshold = 100 × 10.56 × 0.8 = 844.8;

[0140] Serious anomaly threshold: SeriousAnomalyThreshold=100×10.56×1.5=1584;

[0141] It should be noted that the above coefficients (0.8 and 1.5) are empirical values ​​validated by extensive experimental data and can be adjusted appropriately for different business scenarios. For example, for financial systems with extremely high stability requirements, the coefficient of the anomaly threshold can be increased to 0.9 to make the system more sensitive; while for social media systems with high volatility, the coefficient of the severe anomaly threshold can be reduced to 1.2 to reduce excessive alerts.

[0142] In practical applications, threshold calculation also needs to consider the characteristics of different indicators. For example, for call volume indicators, the threshold is usually the upper limit, and exceeding the threshold indicates an anomaly; while for success rate indicators, the threshold is usually the lower limit, and falling below the threshold indicates an anomaly. The threshold calculation unit 33 will use different threshold calculation and judgment logic depending on the indicator type.

[0143] The threshold evaluation unit 34 is used to regularly evaluate the effectiveness of the threshold and optimize the threshold parameters based on anomaly detection results and user feedback. In a preferred embodiment of the present invention, threshold evaluation uses a combination of performance indicators and user feedback. The system records the effectiveness of each alarm (whether it is a true anomaly) and calculates the precision and recall:

[0144] ,

[0145] ,

[0146] in, This is a true example, that is, the system alarm is indeed an abnormal situation; A false positive is a situation where the system alarms but is actually normal (false alarm); It is a false negative example, that is, the system does not alarm but there is actually an abnormality (missed alarm).

[0147] For example, suppose a certain interface generates 10 alarms in a week, 8 of which are true anomalies and 2 are false alarms. There is also one true anomaly but the system does not generate an alarm. The threshold performance indicator for this interface is:

[0148] Accuracy: (80%);

[0149] Recall: (about 89%);

[0150] The threshold evaluation unit 34 adjusts the parameters used for threshold calculation based on these indicators. If the false alarm rate is high (low precision), the threshold is appropriately increased; if the false alarm rate is high (low recall), the threshold is appropriately decreased. Furthermore, user feedback is considered, and user-flagged false alarms and false alarms are analyzed to optimize threshold parameters for specific interfaces.

[0151] In practice, threshold evaluations are typically performed weekly or monthly. The system generates a threshold effectiveness report, including metrics such as precision, recall, and F1 score (the harmonic mean of precision and recall) for each interface, along with optimization recommendations. Based on this report, operations personnel can manually adjust threshold parameters for key interfaces or allow the system to automatically optimize.

[0152] like Figure 4 As shown, the anomaly detection module 4 includes a trend calculation unit 41 , a ratio calculation unit 42 , a progressive detection unit 43 and a fluctuation pattern recognition unit 44 .

[0153] The trend calculation unit 41 is used to calculate the difference in count values ​​between nodes at adjacent time points and generate a trend change value. In a preferred embodiment of the present invention, the trend change value is calculated as follows:

[0154] ,

[0155] in, is the trend change value at the i-th time point; is the count value at the i-th time point; is the count value at the i-1th time point.

[0156] For example, if the count values ​​of an interface at three consecutive time points are 100, 150, and 200, the trend change value is calculated as:

[0157] Second time point: ;

[0158] The third time point: ;

[0159] This shows that the number of calls to this interface is showing a steady growth trend, increasing by 50 at each time point.

[0160] In order to process the trend value differences caused by different time intervals, the trend calculation unit 41 normalizes the trend change value:

[0161] ,

[0162] in, is the normalized trend change value; is the time interval between the i-th time point and the i-1-th time point, in minutes; It is a standard time interval, usually set to 1 minute.

[0163] For example, if the time interval between the second time point and the first time point is 5 minutes, the normalized trend change value is:

[0164] ,

[0165] This means that the number of calls to this interface increases by 10 per minute on average. This normalized value is convenient for comparison with trend values ​​at different time intervals.

[0166] In practical applications, trend calculations must consider the characteristics and business implications of time series. For example, for interfaces with cyclical changes, comparisons may be made with historical data from the same time period, rather than with the previous point in time. For cumulative metrics (such as total transaction volume), trend change values ​​are typically positive, with anomalies manifesting as changes in growth rate. For ratio metrics (such as success rate), trend change values ​​can be positive or negative, with anomalies manifesting as significant decreases in the value.

[0167] The ratio calculation unit 42 is used to calculate the ratio of the node count value at the current time point to the warning threshold. In one embodiment of the present invention, the ratio is calculated as follows:

[0168] ,

[0169] in, is the ratio at the i-th time point; is the count value at the i-th time point; The warning threshold of the interface.

[0170] For example, if the count value of an interface at a certain time point is 1200 and the warning threshold of the interface is 1000, the ratio is calculated as:

[0171] ,

[0172] This indicates that the current count value of the interface exceeds the warning threshold of 20% and may require attention.

[0173] The ratio calculation unit 42 also calculates the changing trend of the ratio to identify the rapid growth or decline of the interface call volume:

[0174] ,

[0175] in, is the ratio change trend at the i-th time point; is the ratio at the i-th time point; is the ratio at the i-1th time point.

[0176] For example, if the ratio of an interface at two consecutive time points is 0.8 and 1.2, the ratio change trend is:

[0177] ,

[0178] This indicates that the load on the interface is increasing rapidly relative to the threshold, going from below the threshold to above the threshold in a short period of time, and may require special attention.

[0179] In practical applications, ratio calculations require different approaches for different types of indicators. For example, for upper-bound indicators (such as error rate), a higher ratio indicates a more abnormal situation; for lower-bound indicators (such as success rate), the inverse ratio is usually used, i.e. , the higher the ratio, the more abnormal it is.

[0180] The progressive detection unit 43 is used to first determine whether the ratio exceeds the warning condition. If the warning condition is exceeded, it further determines whether the trend change value exceeds the abnormal threshold or the severe abnormal threshold. This progressive detection strategy can reduce unnecessary calculations and improve system efficiency. In a preferred embodiment of the present invention, the progressive detection process is as follows:

[0181] 1. First calculate and check the ratio: If , it triggers the early warning detection; otherwise, it is marked as normal state.

[0182] 2. After the early warning detection is triggered, calculate and check the trend change value: If , then anomaly detection is triggered; otherwise, it is only marked as a warning state.

[0183] 3. After the anomaly detection is triggered, further inspection: If , it is marked as a serious exception; otherwise, it is marked as a general exception.

[0184] For example, assume that the current count value of an interface is 1200, the warning threshold is 1000, the abnormal threshold is 800, the severe abnormal threshold is 1500, and the normalized trend change value is 900. The detection process is as follows:

[0185] 1. Calculate the ratio: , triggering early warning detection.

[0186] 2. Check the trend change value: (abnormal threshold), triggering anomaly detection.

[0187] 3. Further inspection: (Severe anomaly threshold), marked as general anomaly.

[0188] This progressive detection method considers both the comparison between the current value and the threshold and the speed of trend change, and can identify anomalies more accurately.

[0189] In practice, progressive detection is tailored to different interfaces and indicators. For example, for core trading interfaces, the focus might be on trend changes. Even if the current value doesn't exceed the warning threshold, an abnormal increase in the trend change value will trigger an alert. For non-critical interfaces, the focus might be on whether the current value exceeds the warning threshold, and further analysis of trend changes will only be conducted if it clearly exceeds the threshold.

[0190] The fluctuation pattern recognition unit 44 is used to identify abnormal patterns such as sudden increase, sudden decrease, continuous increase, continuous decrease and increased fluctuation based on the trend change values ​​at multiple consecutive time points. In one embodiment of the present invention, the identification of the fluctuation pattern is based on the following rules:

[0191] Sudden increase mode: The trend change values ​​for n consecutive time points are all positive, and the trend change value at the most recent time point is significantly greater than that at the previous time points (usually greater than 3 times the standard deviation).

[0192] Sudden drop mode: The trend change values ​​for n consecutive time points are all negative, and the absolute value of the trend change value at the most recent time point is significantly greater than that of the previous time points (usually greater than 3 times the standard deviation).

[0193] Continuous rise: The trend change values ​​at n consecutive time points are all positive and show a gradually increasing trend.

[0194] Continuous decline: The trend change values ​​for n consecutive time points are all negative, and the absolute value shows a trend of gradually increasing.

[0195] Increased volatility: The standard deviation of the trend change values ​​​​at the last n time points is significantly greater than the historical standard deviation (usually greater than 2 times).

[0196] Among them, n is usually set to 3-5, which represents the number of consecutive observation time points; the standard deviation is calculated using the sample standard deviation formula:

[0197] ,

[0198] in, is the standard deviation; is the i-th sample value; is the sample mean, calculated as ; is the sample size; Represents the sum operation from i=1 to n.

[0199] For example, if the trend change values ​​of an interface at five consecutive time points are 10, 15, 20, 25, and 100, then:

[0200] Sample mean: ;

[0201] The square of the difference between each sample and the mean: ;

[0202] Sum of squared differences: ;

[0203] Standard Deviation: ;

[0204] The trend change value 100 at the most recent time point is much greater than that at the previous time points and exceeds 3 times the standard deviation ( ), and therefore may be identified as a bursting pattern.

[0205] Preferably, the fluctuation pattern recognition unit 44 optimizes pattern recognition by integrating business knowledge. For example, for a trading system, it will pay special attention to patterns of sudden increases in transaction failure rates; for a login system, it will pay special attention to patterns of sudden decreases in login success rates. Different abnormal patterns will trigger different handling strategies. For example, sudden increases or decreases generally require immediate response, while persistent increases or decreases may require further observation and analysis.

[0206] In practical applications, fluctuation pattern recognition has different focuses for different business scenarios. For example, in e-commerce systems, a sudden increase in order volume at the start of a promotion is generally normal, but a sudden increase in payment failure rates is abnormal. For content platforms, periodic fluctuations in user traffic (e.g., low on weekdays and high on weekends) are normal, but a sudden drop in content publishing may indicate a problem with the publishing system. Fluctuation pattern recognition unit 44 adjusts the recognition rules and sensitivity of different patterns based on business characteristics.

[0207] like Figure 5 As shown, the feature management module 5 includes a feature extraction unit 51 , a similarity calculation unit 52 , a blacklist management unit 53 and a false alarm learning unit 54 .

[0208] The feature extraction unit 51 is used to extract structured features and unstructured features from the abnormal log and generate feature values. In a preferred embodiment of the present invention, feature extraction supports multiple types of features:

[0209] Structural features: including error code features (such as HTTP status codes and business error codes), performance indicator features (such as response time and resource utilization), and business indicator features (such as transaction success rate and user activity).

[0210] Unstructured features: include exception stack features (such as exception type, key call stack), error message features (based on text similarity analysis), and context-related features (such as operation sequence, environmental information).

[0211] The feature extraction unit 51 uses different extraction strategies depending on the type of feature. For example, for exception stack traces, the exception type name, key method call, and line number information are extracted; for error message traces, the variable portion is removed, retaining the templated error message format. Each feature generates a feature fingerprint for subsequent comparison and management.

[0212] In practical applications, feature extraction is a key step in exception identification. For example, for the NullPointerException, a common problem in Java applications, feature extraction includes not only the exception type but also the class name, method name, and line number where the exception occurred, as well as key call stack information. These features help the system accurately identify identical or similar exceptions, reducing duplicate alerts. For business errors, such as order creation failures, feature extraction focuses on the error code, error message, and associated business parameters to distinguish between different causes of the same error.

[0213] The similarity calculation unit 52 is used to calculate the similarity between the feature value and the feature record in the blacklist library. In one embodiment of the present invention, the similarity calculation uses different algorithms for different types of features:

[0214] For structured features, exact matching or range matching is used. For example, a feature with an error code of 404 will exactly match the 404 feature in the blacklist; a feature with a response time exceeding 1000ms may match the range of features in the blacklist with a response time greater than 800ms.

[0215] For text features, Jaccard similarity is used for calculation:

[0216] ,

[0217] in, is the Jaccard similarity; and are word sets of two text features respectively; is the size of the intersection of the two sets, that is, the and the number of elements in ; is the size of the union of the two sets, which appears or The total number of elements in .

[0218] For example, suppose there are two error messages: "Failed to connect to database server at 192.168.1.10" and "Failed to connect to database server at 192.168.1.20". The sets obtained after tokenizing them are:

[0219] A={Failed,to,connect,to,database,server,at,192.168.1.10},

[0220] B={Failed,to,connect,to,database,server,at,192.168.1.20},

[0221] The intersection is {Failed,to,connect,to,database,server,at}, and its size is 7; the union is {Failed,to,connect,to,database,server,at,192.168.1.10,192.168.1.20}, and its size is 9.

[0222] Therefore, the Jaccard similarity is: .

[0223] This indicates that the two error messages are very similar and are likely the same type of database connection error.

[0224] For mixed features, weighted multi-dimensional similarity calculation is used:

[0225] ,

[0226] in, is the final mixed similarity; For the Similarity in dimensions; For the The weight of each dimension; is the number of feature dimensions; Indicates from arrive The summation operation.

[0227] For example, suppose a multi-dimensional similarity calculation is performed on an exception, including exception type similarity (weight 0.4), error message similarity (weight 0.3), and call stack similarity (weight 0.3). The similarities of each dimension are 1.0, 0.8, and 0.7, respectively. The mixed similarity calculation is:

[0228] ,

[0229] The similarity calculation unit 52 sets an appropriate similarity threshold for each feature type. When the calculated similarity exceeds the threshold, the two features are considered to match. For example, for error code features, a perfect match (similarity = 1) is usually required; for error message features, a similarity > 0.8 may be sufficient to be considered a match.

[0230] In practical applications, similarity calculations need to be optimized for different types of features. For example, for exception stack traces, the exception location (class name, method name, line number) may be more relevant than the complete call stack. For HTTP request errors, the URL pattern and status code may be more relevant than the specific values ​​of the request parameters. Similarity calculation unit 52 adjusts the similarity calculation algorithm and parameters based on the feature type and business requirements.

[0231] The blacklist management unit 53 is used to maintain the blacklist database and dynamically update the blacklist features according to the similarity comparison results and user feedback.

[0232] The blacklist management unit 53 supports blacklist initialization, dynamic updates, and periodic cleanup. During initialization, it can import preset known false positive features and historical problem features. During dynamic updates, it decides whether to add features to the blacklist based on the comparison results of new features and user feedback. During periodic cleanup, it removes low-frequency features that have not been matched for a long time to control the size of the blacklist library.

[0233] In practical applications, blacklist management is a key mechanism for improving anomaly detection accuracy. For example, for known non-problem anomalies, such as resource initialization warnings during application startup and expected errors during scheduled maintenance tasks, their characteristics can be added to the blacklist to avoid unnecessary alerts. The blacklist management unit 53 regularly analyzes blacklist usage, evaluates the effectiveness of each characteristic, and optimizes the blacklist structure and content.

[0234] The false alarm learning unit 54 is used to record features marked as false alarms, analyze the causes and patterns of false alarms, and optimize the blacklist strategy. In one embodiment of the present invention, false alarm learning adopts a combination of frequency statistics and pattern analysis. The system records each feature marked as a false alarm and counts its frequency and distribution. For high-frequency false alarm features, they will be automatically added to the blacklist; for low-frequency false alarm features, their similarity with known false alarm features will be analyzed to try to find common patterns. In addition, the false alarm learning unit 54 will also adjust the weights and thresholds of the similarity calculation based on the false alarm analysis results, and continuously optimize the accuracy of feature matching.

[0235] In practical applications, false positive learning is crucial for improving long-term system availability. For example, in e-commerce systems, sudden increases in traffic during certain periods (such as the start of a promotion) are normal. If the system mistakenly identifies these increases as anomalies and issues an alarm, operations personnel can mark them as false positives. The false positive learning unit 54 analyzes these false positive features, identifies patterns associated with promotional activity and sudden increases in traffic, and blacklists similar features to prevent future false positives. The system also adjusts similarity calculation parameters, such as increasing the weight of activity identification features, to improve pattern recognition accuracy.

[0236] like Figure 6 As shown, the alarm notification module 6 includes an alarm authority unit 61 , a notification routing unit 62 , a notification aggregation unit 63 , an upgrade processing unit 64 and a feedback processing unit 65 .

[0237] The alarm authority unit 61 is used to determine the alarm authority level based on the abnormality level, business importance and impact range. In a preferred embodiment of the present invention, the division of alarm authority is based on the following dimensions:

[0238] Abnormality level: warning level, abnormality level, severe abnormality level;

[0239] Business importance: core business, important business, general business;

[0240] Scope of impact: global impact, local impact, single-point impact;

[0241] Different combinations correspond to different alarm authority levels, which are usually divided into 5 levels:

[0242] Level 1 (highest): serious anomaly + core business + global impact;

[0243] Level 2: Severe anomaly + important business, or anomaly + core business + global impact;

[0244] Level 3: Abnormal + important business, or serious abnormality + general business;

[0245] Level 4: Abnormal + general business, or warning + important business;

[0246] Level 5 (lowest): warning + general business;

[0247] The alarm authority unit 61 automatically determines the alarm authority level according to the configuration information of the interface and the currently detected abnormal situation, which serves as the basis for subsequent notification routing.

[0248] In practice, the level of alarm privilege directly impacts the urgency of notifications and the overall process. For example, if a critical anomaly occurs in a core payment system interface that impacts the entire system, it will be assigned Level 1 alarm privilege, triggering the highest level of notification and response, which may include phone and SMS notifications, work order creation, and automatic capacity expansion. Meanwhile, even if an anomaly occurs in a non-critical content recommendation system interface, it may only be assigned Level 4 or 5 alarm privilege, with notifications limited to email or instant messaging during business hours.

[0249] The notification routing unit 62 is used to select notification recipients and notification channels based on the alarm authority level, time factors, and pre-configured configurations. In one embodiment of the present invention, notification recipients are selected based on a responsibility matrix that clearly defines the responsible teams and individuals for different business operations and exception levels. Notification channels include instant messaging tools, email systems, text messages / phone calls, and work order systems.

[0250] Preferably, the notification routing unit 62 adopts different notification strategies according to time factors:

[0251] Working hours strategy: Prioritize the use of instant messaging tools, with finer notification granularity and shorter response time requirements;

[0252] Non-working hours strategy: prioritize phone calls / text messages, only notify high-level anomalies, and increase notification thresholds to reduce interruptions;

[0253] In practice, notification routing requires a combination of factors. For example, for level 1 alarm privilege anomalies, key personnel will be notified by phone, even during off-hours. For level 3 alarm privilege anomalies, the relevant team will be notified via instant messaging during business hours, but during off-hours, the notification may only be recorded via email and handled during business hours. For level 5 alarm privilege anomalies, the notification may only be recorded in the work ticket system, without proactively notifying personnel. The notification routing unit 62 configures appropriate notification strategies based on the enterprise's organizational structure and operation and maintenance processes.

[0254] The notification aggregation unit 63 is used to intelligently aggregate similar exceptions that occur within a short period of time to prevent notification bombardment. In a preferred embodiment of the present invention, the notification aggregation strategy includes:

[0255] Exceptions on the same interface within a short period of time (e.g., 5 minutes) are aggregated into one notification, including the number of exceptions and time distribution.

[0256] Related interface exceptions are associated and aggregated. For example, exceptions in upstream and downstream interfaces of the call chain may be caused by the same reason.

[0257] Identify and aggregate periodic anomaly patterns. Regular anomalies may be caused by scheduled tasks or periodic business changes.

[0258] The notification aggregation unit 63 also implements an anti-bombing mechanism, including setting a minimum notification interval (such as sending a maximum of one notification within 15 minutes for the same interface), limiting the number of notifications (such as sending a maximum of 10 notifications per hour), and optimizing the notification content (using a summary + details method to facilitate quick understanding of the situation).

[0259] In practical applications, notification aggregation is crucial for improving notification effectiveness. For example, when a database server encounters an issue, it can cause multiple services that rely on it to experience anomalies simultaneously. Without aggregation, the system will send a large number of duplicate or related notifications, resulting in notification bombardment and reduced response efficiency. The notification aggregation unit 63 analyzes the relationships between anomalies, for example through call chain tracing, temporal correlation analysis, or error pattern matching, to aggregate related anomalies into a single notification event. This event also provides complete contextual information to help operations personnel quickly locate the root cause.

[0260] The escalation processing unit 64 is used to trigger the notification escalation process when there is no response to the initial notification or the problem worsens. In one embodiment of the present invention, the triggering conditions for notification escalation include: no response for more than a preset time (e.g., 15 minutes) after the initial notification is issued, the anomaly persists and the trend worsens, and the scope of impact expands from local to global. When the escalation conditions are triggered, the escalation processing unit 64 will perform the following operations: expand the scope of the notification, escalating from the responsible person to the team leader or department head; increase the priority and urgency of the notification; and increase the notification channel, such as upgrading from email to phone notification.

[0261] In practice, notification escalation is a crucial mechanism for ensuring that critical issues are addressed promptly. For example, if there's no response within 15 minutes of the initial notification regarding a serious payment system anomaly, the system automatically escalates the notification to the team leader and notifies them by phone. If there's still no response after another 15 minutes, the notification is escalated to the department head and technical lead, potentially triggering contingency plans such as automatic switchover to a backup system or rate limiting. The escalation processing unit 64 configures different escalation strategies and time thresholds based on the severity of the anomaly and its business impact.

[0262] Feedback processing unit 65 is used to collect user responses to notifications and processing results to provide a basis for system optimization. As described in claim 9, feedback processing unit 65 is further used to record false positive / false negative flag data, calculate notification effectiveness indicators, generate threshold adjustment suggestions, update blacklist feature weights, and optimize notification policy parameters.

[0263] In a preferred embodiment of the present invention, feedback processing adopts a structured approach, including the following aspects: notification response record: recording the viewing time, first response time and processing completion time of the notification; exception marking: the user marks the exception as a real exception or a false alarm; processing result feedback: recording the root cause of the problem, solution and preventive measures; system improvement suggestions: collecting users' opinions and suggestions on system improvements; the feedback processing unit 65 will regularly analyze these feedback data and generate optimization suggestions: for interfaces with high false alarm rates, it is recommended to adjust the threshold parameters or add feature blacklists; for notification channels that do not respond in time, it is recommended to adjust the notification strategy; for system improvement suggestions fed back by users, reasonable improvement measures will be evaluated and implemented.

[0264] In practical applications, feedback processing is a key component of continuous system optimization. For example, by analyzing user feedback, the system may discover a high false alarm rate for a certain type of anomaly due to a low threshold setting, or a low response rate for a notification channel due to inappropriate notification timing. Based on these analysis results, the feedback processing unit 65 automatically adjusts system parameters or generates optimization suggestions to improve the accuracy and effectiveness of the system. Feedback data is also an important basis for evaluating the overall performance of the system. By tracking indicators such as false alarm rate, missed alarm rate, and average response time, the effectiveness of system improvements can be quantified.

[0265] like Figure 7 As shown, the present invention also provides a Java backend log real-time analysis and abnormality warning method, comprising the following steps:

[0266] Collect Java application log data, structure the log data, and store it in a distributed storage system;

[0267] Receive the structured log data, and construct a trend graph based on a time series sharding strategy, wherein the trend graph includes a plurality of time point nodes, each of the time point nodes includes an interface identifier, a count value, and a status mark;

[0268] Dynamically calculate the warning threshold, abnormality threshold and severe abnormality threshold for each interface identifier based on the interface historical call pattern and business importance;

[0269] Calculating the trend change value and the ratio of the time point node, and determining the abnormality level and the fluctuation pattern based on the comparison result of the trend change value, the ratio and the corresponding threshold value;

[0270] Extracting feature values ​​from the anomaly log, comparing the feature values ​​with feature records in the blacklist database, filtering known false positive features, and dynamically updating the blacklist database;

[0271] According to the abnormality level and the preset alarm authority configuration, the notification object and notification channel are determined, the graded alarm is executed, and user feedback is collected to optimize the system parameters.

[0272] The above method steps correspond one-to-one to the system modules, and the implementation method is the same as the aforementioned system embodiment, which will not be repeated here.

[0273] The Java backend log real-time analysis and anomaly warning system and method of the present invention collects and analyzes Java application logs in real time, uses trend analysis and dynamic threshold technology to accurately detect anomalies, and promptly feeds back anomalies to relevant personnel through an intelligent notification mechanism, achieving significant technical effects:

[0274] 1. High-precision anomaly detection: Through multi-dimensional trend analysis and dynamic threshold management, the false alarm rate is reduced by approximately 75%, the missed alarm rate is reduced by approximately 60%, and the average advance warning time for anomalies reaches 15 minutes.

[0275] 2. Efficient operation and maintenance support: Through precise notification and a graded alarm mechanism, invalid alarms have been reduced by more than 50%, 80% of general anomalies do not require manual intervention, and the average anomaly location time has been shortened by 65%.

[0276] 3. High-performance real-time processing: Using a streaming processing architecture and distributed design, the average time from log generation to anomaly detection is less than 2 seconds, supporting the processing of tens of thousands of log records per second.

[0277] 4. Intelligent business insights: Through trend analysis and pattern recognition, it can predict 70% of potential system problems, provide data support for business optimization, and help identify performance bottlenecks.

[0278] 5. Adaptive optimization capability: Through a closed-loop feedback mechanism, the system can continuously learn and optimize, adapt to business changes and system evolution, and continuously improve long-term operating results.

[0279] The preferred embodiments of the present invention have been described in detail above. However, it should be understood by those skilled in the art that the above embodiments are intended to illustrate the present invention only and are not intended to limit the scope of the present invention. Those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A Java backend log real-time analysis and abnormal warning system, characterized by: include: The log collection module is used to obtain log data from Java applications in real time and store the log data in a distributed storage system after structured processing; A trend analysis module, communicatively connected to the log collection module, configured to receive structured log data sent by the log collection module and construct a trend graph based on a time series sharding strategy, wherein the trend graph includes a plurality of time point nodes, each of which includes an interface identifier, a count value, and a status mark; A threshold management module, in communication with the trend analysis module, for dynamically calculating a warning threshold, an abnormality threshold, and a severe abnormality threshold for each interface identifier based on the interface's historical call pattern and the importance of the service; an anomaly detection module, in communication with the trend analysis module and the threshold management module, configured to calculate the trend change value and the ratio of the time point node, and determine the anomaly level and fluctuation pattern based on the comparison result of the trend change value, the ratio and the corresponding threshold; A feature management module, in communication with the anomaly detection module, is configured to extract feature values ​​from the anomaly log, compare the feature values ​​with feature records in a blacklist library, filter known false positive features, and dynamically update the blacklist library; The alarm notification module is in communication with the anomaly detection module and the feature management module, and is used to determine the notification object and notification channel according to the anomaly level and the preset alarm authority configuration, perform hierarchical alarms, and collect user feedback to optimize system parameters.

2. The Java backend log real-time analysis and abnormal warning system according to claim 1 is characterized in that: The trend analysis module includes: The time slicing unit is used to split the continuous time series into discrete time points according to preset rules; A node generation unit, configured to create time point nodes based on the discrete time points, and calculate a count value of each of the time point nodes; A data smoothing unit, configured to perform interpolation processing on the sparse data in the time point node sequence to smooth abnormal fluctuations; The trend graph construction unit is used to associate the time point nodes in chronological order to form a complete trend graph, and perform index management on the trend graph based on the interface identifier.

3. The Java backend log real-time analysis and abnormal warning system according to claim 1 is characterized in that: The threshold management module includes: A reference value calculation unit, configured to calculate a reference value for each interface based on the interface historical call data; an adjustment factor generating unit, configured to generate a threshold adjustment factor according to a fluctuation coefficient, an importance factor, and a time pattern of the interface; a threshold calculation unit, configured to calculate the warning threshold, the abnormality threshold, and the severe abnormality threshold by combining the reference value and the adjustment factor; The threshold evaluation unit is used to regularly evaluate the effectiveness of the threshold and optimize the threshold parameters based on anomaly detection results and user feedback.

4. The Java backend log real-time analysis and abnormal warning system according to claim 1 is characterized in that: The anomaly detection module includes: The trend calculation unit is used to calculate the difference in count values ​​between nodes at adjacent time points and generate a trend change value; A ratio calculation unit, used to calculate the ratio of the node count value at the current time point to the warning threshold; a progressive detection unit, configured to first determine whether the ratio exceeds a warning condition, and if so, further determine whether the trend change value exceeds an abnormal threshold or a severe abnormal threshold; The fluctuation pattern recognition unit is used to identify abnormal patterns such as sudden increase, sudden decrease, continuous increase, continuous decrease and increased fluctuation based on the trend change values ​​of multiple consecutive time points.

5. The Java backend log real-time analysis and abnormal warning system according to claim 1 is characterized in that: The feature management module includes: A feature extraction unit is used to extract structured features and unstructured features from the abnormal log and generate feature values; A similarity calculation unit, configured to calculate the similarity between the feature value and the feature record in the blacklist database; Blacklist management unit, used to maintain the blacklist database and dynamically update blacklist features based on similarity comparison results and user feedback; The false alarm learning unit is used to record features marked as false alarms, analyze the causes and patterns of false alarms, and optimize the blacklist strategy.

6. The Java backend log real-time analysis and abnormal warning system according to claim 1 is characterized in that: The alarm notification module includes: Alarm authority unit, used to determine the alarm authority level based on the abnormality level, business importance and impact scope; A notification routing unit, configured to select notification objects and notification channels based on the alarm authority level, time factors, and preset configurations; Notification aggregation unit, used to intelligently aggregate similar exceptions that occur within a short period of time to prevent notification bombardment; An escalation processing unit, used to trigger the notification escalation process when there is no response to the initial notification or the problem worsens; The feedback processing unit is used to collect user responses to notifications and process results to provide a basis for system optimization.

7. The Java backend log real-time analysis and abnormal warning system according to claim 1 is characterized in that: The log collection module includes: Log receiving unit, used to obtain raw log files from Java containers or applications; A log parsing unit, configured to parse the original log file into structured data and extract interface call information, timestamps, status codes, and error information; A data distribution unit, configured to send the structured data to a distributed message queue; The data storage unit is used to persistently store the structured data in a distributed storage system and provide data indexing and retrieval functions.

8. The Java backend log real-time analysis and abnormal warning system according to claim 3 is characterized in that: The adjustment factor generating unit is further configured to: Adjust threshold parameters based on working day / non-working day identification; Adjust threshold sensitivity based on peak / valley times throughout the day; Adjust threshold elasticity based on seasonal fluctuations in business.

9. The Java backend log real-time analysis and abnormal warning system according to claim 6 is characterized in that: The feedback processing unit is further configured to: Record false positive / false negative flag data; Statistics notification effectiveness indicators; Generate threshold adjustment recommendations; Update blacklist feature weights; Optimize notification policy parameters.

10. A Java backend log real-time analysis and abnormal warning method, characterized in that: include: Collect Java application log data, structure the log data, and store it in a distributed storage system; Receive the structured log data, and construct a trend graph based on a time series sharding strategy, wherein the trend graph includes a plurality of time point nodes, each of the time point nodes includes an interface identifier, a count value, and a status mark; Dynamically calculate the warning threshold, abnormality threshold and severe abnormality threshold for each interface identifier based on the interface historical call pattern and business importance; Calculating the trend change value and the ratio of the time point node, and determining the abnormality level and the fluctuation pattern based on the comparison result of the trend change value, the ratio and the corresponding threshold value; Extracting feature values ​​from the anomaly log, comparing the feature values ​​with feature records in the blacklist database, filtering known false positive features, and dynamically updating the blacklist database; According to the abnormality level and the preset alarm authority configuration, the notification object and notification channel are determined, the graded alarm is executed, and user feedback is collected to optimize the system parameters.

Images