Dynamic log data processing method and device, storage medium and computer equipment
By collecting log flow data in real time and identifying feature values using machine learning models, dynamically adjusting feature thresholds to detect abnormalities, the problems of insufficient real-time and poor adaptability of log analysis and fault detection in the prior art are solved, and efficient and accurate abnormal detection is achieved.
Patent Information
- Application Number
- CN202411658769.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-02-25
AI Technical Summary
The existing log analysis and fault detection methods are insufficient in real-time and poor adaptive environment capabilities, resulting in lag in problem discovery and false positives or missed reports.
By collecting log flow data in real time, using pre-trained machine learning models to identify feature values, and dynamically adjust feature thresholds in multiple dimensions to compare one by one to detect abnormalities.
It improves the accuracy and reliability of feature recognition, enhances the real-time and accuracy of abnormal detection, reduces the occurrence of false positives or missed reports, and ensures stable business operation.
Smart Images
Figure CN119513578A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data analysis and financial technology, and in particular to a dynamic log data processing method, device, storage medium and computer equipment. Background Art
[0002] In modern software development and operation and maintenance environments, log analysis and fault detection are key links to ensure stable system operation. With the popularization of technologies such as cloud computing and big data, the amount of log data generated by the system has increased dramatically, placing higher demands on log processing and analysis capabilities. To meet this challenge, the industry has developed a series of tools, such as static log analysis tools such as ELK Stac, and fault detection tools such as New Relic, which aim to detect and handle potential problems in a timely manner by monitoring and analyzing system logs and performance indicators.
[0003] In the prior art, log analysis and fault detection are mainly achieved in two ways. One method is to use a static log analysis tool, which mainly provides users with log static analysis functions by collecting, storing and querying log data to help users understand system behavior. The second method is to use a rule-based fault detection system, which monitors system indicators through preset rules and thresholds, and triggers an alarm once an anomaly is detected. In addition, there are comprehensive tools such as New Relic, which can not only monitor logs, but also monitor application performance to provide more comprehensive fault detection services.
[0004] However, existing log analysis and fault detection methods still have many shortcomings. First, static log analysis tools lack real-time performance when processing large amounts of data, and often cannot immediately provide feedback on analysis results, resulting in delayed problem discovery. Second, rule-based fault detection systems rely too much on predefined rules, making it difficult to adapt to complex and changing application environments, and are prone to false positives or false negatives. Summary of the invention
[0005] In view of this, the embodiments of the present application provide a dynamic log data processing method, apparatus, storage medium and computer equipment, the main purpose of which is to solve the technical problems of insufficient real-time performance and poor adaptive environment capability of existing log analysis and fault detection methods.
[0006] According to one aspect of the present application, a dynamic log data processing method is provided, the method comprising:
[0007] Collect log stream data of the items to be tested in real time, and identify feature values of the items to be tested in multiple dimensions based on a pre-trained machine learning model;
[0008] Based on the current time characteristics and environmental characteristics, as well as the item characteristics of the item to be measured, the feature thresholds of multiple dimensions are dynamically adjusted to obtain target feature thresholds of multiple dimensions;
[0009] Compare the feature values of the item to be tested in multiple dimensions with the target feature thresholds of the multiple dimensions one by one to obtain feature comparison results of the item to be tested in multiple dimensions;
[0010] Based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes, the abnormality detection results of the item to be tested are obtained.
[0011] According to another aspect of the present application, a dynamic log data processing device is provided, the device comprising:
[0012] A feature recognition module is used to collect log stream data of the items to be tested in real time, and identify feature values of the items to be tested in multiple dimensions based on a pre-trained machine learning model;
[0013] A threshold adjustment module, used to dynamically adjust the feature thresholds of multiple dimensions based on the current time features and environmental features, as well as the item features of the item to be tested, to obtain target feature thresholds of multiple dimensions;
[0014] A feature comparison module, used to compare the feature values of the item to be tested in multiple dimensions with the target feature thresholds of the multiple dimensions one by one, to obtain feature comparison results of the item to be tested in multiple dimensions;
[0015] The anomaly detection module is used to obtain the anomaly detection result of the item to be tested based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes.
[0016] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned dynamic log data processing method is implemented.
[0017] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned dynamic log data processing method when executing the program.
[0018] By means of the above technical scheme, a dynamic log data processing method and device, storage medium and computer equipment provided by the embodiment of the present application can improve the accuracy and reliability of feature recognition by collecting log stream data in real time and performing feature recognition on log stream data through a machine learning model. By dynamically adjusting the feature thresholds of multiple dimensions during anomaly detection, it is possible to better adapt to different environmental and time characteristics, thereby avoiding the problem of false alarms or omissions that may be caused by fixed thresholds in different scenarios. By comparing the feature values of the items to be tested in multiple dimensions with the corresponding target feature thresholds one by one, and comparing the comparison results with the feature detection dimensions corresponding to the preset multiple abnormal modes, it is possible to detect in real time whether an abnormality has occurred in the items to be tested, so that measures can be taken in time to reduce the fault response time and improve the timeliness of anomaly detection. The above method can realize the full process automation processing of log stream data from data collection to anomaly detection, improve the real-time and accuracy of anomaly detection, reduce the occurrence of false alarms or omissions, and provide a strong guarantee for the stable operation of the business.
[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0021] Figure 1 A schematic diagram of an application environment of a dynamic log data processing method provided by an embodiment of the present application is shown;
[0022] Figure 2 A schematic diagram of a flow chart of a dynamic log data processing method provided in an embodiment of the present application is shown;
[0023] Figure 3 A schematic diagram of a process flow of another dynamic log data processing method provided by an embodiment of the present application is shown;
[0024] Figure 4 A schematic diagram of the structure of a dynamic log data processing device provided in an embodiment of the present application is shown;
[0025] Figure 5 A schematic diagram showing the structure of another dynamic log data processing device provided in an embodiment of the present application is shown;
[0026] Figure 6A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] The dynamic log data processing method provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, the client can communicate with the server through the network. The server can collect the log stream data of the item to be tested in real time through the client, and identify the feature values of the item to be tested in multiple dimensions based on the pre-trained machine learning model, and then dynamically adjust the feature thresholds of multiple dimensions based on the current time characteristics and environmental characteristics, as well as the item characteristics of the item to be tested, to obtain the target feature thresholds of multiple dimensions, and then compare the feature values of the item to be tested in multiple dimensions with the target feature thresholds of multiple dimensions one by one, to obtain the feature comparison results of the item to be tested in multiple dimensions, and finally obtain the abnormal detection results of the item to be tested based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes, and finally feed back the abnormal detection results of the item to be tested to the client. The method provided by the present invention can realize the automatic processing of the whole process of log stream data from data collection to abnormal detection, and improve the real-time and accuracy of abnormal detection, and reduce the occurrence of false positives or false negatives. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.
[0029] In one embodiment, Figure 2 As shown, a dynamic log data processing method is provided, which is described by taking the method applied to a computer device such as a client or a server as an example, and includes the following steps:
[0030] Step 101, collect the log stream data of the items to be tested in real time, and identify the feature values of the items to be tested in multiple dimensions based on a pre-trained machine learning model.
[0031] Among them, the items to be tested mainly include the system to be tested (such as the payment system) and the business to be tested (such as the transaction business in the payment system). Different items to be tested have different feature dimensions. For example, when the item to be tested is the system to be tested, the feature dimension may include features of dimensions such as flow change feature value and network attack feature value; when the item to be tested is the business to be tested, the feature dimension may include features of dimensions such as transaction flow feature value, transaction behavior feature value and transaction environment feature value.
[0032] Specifically, a data collector (such as Fluentd, Logstash and other tools) can be used to obtain the log stream data of the items to be tested in real time from multi-source data such as system logs, network traffic logs, security logs, etc. Then, the collected data can be cleaned to remove invalid information, and the cleaned data can be converted to ensure data consistency and availability. Then, the pre-processed log stream data can be parsed and feature extracted by a pre-trained machine learning model, such as a deep learning model or a traditional machine learning model, to extract feature values of multiple dimensions, such as extracting traffic change feature values (such as the number of requests per minute, bandwidth usage), network attack feature values (such as abnormal request frequency, unknown protocol usage), etc. In this embodiment, one or more machine learning models can be used to extract feature values of multiple dimensions of the items to be tested, wherein the same machine learning model can extract features of one dimension or multiple dimensions, and this embodiment is not specifically limited here.
[0033] Step 102, based on the current time characteristics and environmental characteristics, as well as the item characteristics of the item to be measured, dynamically adjust the feature thresholds of multiple dimensions to obtain target feature thresholds of multiple dimensions.
[0034] Among them, time characteristics refer to characteristics related to time, such as historical periodic time (such as the same period last week, the same period last month, the same period last year, etc.) and the current time period (such as morning, noon, afternoon, weekends, weekdays, holidays, etc.); environmental characteristics refer to characteristics related to the environment, such as the region, equipment used, network environment, etc.; the item characteristics of the item to be tested refer to characteristics related to the item to be tested itself, such as the system performance of the system to be tested, the name of the transaction object of the business to be tested, etc.
[0035] Specifically, after identifying the characteristic values of the items to be tested in multiple dimensions, the current time period (such as date, daytime, holidays, etc.) and environmental characteristics (such as geographical location, network environment, etc.) can be obtained in real time, as well as the specific item characteristics of the system or business to be tested (such as the current load of the system, the type of transaction object of the business, etc.), and then the characteristic thresholds of multiple dimensions can be adjusted according to the current actual situation based on historical data. For example, by analyzing historical data, the traffic changes and network attack patterns of the system in different time periods (such as weekdays, weekends, holidays) can be identified. At the same time, external events (such as promotional activities, news reports, etc.) within the preset time period are monitored, and the thresholds are adjusted according to the impact of the events. In addition, considering the network architecture characteristics of the system (such as distributed deployment, load balancing, etc.), the thresholds are adjusted to adapt to different architecture requirements. The load conditions (such as CPU usage, memory usage) and performance indicators (such as response time, throughput) of the system are monitored in real time, and the thresholds are dynamically adjusted according to these indicators. Finally, an algorithm (such as a statistical method or a machine learning algorithm) is used to calculate the target traffic change threshold and the target network attack threshold.
[0036] Step 103 , the feature values of the item to be tested in multiple dimensions are compared one by one with the target feature thresholds of the multiple dimensions to obtain feature comparison results of the item to be tested in multiple dimensions.
[0037] Specifically, after obtaining the target characteristic thresholds of multiple dimensions, the characteristic values of the item to be tested in multiple dimensions can be compared one by one with the target characteristic thresholds of multiple dimensions. For example, the flow change characteristic value of the system to be tested can be compared with the target flow change threshold to generate a flow change characteristic comparison result. If the current flow change characteristic value exceeds the target flow change threshold, the characteristic value is recorded as abnormal. For another example, the network attack characteristic value of the system to be tested is compared with the target network attack threshold to generate a network attack characteristic comparison result. If the current network attack characteristic value exceeds the target network attack threshold, the characteristic value is recorded as abnormal. In this embodiment, the characteristic value of the item to be tested is compared with the target characteristic threshold of the corresponding dimension. It can be a simple threshold comparison, a deviation value comparison, etc. The specific comparison method can be selected according to the actual situation, and this embodiment is not specifically limited here.
[0038] Step 104, based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes, obtain the abnormality detection result of the item to be tested.
[0039] Specifically, it is possible to determine whether the system or business under test has an abnormality based on the feature comparison results of the items under test in multiple dimensions, combined with a plurality of preset abnormal modes (such as network failure mode, network attack mode or transaction fraud mode, etc.) and their corresponding feature detection dimensions (such as flow change feature value, network attack feature value or transaction flow feature value, transaction behavior feature value, etc.). If the feature value of one or more dimensions exceeds the target feature threshold, and these feature values match the preset abnormal mode, corresponding abnormal detection results can be generated, such as network failure warning, network attack alarm or transaction fraud prompt, etc. For example, when the flow change feature value of the system under test is greater than the target flow change threshold, an abnormal detection result of a network failure in the system under test can be generated; when the network attack feature value of the system under test is greater than the target network attack threshold, an abnormal detection result of a network attack on the system under test can be generated; when the flow change feature value is greater than the target flow change threshold and the network attack feature value is greater than the target network attack threshold, an abnormal detection result of a composite attack on the system under test can be generated. In this embodiment, the abnormal detection result can be presented to the operation and maintenance personnel through alarm notifications, log records, etc., so that timely countermeasures can be taken.
[0040] The dynamic log data processing method provided in this embodiment can improve the accuracy and reliability of feature recognition by collecting log stream data in real time and performing feature recognition on log stream data through a machine learning model. By dynamically adjusting the feature thresholds of multiple dimensions during anomaly detection, it can better adapt to different environmental and time characteristics, thereby avoiding the problem of false alarms or omissions that may be caused by fixed thresholds in different scenarios. By comparing the feature values of the items to be tested in multiple dimensions with the corresponding target feature thresholds one by one, and comparing the comparison results with the feature detection dimensions corresponding to the preset multiple abnormal modes, it is possible to detect in real time whether an abnormality has occurred in the items to be tested, so that measures can be taken in time to reduce the fault response time and improve the timeliness of anomaly detection. The above method can realize the automated processing of the entire process of log stream data from data collection to anomaly detection, improve the real-time and accuracy of anomaly detection, reduce the occurrence of false alarms or omissions, and provide a strong guarantee for the stable operation of the business.
[0041] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another dynamic log data processing method is provided, such as Figure 3 As shown, the method includes:
[0042] Step 201 , collect historical log data and historical exception records of the items to be tested, and determine feature detection dimensions and feature detection conditions corresponding to multiple abnormal patterns based on the historical log data and historical exception records.
[0043] Specifically, the computer equipment can periodically collect historical log data and historical abnormal records of the items to be tested (including the system to be tested and the business to be tested) in batches. Among them, for the historical abnormal records, information extraction can be performed to identify multiple abnormal modes related to the items to be tested and the corresponding occurrence period of each abnormal mode. Subsequently, the log data of the items to be tested can be deeply analyzed for the occurrence period corresponding to each abnormal mode using a pre-trained machine learning model to identify the characteristic values of the log data in multiple dimensions under the corresponding period. For example, for the system to be tested, the characteristic dimension can include the flow change characteristic value and the network attack characteristic value; for the business to be tested, the characteristic dimension can include the transaction flow characteristic value, the transaction behavior characteristic value and the transaction environment characteristic value. Finally, based on these characteristic values, the feature detection dimension and feature detection condition corresponding to each abnormal mode can be further determined, thereby providing a basis for subsequent abnormal detection. For example, for network failures, the feature detection dimension can be the flow change characteristic value, and the feature detection condition can be that the flow change characteristic value exceeds a certain threshold; for network attacks, the feature detection dimension can be the network attack characteristic value, and the feature detection condition can be that the network attack characteristic value exceeds a certain threshold, and so on.
[0044] Step 202: collect log stream data of the items to be tested in real time, and identify feature values of the items to be tested in multiple dimensions based on a pre-trained machine learning model.
[0045] Specifically, the items to be tested may include the system to be tested, and the characteristic values of multiple dimensions may include flow change characteristic values and network attack characteristic values, etc. Among them, step 202 can be implemented by the following method: First, the log stream data of the system to be tested is obtained in real time from multi-source data such as system logs, network flow logs, and security logs through a data collector (such as Fluentd, Logstash). Then, the collected data is cleaned, invalid information is removed, and the format is converted to ensure the consistency and availability of the data. Finally, a pre-trained machine learning model, such as an LSTM model based on time series analysis, is used to process the system flow data to extract flow change characteristic values, such as the number of requests per minute, bandwidth usage, etc. In addition, a pre-trained machine learning model, such as an Isolation Forest model based on anomaly detection, is used to process the network flow data to extract network attack characteristic values, such as abnormal request frequency, unknown protocol usage, etc. In this way, the characteristic values of the system to be tested in multiple dimensions can be finally identified.
[0046] Further, the items to be tested may also include the business to be tested, and the characteristic values of multiple dimensions may include transaction flow characteristic values, transaction behavior characteristic values, transaction environment characteristic values, and the like. Among them, step 202 may be implemented by the following method: First, the log stream data of the business to be tested is obtained in real time from multi-source data such as transaction logs and user behavior logs through a data collector. Then, the collected data is cleaned, invalid information is removed, and the format is converted to ensure the consistency and availability of the data. Subsequently, a pre-trained machine learning model, such as an LSTM model based on time series analysis, is used to process the transaction data to extract transaction flow characteristic values, such as transaction amount, transaction frequency, and the like. At the same time, a pre-trained machine learning model, such as an Isolation Forest model based on anomaly detection, is used to process user behavior data to extract transaction behavior characteristic values, such as login frequency, operation anomaly, and the like. Finally, a pre-trained machine learning model, such as a decision tree model based on classification, is used to process the transaction environment data to extract transaction environment characteristic values, such as transaction device type, transaction location, and the like. In this way, the characteristic values of the business to be tested in multiple dimensions can be finally identified.
[0047] Step 203 , based on the current time characteristics and environmental characteristics, as well as the item characteristics of the item to be measured, dynamically adjust the feature thresholds of multiple dimensions to obtain target feature thresholds of multiple dimensions.
[0048] Specifically, when the item to be tested is the system to be tested, the characteristic thresholds of multiple dimensions may include flow change thresholds and network attack thresholds, etc. Among them, step 203 can be implemented by the following method: for the system to be tested, the current time period (such as daytime, night) and geographical location information can be obtained in real time, and the threshold can be adjusted in combination with historical data. For example, historical data is analyzed to identify the flow changes and network attack patterns of the system in different time periods (such as weekdays, weekends, holidays). At the same time, external events (such as promotional activities, news reports, etc.) within a preset time period are monitored, and the threshold is adjusted according to the impact of the event. In addition, considering the network architecture characteristics of the system (such as distributed deployment, load balancing, etc.), the threshold can be adjusted to adapt to different architecture requirements. Further, the load of the system (such as CPU usage, memory usage) and performance indicators (such as response time, throughput) can also be monitored in real time, and the threshold can be dynamically adjusted according to these indicators. Finally, an algorithm (such as a statistical method or a machine learning algorithm) can be used to calculate the target flow change threshold and the target network attack threshold to obtain target characteristic thresholds of multiple dimensions.
[0049] Further, when the item to be tested is a business to be tested, the characteristic thresholds of multiple dimensions may include a transaction flow threshold, a transaction behavior threshold, a transaction environment threshold, and the like. Among them, step 203 may be implemented by the following method: for the business to be tested, the preset transaction flow threshold, transaction behavior threshold, and transaction environment threshold may be dynamically adjusted based on the transaction time distribution, the transaction periodic change trend, the transaction market environment, the external events occurring within the preset time period, the transaction user type, and the transaction user behavior characteristics. For example, by analyzing historical data, the transaction patterns in different time periods (such as working days, weekends, holidays) and different external environments (such as promotional activities, news reports, etc.) may be identified, and the thresholds may be dynamically adjusted in combination with the current transaction time distribution and market environment, wherein the process of dynamically adjusting the thresholds may also be implemented based on statistical methods or machine learning algorithms. Finally, the target characteristic thresholds of multiple dimensions may be obtained based on the adjusted target transaction flow threshold, target transaction behavior threshold, and target transaction environment threshold to ensure that the system can more accurately identify and respond to abnormal transaction behaviors.
[0050] Step 204 , the feature values of the item to be tested in multiple dimensions are compared one by one with the target feature thresholds of the multiple dimensions to obtain feature comparison results of the item to be tested in multiple dimensions.
[0051] Specifically, after obtaining the target characteristic thresholds of multiple dimensions, the characteristic values of the identified items to be tested in multiple dimensions can be compared one by one with the target characteristic thresholds of multiple dimensions, so as to obtain the characteristic comparison results of the items to be tested in multiple dimensions. For example, the flow change characteristic value of the system to be tested can be compared with the target flow change threshold to generate the flow change characteristic comparison result. If the current flow change characteristic value exceeds the target flow change threshold, the characteristic value is recorded as abnormal. For another example, the network attack characteristic value of the system to be tested collected in real time is compared with the target network attack threshold to generate the network attack characteristic comparison result. If the current network attack characteristic value exceeds the target network attack threshold, the characteristic value is recorded as abnormal. Similarly, for the business to be tested, the transaction flow characteristic value of the business to be tested can be compared with the target transaction flow threshold to generate the transaction flow characteristic comparison result; the transaction behavior characteristic value of the business to be tested can be compared with the target transaction behavior threshold to generate the transaction behavior characteristic comparison result; the transaction environment characteristic value of the business to be tested can be compared with the target transaction environment threshold to generate the transaction environment characteristic comparison result. In this embodiment, the threshold comparison may be a simple numerical comparison or a deviation value comparison. The specific method may be selected according to the actual situation and is not specifically limited in this embodiment.
[0052] Step 205 , based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes, obtain the abnormality detection result of the item to be tested.
[0053] Specifically, when it is detected that the item to be tested is the system to be tested, the following method can be used to detect the system to be tested: when it is detected that the feature comparison result of the traffic change characteristic value of the system to be tested meets the feature detection condition of the network fault, the abnormal detection result of the network fault is obtained; when it is detected that the feature comparison result of the network attack characteristic value of the system to be tested meets the feature detection condition of the network attack, the abnormal detection result of the network attack is obtained; when it is detected that the feature comparison result of the traffic change characteristic value of the system to be tested meets the feature detection condition of the network fault, and the feature comparison result of the network attack characteristic value of the system to be tested meets the feature detection condition of the network attack, the abnormal detection result of the compound attack is obtained.
[0054] Furthermore, when it is detected that the business to be tested is the system to be tested, the following method can be used to detect the business to be tested: when it is detected that the feature comparison result of the transaction flow characteristic value of the business to be tested meets the feature detection condition of the transaction abnormality, an anomaly detection result of the transaction abnormality is obtained; when it is detected that the feature comparison result of the transaction behavior characteristic value of the business to be tested meets the feature detection condition of the account abnormality, an anomaly detection result of the account abnormality is obtained; when it is detected that the feature comparison result of the transaction environment characteristic value of the business to be tested meets the feature detection condition of the equipment abnormality, an anomaly detection result of the equipment abnormality is obtained.
[0055] Step 206: Based on the abnormal detection result of the item to be tested, execute response measures and / or send the time point and cause of the abnormal mode occurrence and the corresponding measure suggestions to the terminal device of the operator.
[0056] Specifically, when the abnormal detection results of the item to be tested include at least one abnormal pattern, the following method can be used for response processing: First, collect the log stream data corresponding to the item to be tested, and perform root cause analysis on the log stream data corresponding to the item to be tested to obtain the time point and preliminary cause of the abnormal pattern. Then, based on the abnormal pattern corresponding to the item to be tested, query the response measures and measure suggestions corresponding to the abnormal pattern in the historical abnormal records. Finally, execute response measures, such as sending alarm notifications, starting the backup system, blocking abnormal IPs, automatically restoring the system, etc. At the same time, the time point and cause of the abnormal pattern, as well as the measure suggestions corresponding to the abnormal pattern, are sent to the terminal device of the operator, so that the operation and maintenance personnel can understand the abnormal situation in time and take further processing measures to reduce losses.
[0057] By applying the technical solution of this embodiment, it is possible to collect log stream data of the items to be tested in real time, and perform feature recognition and multi-dimensional feature comparison based on a pre-trained machine learning model. At the same time, the feature threshold can be dynamically adjusted according to the current time characteristics, environmental characteristics, and item characteristics, so as to improve the accuracy and timeliness of anomaly detection and reduce the occurrence of false positives or false negatives. In addition, the above method can also perform anomaly detection based on feature comparison results and preset anomaly patterns, and automatically generate anomaly detection results and measures recommendations, thereby providing powerful decision-making support for operators. Based on this, the above method can improve the security and stability of the items to be tested, thereby reducing potential security risks.
[0058] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. In addition, the numbers corresponding to the various steps in the above embodiments only serve as identification and are not used as a limitation on the order in which the steps are executed. The order in which the steps are executed in each embodiment can be set according to actual conditions.
[0059] Further, as Figure 2 and Figure 3 The specific implementation of the method, the embodiment of the present application provides a dynamic log data processing device, such as Figure 4 As shown, the device comprises:
[0060] The feature recognition module 31 is used to collect the log stream data of the items to be tested in real time, and identify the feature values of the items to be tested in multiple dimensions based on the pre-trained machine learning model;
[0061] A threshold adjustment module 32 is used to dynamically adjust the feature thresholds of multiple dimensions based on the current time features and environmental features, as well as the feature features of the item to be measured, to obtain target feature thresholds of multiple dimensions;
[0062] A feature comparison module 33 is used to compare the feature values of the item to be tested in multiple dimensions with the target feature thresholds of the multiple dimensions one by one to obtain feature comparison results of the item to be tested in multiple dimensions;
[0063] The anomaly detection module 34 is used to obtain an anomaly detection result of the item to be tested based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple anomaly modes.
[0064] In a specific application scenario, the item to be tested includes the system to be tested, and the characteristic thresholds of the multiple dimensions include a traffic change threshold and a network attack threshold; the threshold adjustment module 32 is specifically used to dynamically adjust the preset traffic change threshold and network attack threshold based on historical periodic time, the current time period and geographical location, external events occurring within a preset time period, system network architecture, system load conditions and system performance to obtain a target traffic change threshold and a target network attack threshold; based on the target traffic change threshold and the target network attack threshold, obtain target characteristic thresholds of multiple dimensions.
[0065] In a specific application scenario, the anomaly detection module 34 is specifically used to obtain an anomaly detection result of a network fault when it is detected that the feature comparison result of the flow change characteristic value of the system to be tested meets the feature detection condition of the network fault; to obtain an anomaly detection result of a network attack when it is detected that the feature comparison result of the network attack characteristic value of the system to be tested meets the feature detection condition of the network attack; to obtain an anomaly detection result of a compound attack when it is detected that the feature comparison result of the flow change characteristic value of the system to be tested meets the feature detection condition of the network fault and the feature comparison result of the network attack characteristic value of the system to be tested meets the feature detection condition of the network attack.
[0066] In a specific application scenario, the items to be tested include the business to be tested, and the characteristic thresholds of the multiple dimensions include a transaction flow threshold, a transaction behavior threshold and a transaction environment threshold; the threshold adjustment module 32 is specifically used to dynamically adjust the preset transaction flow threshold, transaction behavior threshold and transaction environment threshold based on transaction time distribution, transaction periodic change trend, transaction market environment, external events occurring within a preset time period, transaction user type, and transaction user behavior characteristics to obtain the target transaction flow threshold, target transaction behavior threshold and target transaction environment threshold; based on the target transaction flow threshold, the target transaction behavior threshold and the target transaction environment threshold, the target characteristic thresholds of multiple dimensions are obtained.
[0067] In a specific application scenario, the anomaly detection module 34 is specifically used to obtain an anomaly detection result of a transaction anomaly when it is detected that the feature comparison result of the transaction flow characteristic value of the business to be tested meets the feature detection condition of the transaction anomaly; to obtain an anomaly detection result of an account anomaly when it is detected that the feature comparison result of the transaction behavior characteristic value of the business to be tested meets the feature detection condition of the account anomaly; to obtain an anomaly detection result of an equipment anomaly when it is detected that the feature comparison result of the transaction environment characteristic value of the business to be tested meets the feature detection condition of the equipment anomaly.
[0068] In specific application scenarios, such as Figure 5As shown, the device also includes an anomaly extraction module 35, which is specifically used to periodically batch collect historical log data and historical anomaly records of the items to be tested, wherein the items to be tested include the systems to be tested and the services to be tested; extract information from the historical anomaly records to obtain a plurality of anomaly patterns corresponding to the items to be tested and an occurrence period corresponding to each anomaly pattern; for the occurrence period corresponding to each anomaly pattern, based on a pre-trained machine learning model, identify the characteristic values of the items to be tested in multiple dimensions; based on the characteristic values of the items to be tested in multiple dimensions corresponding to each anomaly pattern, determine the feature detection dimensions and feature detection conditions corresponding to each anomaly pattern.
[0069] In specific application scenarios, such as Figure 5 As shown, the device also includes a measure response module 36, which is specifically used to collect log stream data corresponding to the item to be tested in response to the abnormal detection result of the item to be tested including at least one abnormal pattern; perform root cause analysis on the log stream data corresponding to the item to be tested to obtain the time point and preliminary cause of the abnormal pattern; based on the abnormal model corresponding to the item to be tested, query the response measures and measure suggestions corresponding to the abnormal pattern in the historical abnormal records; execute the response measures, and / or send the time point and cause of the abnormal pattern and the measure suggestions corresponding to the abnormal pattern to the terminal device of the operator.
[0070] It should be noted that other corresponding descriptions of the functional units involved in the dynamic log data processing device provided in the embodiment of the present application can be referred to Figure 2 and Figure 3 The corresponding description in the method will not be repeated here.
[0071] The present application also provides a computer device, such as Figure 6 As shown, the computer device can specifically be a personal computer, a server, a network device, etc. The computer device includes a system bus, a processor, a memory and a communication interface, and can also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.
[0072] Those skilled in the art will appreciate that the structure of the above-mentioned computer device is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.
[0073] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0074] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0075] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0076] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0077] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A dynamic log data processing method, characterized in that: The method comprises: Collect log stream data of the items to be tested in real time, and identify feature values of the items to be tested in multiple dimensions based on a pre-trained machine learning model; Based on the current time characteristics and environmental characteristics, as well as the item characteristics of the item to be tested, dynamically adjust the feature thresholds of multiple dimensions to obtain target feature thresholds of multiple dimensions; Compare the feature values of the item to be tested in multiple dimensions with the target feature thresholds of the multiple dimensions one by one to obtain feature comparison results of the item to be tested in multiple dimensions; Based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes, the abnormality detection results of the item to be tested are obtained.
2. The method according to claim 1, characterized in that The item to be tested includes a system to be tested, and the characteristic thresholds of the multiple dimensions include a flow change threshold and a network attack threshold; then based on the current time characteristics and environmental characteristics, and the item characteristics of the item to be tested, the characteristic thresholds of the multiple dimensions are dynamically adjusted to obtain target characteristic thresholds of the multiple dimensions, including: Based on historical periodic time, current time period and geographical location, external events occurring within a preset time period, system network architecture, system load and system performance, dynamically adjust the preset traffic change threshold and network attack threshold to obtain the target traffic change threshold and target network attack threshold; Based on the target traffic change threshold and the target network attack threshold, target feature thresholds in multiple dimensions are obtained.
3. The method according to claim 2, characterized in that The abnormality detection result of the item to be tested is obtained based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes, including: When it is detected that the characteristic comparison result of the flow change characteristic value of the system to be tested meets the characteristic detection condition of the network fault, an abnormal detection result of the network fault is obtained; When it is detected that the feature comparison result of the network attack feature value of the system to be tested meets the feature detection condition of the network attack, an abnormal detection result of the network attack is obtained; When it is detected that the feature comparison result of the flow change characteristic value of the system to be tested meets the feature detection condition of the network fault, and the feature comparison result of the network attack characteristic value of the system to be tested meets the feature detection condition of the network attack, an abnormal detection result of the composite attack is obtained.
4. The method according to claim 1, characterized in that: The item to be tested includes the business to be tested, and the characteristic thresholds of the multiple dimensions include a transaction flow threshold, a transaction behavior threshold, and a transaction environment threshold; then based on the current time characteristics and environmental characteristics, and the item characteristics of the item to be tested, the characteristic thresholds of the multiple dimensions are dynamically adjusted to obtain target characteristic thresholds of the multiple dimensions, including: Based on the transaction time distribution, transaction periodic change trend, transaction market environment, external events occurring within a preset time period, transaction user type, and transaction user behavior characteristics, the preset transaction flow threshold, transaction behavior threshold, and transaction environment threshold are dynamically adjusted to obtain the target transaction flow threshold, target transaction behavior threshold, and target transaction environment threshold; Based on the target transaction flow threshold, the target transaction behavior threshold and the target transaction environment threshold, target feature thresholds of multiple dimensions are obtained.
5. The method according to claim 4, characterized in that The abnormality detection result of the item to be tested is obtained based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes, including: When it is detected that the feature comparison result of the transaction flow feature value of the business to be tested meets the feature detection condition of the transaction anomaly, an abnormality detection result of the transaction anomaly is obtained; When it is detected that the feature comparison result of the transaction behavior feature value of the business to be tested meets the feature detection condition of the account abnormality, an abnormality detection result of the account abnormality is obtained; When it is detected that the feature comparison result of the transaction environment feature value of the business to be tested meets the feature detection condition of the device abnormality, an abnormality detection result of the device abnormality is obtained.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Periodically batch collect historical log data and historical exception records of items to be tested, where the items to be tested include the systems to be tested and the services to be tested; Extracting information from the historical abnormal records to obtain a plurality of abnormal modes corresponding to the items to be tested and an occurrence period corresponding to each abnormal mode; For each abnormal mode corresponding to the occurrence period, based on the pre-trained machine learning model, identify the characteristic values of the item to be tested in multiple dimensions; Based on the characteristic values of the item to be detected in multiple dimensions corresponding to each abnormal pattern, the characteristic detection dimension and characteristic detection condition corresponding to each abnormal pattern are determined.
7. The method according to any one of claims 1 to 5, characterized in that: After obtaining the abnormality detection result of the item to be tested, the method further includes: In response to the abnormal detection result of the item to be tested including at least one abnormal pattern, collecting log stream data corresponding to the item to be tested; Perform root cause analysis on the log stream data corresponding to the item to be tested to obtain the time point and preliminary cause of the abnormal pattern; Based on the abnormal model corresponding to the item to be tested, query the historical abnormal records for response measures and measure suggestions corresponding to the abnormal pattern; Execute the response measures, and / or send the time point and cause of occurrence of the abnormal mode and the measure suggestions corresponding to the abnormal mode to the terminal device of the operator.
8. A dynamic log data processing device, characterized in that: The device comprises: A feature recognition module is used to collect log stream data of the items to be tested in real time, and identify feature values of the items to be tested in multiple dimensions based on a pre-trained machine learning model; A threshold adjustment module, used to dynamically adjust the feature thresholds of multiple dimensions based on the current time features and environmental features, as well as the feature features of the item to be tested, to obtain target feature thresholds of multiple dimensions; A feature comparison module is used to compare the feature values of the item to be tested in multiple dimensions with the target feature thresholds of the multiple dimensions one by one to obtain feature comparison results of the item to be tested in multiple dimensions; The anomaly detection module is used to obtain the anomaly detection result of the item to be tested based on the feature comparison results of the item to be tested in multiple dimensions and the feature detection dimensions corresponding to the preset multiple abnormal modes.
9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.