Fault analyzing and positioning method and system based on log data

Abstract

The invention discloses a fault analysis and positioning method and system based on log data, and relates to the technical field of log data analysis and fault positioning, and the method comprises the steps: collecting first log data, and carrying out the preprocessing; identifying a log mode through a first identification method, and establishing a basic mode multi-dimensional vector of the abnormal log; and through a first log analysis tool, comparing the real-time log data with a historical baseline, identifying abnormity and carrying out fault positioning. According to the method, the log modes with similar characteristics are recognized and the multi-dimensional vector representation is established through the application of the first recognition method, so that the abnormal log modes can be accurately recognized, the accuracy of anomaly detection is improved, the false alarm rate is reduced through the comparison of angular distance calculation and the preset threshold value, and the accuracy of anomaly detection is improved. By using the first log analysis tool, the real-time log data can be compared with the historical base line, so that the abnormity is identified and the fault is positioned.

Description

Technical Field

[0001] The present invention relates to the technical field of log data analysis and fault location, and in particular to a fault analysis and location method and system based on log data. Background Art

[0002] With the development of information technology, log data analysis plays an increasingly important role in system monitoring and fault diagnosis. Existing log management technologies have been able to realize log collection, storage and basic query functions, but there are still certain limitations in log data preprocessing, pattern recognition and fault location. Traditional log analysis systems often rely on simple keyword matching and threshold alarms, which are difficult to cope with the diverse log data and high-dimensional abnormal patterns in complex systems.

[0003] Although existing technologies can help operation and maintenance personnel identify system anomalies to a certain extent, they still have shortcomings in the life cycle management of log data, efficient preprocessing, accurate pattern recognition and intelligent fault location. For example, existing technologies often ignore the life cycle management of log data, resulting in high storage costs and low efficiency. In the preprocessing stage, there is a lack of effective data cleaning and formatting methods, which affects the accuracy of subsequent analysis. In terms of pattern recognition, existing methods are difficult to capture complex patterns and potential anomalies in log data. In terms of fault location, existing technologies cannot provide accurate positioning information, and often require operation and maintenance personnel to spend a lot of time on manual troubleshooting. Therefore, how to achieve efficient management of log data, accurate analysis and rapid fault location has become an urgent problem to be solved in current log analysis technology. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: the existing log data analysis methods have the problems of imperfect log data life cycle management, low preprocessing efficiency, low pattern recognition accuracy, and inaccurate fault location, as well as how to achieve intelligent management of log data, efficient preprocessing, accurate pattern recognition and rapid fault location.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a fault analysis and positioning method based on log data, comprising collecting first log data and performing preprocessing; identifying log patterns through a first identification method, and establishing a basic pattern multi-dimensional vector of abnormal logs; comparing real-time log data with historical baselines through a first log analysis tool, identifying anomalies and locating faults.

[0007] As a preferred solution of the log data-based fault analysis and positioning method described in the present invention, wherein: the first log data includes managing the log data according to the life cycle and setting different storage strategies.

[0008] As a preferred solution of the log data-based fault analysis and location method described in the present invention, the preprocessing includes collecting log data from microservices and performing preprocessing.

[0009] As a preferred solution of the fault analysis and positioning method based on log data described in the present invention, wherein: the identification of log patterns includes identifying features of the preprocessed log data through a first identification method, statistically calculating the average value of specific common features of each category, and establishing a multi-dimensional vector of the basic pattern of each log category.

[0010] As a preferred solution of the fault analysis and positioning method based on log data described in the present invention, wherein: the establishment of the basic pattern multi-dimensional vector of the abnormal log includes comparing the real-time log data with the basic pattern multi-dimensional vector of each log category, identifying the abnormal log pattern, and establishing the basic pattern multi-dimensional vector of the abnormal log.

[0011] As a preferred solution of the fault analysis and positioning method based on log data described in the present invention, wherein: the identification of anomalies includes collecting and learning log classification patterns, matching real-time log data and recording results, generating historical data to form a baseline, comparing real-time data with the historical baseline, and determining whether various types of logs are within the normal fluctuation range. If the log type exceeds the normal fluctuation range, an alarm is issued to remind operation and maintenance personnel to pay attention.

[0012] As a preferred solution of the fault analysis and positioning method based on log data described in the present invention, the fault positioning includes generating sub-instance ID information according to the abnormal log, querying the configuration management database to obtain asset information, locating the corresponding services, modules and systems, judging the log type according to the specific abnormal pattern matching the abnormal log, and giving processing suggestions.

[0013] Another object of the present invention is to provide a fault analysis and location system based on log data, which can identify abnormal patterns and locate faults by comparing real-time log data with historical baselines, thereby solving the problems of low pattern recognition accuracy and inaccurate fault location in current log data analysis methods.

[0014] As a preferred solution of the fault analysis and positioning system based on log data described in the present invention, it includes: a log collection module, a log identification module, and a fault positioning module; the log collection module is used to collect log data from each microservice and perform preprocessing to provide high-quality original data for subsequent log pattern recognition and anomaly detection; the log identification module is used to statistically calculate the average value of specific common features of each category, establish a basic pattern multi-dimensional vector of each log category, compare real-time log data with the basic pattern multi-dimensional vector of each log category, identify abnormal log patterns, and establish a basic pattern multi-dimensional vector of abnormal logs; the fault positioning module is used to compare real-time log data with historical baselines, identify abnormal patterns and perform fault positioning.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a fault analysis and positioning method based on log data.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a fault analysis and location method based on log data.

[0017] Beneficial effects of the present invention: The fault analysis and positioning method based on log data provided by the present invention manages log data according to the life cycle and sets different storage strategies, which not only optimizes storage efficiency and cost, but also ensures rapid access to key business data. By applying the first identification method, log patterns with similar characteristics are identified, and the mean of the log characteristics in each cluster is calculated to establish a multi-dimensional vector representation, so that abnormal log patterns can be accurately identified, which not only improves the accuracy of anomaly detection, but also reduces the false alarm rate through angular distance calculation and comparison with preset thresholds. For operation and maintenance personnel, it means that they can focus on real system problems more effectively. Through the use of the first log analysis tool, real-time log data can be compared with historical baselines to identify anomalies and locate faults. By collecting and learning the log classification patterns and regularly updating historical data, the system can adapt to changes in log data and maintain the timeliness of anomaly detection. The present invention achieves better results in terms of accuracy, efficiency and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 An overall flow chart of a fault analysis and location method based on log data provided by the first embodiment of the present invention.

[0020] Figure 2 A module diagram of a fault analysis and location system based on log data provided by the third embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0022] Example 1, reference Figure 1 , is an embodiment of the present invention, and provides a fault analysis and location method based on log data, comprising:

[0023] S1: Collect the first log data and perform preprocessing.

[0024] Furthermore, the first log data includes managing log data according to the life cycle, using hot, warm, and cold architecture to manage the log data life cycle, optimizing storage efficiency and cost, and automatically migrating log data to different storage stages according to data importance and access frequency. The storage stage includes four stages: hot, warm, cold, and deletion, and setting different storage strategies. For example, the hot stage processes the read and write operations of recent data, uses SSD to store hot data, the warm stage manages the reading of old data, and uses high-performance disks to store warm data. The cold stage archives rarely accessed data and uses large-capacity disks to store cold data to optimize cost and space. In the deletion stage, according to the retention policy, the data is transferred to external storage or permanently deleted.

[0025] It should be noted that the preprocessing collects raw log data from each microservice and performs preprocessing, cleans the raw log data, removes invalid or redundant information, formats the cleaned log data to conform to a unified format standard, and normalizes the formatted log data to eliminate dimensional differences between the data. Data cleaning includes removing empty or invalid logs, removing duplicate logs, and removing irrelevant information in the logs.

[0026] It should also be noted that the first log data includes but is not limited to timestamp, service identifier, event level, event content, error code, call chain information, user information, system information, etc., and the first log data can be collected by a variety of methods, such as distributed log collection tools or microservice framework built-in log collection functions. As a preferred embodiment, in this embodiment, the first log data is collected using the built-in log collection function of the microservice framework. For example, for the order service of the e-commerce platform, the first log data includes order creation time, order status, user information, payment information, logistics information, etc. The built-in log collection function of the Kubernetes microservice framework is used to automatically collect the log data of the order service. The key business log uses SSD to store hot data with a retention period of 7 days, and the ordinary business log uses a high-performance disk to store warm data with a retention period of 30 days. The debug log uses a large-capacity disk to store cold data with a retention period of 1 year, and sets abnormal detection rules such as abnormal order status, payment failure, and missing logistics information, and triggers an alarm.

[0027] It should also be noted that in another feasible embodiment, a distributed log collection tool is used to collect the first log data. For example, for the payment service of a financial platform, the first log data includes payment time, payment amount, payment status, user information, account information, etc. The Fluentd distributed log collection tool is used to collect log data from each payment service instance. All logs use SSD to store hot data with a retention period of 90 days. Anomaly detection rules such as payment failure, abnormal payment amount, and insufficient account balance are set, and alarms are triggered.

[0028] S2: Identify the log pattern through the first identification method, and establish a basic pattern multi-dimensional vector of the abnormal log.

[0029] Furthermore, identifying log patterns includes identifying features of the preprocessed log data through a first identification method, the identification features including but not limited to timestamp, service identifier, event level, event content, error code, call chain information, etc., statistically calculating the average value of specific common features of each category, and establishing a multi-dimensional vector of the basic pattern of each log category.

[0030] It should be noted that establishing the basic pattern multidimensional vector of the abnormal log includes comparing the real-time log data with the basic pattern multidimensional vector of each log category, calculating the angular distance between the real-time log and the basic pattern vector of the abnormal category, and comparing it with a preset threshold, identifying the abnormal log pattern, and establishing the basic pattern multidimensional vector of the abnormal log.

[0031] It should also be noted that the first identification method can be a variety of methods, such as the DP-Kmeans clustering algorithm or the Kmeans clustering algorithm. This embodiment uses the DP-Kmeans clustering algorithm to cluster logs according to the characteristics of abnormal log data. By selecting a log structure that accounts for a large proportion of the total logs, the DP-Kmeans clustering algorithm evaluates the number of common features between logs that account for a large proportion of the total logs. Then, the DP-Kmeans algorithm classifies the log set, thereby being able to identify log categories based on common feature characteristics. This method helps to perform structured analysis of logs and determine key patterns and classifications that are critical to comprehensive log management. As a preferred embodiment, in this embodiment, the DP-Kmeans clustering algorithm is used to cluster the logs, identify log patterns with similar features, calculate the mean of the log features in each cluster, and establish a multidimensional vector representation of the log pattern. For example, for the order service of the e-commerce platform, the identification features include order creation time, order status, user information, payment information, logistics information, etc. The DP-Kmeans algorithm is used, and the number of clusters is set to 3, representing normal logs, abnormal logs, and unknown logs, respectively. The mean of the log features in each cluster is calculated to form a multidimensional vector, and the angular distance between the real-time log and the basic pattern vector of the abnormal category is calculated and compared with a preset threshold. The threshold is set to 0.8. When the angular distance is less than 0.8, it is identified as a new abnormal log.

[0032] It should also be noted that, in another feasible embodiment, the Kmeans clustering algorithm is used to cluster the logs to identify log patterns with similar characteristics. For example, for payment services on financial platforms, the identification features include payment time, payment amount, payment status, user information, account information, etc. The Kmeans clustering algorithm is used to set the number of clusters to 2, representing normal logs and abnormal logs, respectively. The mean of the log features in each cluster is calculated to form a multidimensional vector. The Euclidean distance between the real-time log and the basic pattern vector of the abnormal category is calculated and compared with a preset threshold. The threshold is set to 1.5. When the Euclidean distance is less than 1.5, it is identified as a new abnormal log.

[0033] S3: Use the first log analysis tool to compare real-time log data with historical baselines, identify anomalies, and locate faults.

[0034] Furthermore, identifying anomalies includes collecting learned log classification patterns, matching real-time log data and recording the results, generating historical data to form a baseline, comparing real-time data with the historical baseline, and determining whether various types of logs are within the normal fluctuation range. If the log type exceeds the normal fluctuation range, an alarm will be issued to alert the operation and maintenance personnel.

[0035] It should be noted that fault location includes generating secondary instance ID information based on the exception log, querying the configuration management database to obtain asset information, locating the corresponding services, modules and systems, judging the log type based on the specific exception pattern that matches the exception log, and giving processing suggestions.

[0036] It should also be noted that the log classification patterns learned are collected, stored, and updated regularly to form historical data. The historical data can be stored in a relational database or a NoSQL database for real-time data matching. The real-time log data is matched with the historical data using a first log analysis tool, and the matching results are recorded. For example, tools such as ELK Stack or Kibana can be used for data visualization, and the real-time data and historical data can be displayed in the form of charts. By analyzing the historical data, baselines of various types of logs can be generated, such as the average number, maximum number, and minimum number of each type of log under normal circumstances. The real-time data is compared with the historical baseline. If the number of logs of a certain type is found to exceed the normal fluctuation range, an alarm is issued to alert the operation and maintenance personnel. For example, threshold judgment or statistical test methods can be used for anomaly detection. As a preferred embodiment, in this embodiment, ELK Stack (Elasticsearch Logstash Kibana) is used as the first log analysis tool. For example, Logstash is used to collect and parse real-time log data, and the parsed data is stored in Elasticsearch. Kibana is used to create a visual dashboard to display real-time data and historical baselines, and a threshold is set. For ERROR-level logs, the threshold is set to 2 times the standard deviation of the baseline average value. For WARNING-level logs, the threshold is set to 1.5 times the standard deviation of the baseline average value. The threshold is automatically adjusted according to the seasonal changes of historical data. For example, if historical data shows that 2 pm is the peak trading period, the threshold is increased during this period. The threshold adjustment formula is: dynamic threshold = baseline average value + (seasonal factor × baseline standard deviation). When the number of real-time logs exceeds the dynamic threshold, an alarm is triggered. The alarm condition formula is: real-time log number > dynamic threshold. After the alarm is triggered, the system automatically notifies the operation and maintenance personnel by email, SMS or Slack message. The alarm notification contains detailed information about the exception, such as the exception level, occurrence time, affected system components, etc.

[0037] It should also be noted that, in another feasible embodiment, Kibana is used as the first log analysis tool, combined with Elasticsearch for data storage and analysis, and Kibana's Timelion function or Elasticsearch's aggregation query is used to perform statistical tests. The confidence level of the statistical test is set, for example, 95%, to determine the significance of the anomaly. When the statistical test results show that there is a significant difference between the real-time data and the historical baseline, an alarm mechanism is triggered. For example, a T test is used to compare the difference between the real-time log data and the historical baseline, which is suitable for small sample data. For large sample data, a Z test is used, and the confidence level is set to 95%, which means that if an anomaly is detected, there is a 95% certainty that it is not a random fluctuation. The statistical test steps include extracting samples from real-time data, calculating sample mean and standard deviation, using historical baseline data as a population reference, calculating population mean and standard deviation, performing T test or Z test, calculating test statistics and corresponding p-values, where p-values ​​refer to the probability of observed results or more extreme results in a hypothesis test when the null hypothesis is true. The null hypothesis refers to a situation where there is no difference or no effect, such as no significant difference between real-time data and historical baselines. The alternative hypothesis is opposite to the null hypothesis and refers to a situation where there is a difference or effect, such as a significant difference between real-time data and historical baselines. The abnormal judgment logic includes that if the p-value is less than the significance level (α=0.05), the null hypothesis is rejected, and it is considered that there is a significant difference between real-time data and historical baselines, triggering an alarm.

[0038] It should also be noted that fault location includes generating sub-instance ID information based on the exception log. The exception log usually contains some key information, such as service name, service instance ID, timestamp, error code, etc., which can be extracted from this information and combined with the log context to generate a sub-instance ID information. For example, the service name and instance ID can be used as part of the sub-instance ID information, and the timestamp and error code can be used as the other part of the sub-instance ID information. After the sub-instance ID information is generated, the sub-instance ID information can be used to query the CMDB to obtain asset information. The CMDB usually stores the configuration information of each service, such as the module, system, host and other information to which the service belongs. By querying the CMDB, the specific location where the fault occurs can be located, such as the physical host or virtual machine where the service instance is located, and the queried asset information can be displayed to the operation and maintenance personnel, such as displaying the physical host or virtual machine information where the service instance is located, as well as the module and system information to which the service instance belongs on the interface of the log analysis tool. As a preferred embodiment, in this embodiment, the secondary instance ID information consists of the following parts, each of which has a clear character length limit. The service name (ServiceName) is a maximum of 20 characters, the service instance ID (InstanceID) is fixed to a 12-character UUID, the timestamp (Timestamp) is accurate to milliseconds and has a total of 13 digits, the error code (ErrorCode) is a maximum of 5 characters, and the format of the secondary instance ID is: ServiceName_InstanceID_Timestamp_ErrorCode, with a total length of no more than 50 characters. When an abnormal log is detected, the system first parses the log and extracts key information fields. If the log format meets expectations, the system will extract the service name, instance ID, timestamp and error code according to predefined rules. If the log format does not meet expectations, the system will record the parsing error and mark the original log as requiring manual review. The secondary instance ID generation logic includes the system extracting the service name and instance ID from the abnormal log, extracting the timestamp to ensure that its accuracy meets the requirements of fault location, extracting the error code for subsequent error classification and analysis, and splicing this information into a secondary instance ID according to a preset format.Specifically, during the peak transaction period of the e-commerce platform, a large number of abnormal logs suddenly appeared in the system, indicating that the service instance OrderService_1234567890AB had an exception with a 500 error code at the 20230101123000000 timestamp. The operation and maintenance team needed to quickly locate the fault to reduce the transaction failure rate and user experience loss. Through the generated secondary instance ID, OrderService_1234567890AB_20230101123000000_500, the system queried the CMDB within 5 seconds and located that the fault occurred on the OrderService service instance, which ran on the physical host PHX-Server-01. The detailed information of the OrderService instance was immediately displayed on the log analysis tool interface, including the module PaymentModule, the system TransactionSystem, and the host PHX-Server-01. Based on this information, the operation and maintenance personnel quickly checked the resources of PHX-Server-01 and the configuration of OrderService, and found that the service crashed due to insufficient resources during the peak period. The resources were expanded immediately and normal service was restored after 10 minutes.

[0039] Example 2 is an embodiment of the present invention, which provides a fault analysis and location method based on log data. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0040] Firstly, an experiment was conducted on the log data of an e-commerce platform. Six different service logs were selected as research objects, including order service logs, user service logs, payment service logs, logistics service logs, promotion service logs and product service logs. The first step of the experiment was to collect the original log data of these services and preprocess them. The preprocessing process included data cleaning, formatting and normalization to eliminate invalid or redundant information and ensure the consistency and accuracy of the data.

[0041] In terms of data collection, we use the built-in log collection function of the microservice framework to collect log data for each service. According to the importance and access frequency of the log data, we use a hot, warm, and cold architecture for lifecycle management. Specifically, critical business logs are stored on SSDs with a retention period of 7 days, ordinary business logs are stored on high-performance disks with a retention period of 30 days, and debug logs are stored on large-capacity disks with a retention period of 1 year.

[0042] Next, the DP-Kmeans clustering algorithm is used to perform pattern recognition on the preprocessed log data, and a multi-dimensional vector of the basic pattern of the abnormal log is established. The key to this step is to calculate the mean of the log features in each cluster to form a multi-dimensional vector representation. The number of clusters is set to 3, representing normal logs, abnormal logs, and unknown logs, respectively. The angular distance between the real-time log and the basic pattern vector of the abnormal category is calculated to identify new abnormal logs.

[0043] Finally, ELK Stack is used as a log analysis tool to compare real-time log data with historical baselines to identify anomalies and locate faults. The threshold for ERROR-level logs is set to 2 times the standard deviation of the baseline average, and the threshold for WARNING-level logs is set to 1.5 times the standard deviation of the baseline average. The threshold is dynamically adjusted according to the seasonal changes in historical data. After abnormal logs are detected, the system will automatically notify the operation and maintenance personnel by email, SMS or Slack message, and provide detailed abnormal information.

[0044] Refer to Table 1 to record and analyze the experimental data.

[0045] Table 1 Experimental data record table

[0046]

[0047] Through the data in the experimental data record table, it can be seen that the content of the present invention has advantages in log data analysis and fault location. First, the preprocessing step effectively reduces the amount of original log data, ranging from 100,000 to 95,000, indicating that the data cleaning and normalization process effectively removes invalid and redundant information and improves the efficiency of subsequent analysis steps. Secondly, the abnormal log detection rate is generally high, reaching 85% to 95%, indicating the effectiveness of the DP-Kmeans clustering algorithm in identifying abnormal log patterns. The high detection rate means that the system can detect potential system problems in a timely manner, thereby reducing the impact of faults on the business. Finally, the fault location accuracy rate reaches 88% to 98%, highlighting the innovation and advantages of the present invention in accurately identifying the fault location. High-accuracy fault location not only speeds up the problem solving speed, but also reduces the waste of resources caused by mislocation. Therefore, our invention is creative.

[0048] Example 3, reference Figure 2 , as an embodiment of the present invention, provides a fault analysis and location system based on log data, including a log collection module, a log identification module, and a fault location module.

[0049] The log collection module is used to collect log data from various microservices and perform preprocessing to provide high-quality raw data for subsequent log pattern recognition and anomaly detection; the log recognition module is used to statistically calculate the average value of specific common features of each category, establish a basic pattern multi-dimensional vector for each log category, compare real-time log data with the basic pattern multi-dimensional vector of each log category, identify abnormal log patterns, and establish a basic pattern multi-dimensional vector of abnormal logs; the fault location module is used to compare real-time log data with historical baselines, identify abnormal patterns and perform fault location.

[0050] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0051] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0052] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0053] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0054] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A fault analysis and location method based on log data, characterized in that: include: Collecting first log data and performing preprocessing; Using the first identification method, identifying the log pattern, and establishing a basic pattern multi-dimensional vector of the abnormal log; Through the first log analysis tool, real-time log data is compared with historical baselines to identify anomalies and locate faults.

2. The fault analysis and location method based on log data according to claim 1, characterized in that: The first log data includes managing the log data according to the life cycle and setting different storage strategies.

3. The fault analysis and location method based on log data according to claim 2, characterized in that: The preprocessing includes collecting log data from microservices and performing preprocessing.

4. The fault analysis and location method based on log data according to claim 3, characterized in that: The identifying log pattern includes identifying features of the preprocessed log data by a first identification method, and establishing a basic pattern multi-dimensional vector for each log category.

5. The fault analysis and location method based on log data according to claim 4, characterized in that: The step of establishing the basic pattern multi-dimensional vector of the abnormal log includes comparing the real-time log data with the basic pattern multi-dimensional vector of each log category to identify the abnormal log.

6. The fault analysis and location method based on log data according to claim 5, characterized in that: The anomaly identification includes collecting learned log classification patterns, matching real-time log data and recording results, generating historical data to form a baseline, comparing real-time data with the historical baseline, and determining whether various types of logs are within a normal fluctuation range. If the log type exceeds the normal fluctuation range, an alarm is issued to alert operation and maintenance personnel to pay attention.

7. The fault analysis and location method based on log data according to claim 6, characterized in that: The fault location includes generating secondary instance ID information based on the exception log, querying the configuration management database to obtain asset information, locating the corresponding service, module and system, judging the log type based on the specific exception pattern matching the exception log, and giving processing suggestions.

8. A system using the fault analysis and location method based on log data as claimed in any one of claims 1 to 7, characterized in that: Including log collection module, log identification module, and fault location module; The log collection module is used to collect log data from each microservice and perform preprocessing to provide high-quality raw data for subsequent log pattern recognition and anomaly detection; The log recognition module is used to statistically calculate the average value of specific common features of each category, establish a basic pattern multi-dimensional vector of each log category, compare the real-time log data with the basic pattern multi-dimensional vector of each log category, identify abnormal log patterns, and establish a basic pattern multi-dimensional vector of abnormal logs; The fault location module is used to compare real-time log data with historical baselines, identify abnormal patterns and perform fault location.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the fault analysis and location method based on log data described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the fault analysis and location method based on log data described in any one of claims 1 to 7 are implemented.

Images