Log processing method and apparatus, and computing device

Abstract

This application relates to the field of big data technology, and in particular provides a log processing method, apparatus, and computing device. The method includes: acquiring multiple target log texts; classifying the multiple target log texts to obtain at least one log set corresponding to each log category, wherein each log set includes multiple target log texts belonging to the same log category; performing early warning analysis on each log set to obtain early warning results for each log set, where the early warning results include clustering anomalies or normal clustering; and outputting the log sets whose early warning results are clustering anomalies. Through log classification and error detection, abnormal logs can be identified promptly and accurately, thereby preventing abnormal logs from participating in log analysis and improving the accuracy of log analysis.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a log processing method, apparatus and computing device. Background Technology

[0002] With the rapid development of information technology, the amount of log data generated by various systems and applications is growing exponentially. This log data contains crucial information such as system operational status, user behavior, and potential problems. However, traditional log analysis methods often rely on manual review or simple rule matching, making it difficult to efficiently and accurately extract valuable information from massive amounts of logs. Especially when system anomalies occur, quickly locating and resolving abnormal logs is crucial to avoid delays or even further system crashes caused by erroneous log analysis. Summary of the Invention

[0003] This application provides a log processing method, apparatus, and computing device that can promptly and accurately detect abnormal logs through log classification and error detection, thereby preventing abnormal logs from participating in log analysis and improving the accuracy of log analysis.

[0004] According to a first aspect of this application, a log processing method is provided, comprising: Retrieve multiple target log texts; Multiple target log texts are classified to obtain at least one log set corresponding to each log category. The log set includes multiple target log texts belonging to the same log category. Perform early warning analysis on each log set to obtain early warning results for each log set. The early warning results include abnormal clustering or normal clustering. The output alert is a log set showing cluster anomalies.

[0005] In conjunction with the first aspect, in some embodiments of the first aspect, multiple target log texts are obtained, including: Connect the log data source to the message queue; The data input component reads the log data stream from the log data source in the message queue and sends the log data stream to the data processing component. The data processing component preprocesses the logs in the log data stream to obtain multiple preprocessed target log texts.

[0006] In conjunction with the first aspect, in some embodiments of the first aspect, the logs in the log data stream are preprocessed to obtain multiple preprocessed target log texts, including: Filter invalid log records from the log data stream to obtain the remaining valid log records; Use regular expressions to identify variable text in each valid log record and replace it with general text to obtain the replacement log text after text replacement for each valid log record. The format of each replaced log text is standardized to obtain multiple preprocessed target log texts.

[0007] In conjunction with the first aspect, in some embodiments of the first aspect, it further includes: Establish the topology of the data input component and the data processing component, configure the data flow relationship between the data input component and the data processing component, and obtain the topology code; Package the topology code into an installable file; The installable file is deployed on the computing device so that the computing device can run the data input components and data processing components corresponding to the installable file's installation topology.

[0008] In conjunction with the first aspect, in some embodiments of the first aspect, packaging the topology code into an installable file includes: In response to the packaging command executed on the topology code, the topology code is compiled to obtain an installable file; Deploying installable files to computing devices includes: Copy the installable file to the computing device so that the computing device can start the Worker process to start the data input component and the data processing component.

[0009] In conjunction with the first aspect, in some embodiments of the first aspect, classifying multiple target log texts to obtain at least one log set corresponding to each log category includes: For two target log texts among multiple target log texts, calculate the similarity between the two target log texts; If the similarity between any two target log texts is greater than or equal to the similarity threshold, then the two target log texts are determined to belong to the same log category, so as to obtain at least one log set corresponding to each log category.

[0010] In conjunction with the first aspect, in some embodiments of the first aspect, calculating the similarity between two target log texts includes: Calculate the similarity between two target log texts based on the Levenshtein distance.

[0011] In conjunction with the first aspect, in certain embodiments of the first aspect, early warning analysis is performed on each log set to obtain early warning results, including: If the number of multiple target log files in any log set exceeds the number threshold, then clustering anomaly is determined as the warning result for the log set; Alternatively, if there are target log files with abnormal similarity in any log set, then clustering anomaly is determined as the warning result for the log set; Alternatively, if an erroneous target log file exists in any log set, then cluster anomaly is determined as the warning result for the log set.

[0012] According to a second aspect of this application, a log processing apparatus is provided, comprising: The log acquisition unit is used to acquire multiple target log texts; The log classification unit is used to classify multiple target log texts to obtain at least one log set corresponding to each log category. The log set includes multiple target log texts belonging to the same log category. The early warning analysis unit is used to perform early warning analysis on each log set and obtain the early warning results for each log set. The early warning results include clustering anomalies or normal clustering. The anomaly output unit is used to output the log set whose early warning result is cluster anomaly.

[0013] According to a third aspect of this application, a communication device is provided, which may include a transceiver unit and a processing unit, wherein the transceiver unit is used to transmit and receive data, and the processing unit is used to execute any of the above-described log processing methods.

[0014] According to a fourth aspect of this application, a computing device is provided, comprising: a memory and a processor; the memory is coupled to the processor; Memory is used to store computer programs; The processor is used to execute computer programs to implement any of the above log processing methods.

[0015] According to a fifth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a controller, implements the steps of any log processing method.

[0016] According to a sixth aspect of this application, a computer product is provided, comprising: a computer program that, when executed by a controller, implements the steps of any log processing method.

[0017] As will be described in detail below, the solution provided in this application, after obtaining multiple target log texts, categorizes these texts into corresponding log sets, achieving the classification and orderly management of a large number of log texts and effectively avoiding interference between different types of logs. Through early warning analysis of similar log texts, clustering results of abnormal or normal clustering are obtained, achieving fine-grained anomaly detection and accurately locating log sets where anomalies occur. The solution outputs early warning results for log sets with abnormal clustering, filtering out log sets with normal clustering, improving the response efficiency of log anomaly monitoring, thereby enabling rapid anomaly handling and fault repair for abnormal log sets, and enhancing system security and stability.

[0018] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0019] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 The figure shows a structural example of a log processing system according to an embodiment of this application; Figure 2 The figure shows a flowchart of a log processing method according to an embodiment of this application; Figure 3 The figure shows an example diagram of a computing device including components according to an embodiment of this application; Figure 4 The figure shows a schematic diagram of a log processing device according to an embodiment of this application; Figure 5 The figure shows a hardware block diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0022] The technical solution of this application can be applied to a log processing system. By classifying and analyzing logs and issuing early warnings, abnormal logs can be detected in a timely manner and relevant prompts can be given, thereby avoiding analysis errors caused by log errors and improving the accuracy of system monitoring.

[0023] In related technologies, the diverse nature of log data sources creates certain technical barriers to log monitoring. Different logs are read in different ways, and when log sources are varied, this diversity makes log analysis difficult and hinders the timely detection of problems within the logs. This results in a current lack of real-time log processing capabilities, making it impossible to detect system anomalies promptly.

[0024] like Figure 1 The diagram shown is a structural schematic of a log processing system provided in an embodiment of this application. The log processing system includes one or more terminals 10 and a server 20.

[0025] One or more terminals 10 may generate one or more log texts during operation and synchronize the generated one or more log texts to the server 20 as a log data source.

[0026] Server 20 can receive log data sources from one or more terminals 10 and execute the log processing method of this application embodiment. Specifically, it can perform the following steps: 101. Log Acquisition: Retrieve multiple target log texts; 102. Log Classification: Classify multiple target log texts to obtain at least one log set corresponding to each log category. The log set includes multiple target log texts belonging to the same log category. 103. Early Warning Analysis: Perform early warning analysis on each log set to obtain the early warning results for each log set. The early warning results include abnormal clustering or normal clustering. 104. Warning Prompt: Outputs a log set with clustering anomalies as the warning result.

[0027] like Figure 2 The diagram shown is a flowchart of a log processing method provided in an embodiment of this application. The log processing method includes the following steps: S201. Obtain multiple target log texts.

[0028] Optionally, log text can refer to a detailed log record. This could record critical system operations (such as API (Application Programming Interface) calls, database accesses, network requests, etc.) and associate them with user or device information. A logging framework (SLF4J (Simple Logging Facade for Java)) can be used to record detailed information such as operation time, user ID, and IP (Internet Protocol) address. Log recording can also be used to record abnormal behavior in the monitoring system (such as unauthorized API calls or data access) and document relevant evidence.

[0029] Logging can also record detailed information about all external API calls (such as request parameters, response results, and caller information). Alternatively, it can add access control to sensitive APIs and log unauthorized call attempts.

[0030] Log recording can also refer to the recording of database auditing functions, documenting all CRUD operations. Log recording can also include user information; linking logs with user information facilitates the tracking of infringement activities.

[0031] S202. Classify multiple target log texts to obtain at least one log set corresponding to each log category. The log set includes multiple target log texts belonging to the same log category.

[0032] Alternatively, considering the potentially large volume of log text, clustering algorithms can be optimized to improve processing speed and efficiency. For example, using distributed computing frameworks (such as Apache Spark, a unified analytics engine) for parallel processing can reduce computation time; data structures can be optimized, such as using more efficient data indexing and caching mechanisms, to improve data access speed.

[0033] Log data may exhibit class imbalance, meaning that some categories have significantly more log entries than others. Oversampling or undersampling techniques can be used to balance the data distribution and improve the robustness of clustering algorithms. For example, oversampling can be performed on log entries from a minority of categories to increase their sample size, while undersampling can be performed on log entries from the majority of categories to reduce their sample size.

[0034] The characteristics and distribution of log data may change dynamically over time, requiring the design of clustering algorithms that can adapt to these changes. For example, introducing online learning mechanisms can enable the algorithm to update the clustering model in real time to adapt to new data features; periodically retraining the clustering model ensures that it always has good clustering performance.

[0035] When training clustering models, avoid overfitting. Evaluate the model's generalization ability using methods such as cross-validation; choose an appropriate model complexity to balance fit and generalization ability. For example, using K-fold cross-validation, divide the dataset into K subsets, alternately using one subset as the validation set and the others as the training set to evaluate the model's performance.

[0036] Log clustering allows for the rapid classification of massive amounts of log data, helping operations and maintenance personnel quickly locate problems. For example, when a system generates a large number of alarms, log clustering can group similar error logs into one category, enabling operations and maintenance personnel to quickly identify key issues and reduce troubleshooting time.

[0037] Log clustering can monitor system logs in real time and promptly detect abnormal logs. For example, by analyzing the results of log clustering, abnormal behavior in the system can be identified, providing early warnings and enabling measures to be taken to prevent system failures.

[0038] Log clustering can be used for security monitoring to promptly identify potential security threats. For example, by analyzing abnormal behavior in logs, security incidents such as malicious attacks and data breaches can be identified, allowing for the implementation of appropriate security measures.

[0039] Log clustering can help enterprises conduct compliance checks, ensuring that system logs comply with relevant laws, regulations, and industry standards. For example, cluster analysis can identify violations in logs, allowing for timely rectification.

[0040] S203. Perform early warning analysis on each log set to obtain early warning results for each log set. The early warning results include abnormal clustering or normal clustering.

[0041] Optionally, early warning analysis can be performed on each log set according to a preset early warning frequency to obtain early warning results for each log set. The early warning results include clustering anomalies or clustering normality.

[0042] Alternatively, real-time alert analysis can be performed on each log set to obtain alert results for each log set, including whether clustering is abnormal or normal.

[0043] Optionally, clustering parameters (such as the number of log categories) can be automatically adjusted using machine learning models to optimize clustering performance.

[0044] S204. Output the log set whose early warning result is cluster anomaly.

[0045] In this embodiment, after acquiring multiple target log texts, these texts are categorized and grouped into corresponding log sets, achieving the classification and orderly management of a large number of log texts and effectively avoiding interference between different types of logs. Through early warning analysis of similar log texts, clustering results of abnormal or normal clustering are obtained, achieving fine-grained anomaly detection and accurately locating log sets with anomalies. The system outputs early warning results for log sets with abnormal clustering, filtering out log sets with normal clustering, improving the response efficiency of log anomaly monitoring. This allows for rapid anomaly handling and fault repair of abnormal log sets, enhancing system security and stability.

[0046] The target log text can be obtained from a log data source. As an example, multiple target log texts can be obtained, including: A1. Connect at least one log data source to the message queue.

[0047] Optionally, log data sources can originate from different application scenarios, such as financial risk monitoring and medical data analysis. For example, in the financial field, abnormal transaction behavior can be identified by clustering transaction logs; in the medical field, disease patterns and treatment effects can be discovered by clustering patient medical record logs.

[0048] Optionally, if the log text in the log data source contains user personal information, this personal information can be anonymized to prevent information leakage. For example, encryption technology can be used to encrypt user information in the logs to ensure data security during transmission and storage.

[0049] A2. The data input component reads the log data stream from the log data source in the message queue and sends the log data stream to the data processing component.

[0050] Alternatively, the data input component can be a Spout component, which is a log data reading component. The Spout component reads the log data stream from the message queue and transforms it into a Tuple, which is then sent to downstream data processing components (such as Bolt nodes) for processing.

[0051] Optionally, the log data stream can be converted into a tuple. Specifically, the logs in the log data stream can be represented according to various fields defined in the tuple (such as log content, identifier ID, processing status, etc.) to facilitate the subsequent data processing components to quickly parse and extract the log text.

[0052] A3. Using the data processing component, preprocess the logs in the log data stream to obtain multiple preprocessed target log texts.

[0053] Optionally, A3 may include: sending log text from the log data stream to a data processing component, where the data processing component preprocesses the log text to obtain multiple preprocessed target log texts; and receiving multiple target log texts from the data processing component.

[0054] like Figure 3 As shown, the computing device 300 may include a data input component 301 and a data processing component 302. The data input component 301 reads log data streams from a log data source in a message queue and sends the log data streams to the data processing component. The data processing component 302 preprocesses the log data streams to obtain multiple preprocessed target log texts.

[0055] In this embodiment, one or more log data sources are connected to a message queue, which improves the maintainability and scalability of the entire log processing system. A dedicated data input component reads log data streams from the message queue and transmits them to the data processing component, standardizing and modularizing the log data reading logic and ensuring the stability and continuity of the log data stream transmission. The data processing component then performs preprocessing operations on the log data stream, transforming the raw, messy log data into a unified target log text, removing invalid and redundant information, and standardizing the data format. This provides high-quality, standardized data for subsequent log classification, early warning analysis, anomaly detection, and other business processes. Furthermore, centralizing the preprocessing logic on a dedicated processing node enables unified scheduling and horizontal scaling of processing resources, adapting to scenarios involving large volumes of log text and further improving the processing efficiency and compatibility of the entire log processing chain.

[0056] In one possible design, A3 preprocesses the log data stream to obtain multiple preprocessed target log texts, including: Filter invalid log records from the log data stream to obtain the remaining valid log records; Use regular expressions to identify variable text in each valid log record and replace it with general text to obtain the replacement log text after text replacement for each valid log record. The format of each replaced log text is standardized to obtain multiple preprocessed target log texts.

[0057] Optionally, invalid log records can refer to logs that do not conform to preset rules. Logs that do not conform to the preset rules can be directly removed from the received log data stream according to preset rules (such as log type, collection source, key fields, etc.).

[0058] Regular expressions can be pre-built tools for text matching. They can be used to identify dynamically changing, non-common variables in logs (such as timestamps, IP addresses, user IDs, device numbers, request links, random identifiers, etc., which can cause different text formats in similar logs and interfere with feature identification). All identified variables can then be uniformly replaced with common markers (e.g., {TIME} for all timestamps, {IP} for all IP addresses, and {ID} for all identifier-related content).

[0059] Optionally, standardizing the format of the replaced log texts can include: removing redundant formatting symbols from the text, such as removing extra spaces, tabs, line breaks, and special delimiters. Secondly, standardizing the log text format, such as standardizing field order, symbol usage, and text line style.

[0060] In this embodiment, invalid log records in the log data stream are first filtered out, directly eliminating redundant and invalid interference data, significantly reducing the amount of data and computational overhead for subsequent log processing, reducing the impact of noise on subsequent classification and early warning analysis, and improving the overall efficiency of log processing. Then, variable text in each valid log record is accurately identified using regular expressions and replaced with general text, preserving the core semantics and common features of the logs, laying a unified text foundation for the clustering and classification of similar logs in the future. Finally, all replaced log texts are formatted uniformly, standardizing logs with different structures and expressions into standardized target log text, achieving consistency and standardization of log data formats, eliminating processing barriers caused by format differences, and providing high-quality, highly consistent standardized data input for downstream log classification, early warning analysis, and other links, significantly improving the accuracy, adaptability, and efficiency of log processing, and ensuring the stable and reliable operation of the overall log analysis chain.

[0061] Another possible design also includes: Establish the topology of the data input component and the data processing component, configure the data flow relationship between the data input component and the data processing component, and obtain the topology code; Package the topology code into an installable file; The installable file is deployed on the computing device so that the computing device can run the data input components and data processing components corresponding to the installable file's installation topology.

[0062] Alternatively, a topology creation tool (such as TopologyBuilder) can be used to build the topology of the data input component and the data processing component.

[0063] The data flow relationship between the data input component and the data processing component can refer to the data transmission relationship between the data input component and the data processing component. For example, the data component sends data from the data processing component to the data processing component according to preset data format, quantity, data transmission protocol and other parameters.

[0064] Topology code refers to computer code that establishes the topology and data flow relationships of configuration data input components and data processing components. It can be generated by a third-party code generator. For example, the description text of the topology of the data components and data processing components, as well as the description text corresponding to the data flow relationships, can be input into a third-party code generator to obtain the topology code output by the third-party code generator.

[0065] Optionally, establishing the topology of the data input component and the data processing component, and configuring the data flow relationship between the data input component and the data processing component to obtain the topology code, may include: creating a new work project using a topology creation tool; defining a data input component for reading log data sources; and defining a data processing component for processing log data. Configuring the data flow relationship between the data input component and the data processing component determines how data is transferred between the components to obtain the corresponding topology code.

[0066] After completing the topology code setup, you can use the Maven (Project Object Model) command `mvn clean package` in the root directory of the new working project to compile and package the project. The Maven command will instruct the generation of an executable file, such as a Java Archive (JAR) file, based on the configuration in the `pom.xml` (Project Object Model configuration file).

[0067] An executable file can be sent to a computing device, enabling the device to run the data input and data processing components corresponding to the installation topology. This computing device could be, for example, a Nimbus (master node) node in a Storm cluster.

[0068] The computing device detects the topology submitted by the user via command-line tools and then allocates tasks to the Supervisor (master) node based on cluster resources and configuration. The Supervisor node starts Worker processes to execute tasks from the data input and data processing components. For example, the data input component reads data from the log data source and emits the data into the topology. The data processing component receives and processes the data from the data input component and uses the Storm UI (User Interface) to monitor the topology's running status and performance metrics. The Storm UI can run on the Nimbus node.

[0069] The topology creation tool provides a flexible way to build topology structures, allowing developers to define data input and processing components, as well as the data flow relationships between them, according to actual needs. Topologies built with Storm can read and process log data in real time and respond promptly to system events. Data processing components can implement various complex processing logics for log data, such as filtering, transformation, and aggregation, to meet the needs of real-time monitoring and analysis. Storm provides fault tolerance mechanisms to ensure that the topology can automatically redistribute tasks and recover its state in the event of node failure, thereby guaranteeing the continuity and reliability of log data processing. The Nimbus and Supervisor architecture makes Storm clusters highly available, capable of handling abnormal situations such as node failures and network outages. The Storm UI provides an intuitive interface to monitor the topology's running status and performance metrics, such as throughput and latency, helping developers and operations personnel understand the system's health status in real time. The Storm UI allows for convenient topology management operations, such as starting, pausing, and stopping. Using Maven for project management and building simplifies the development process and improves code maintainability and reusability. The JAR file generated by Maven contains all the dependencies and configurations for the topology, making deployment to the cluster simple and quick. Storm, as open-source software, reduces the cost of log data processing solutions. Its high resource utilization and elastic scalability allow enterprises to dynamically adjust the cluster size according to actual needs, thereby optimizing operating costs.

[0070] Optionally, a unique identifier (dedicated version number) can be embedded in the topology code to facilitate the identification of infringement.

[0071] For example, in topology code, all dependencies and their license information can be explicitly declared in the `pom.xml` (Maven). Tools such as the License Maven Plugin can be used to check dependency compliance and prevent infringement. Tools such as Black Duck and FOSSA analysis tools can also be used to scan public open-source code repositories (such as GitHub and Gitee) to ensure there is no unauthorized open-source code. Furthermore, an infringement notification mechanism can be embedded in the topology code to remind users to comply with patent protection regulations.

[0072] In this embodiment, by establishing a topology structure for data input and data processing components and configuring dedicated data flow relationships to generate topology code, the linkage logic between components can be customized according to the actual business needs of log processing, accurately defining the data flow path from reading to processing. Under the trigger of the first user, the topology code is packaged into an installable file, achieving integrated encapsulation of log installation files. Under the trigger of the second user, the installable file is standardized and deployed on computing devices. Running the installable file automatically completes the installation of the corresponding components in the topology structure and the configuration of flow relationships, significantly simplifying the deployment process, reducing technical maintenance costs, and enabling rapid deployment and batch deployment of the log processing architecture. It also adapts to the clustered deployment needs of multiple computing devices, further enhancing the flexible deployment of log processing.

[0073] Optionally, the topology code is packaged into an installable file, including: In response to the first user's packaging command triggered by the topology code, the topology code is compiled to obtain an installable file; Deploying installable files to computing devices includes: In response to an installation operation triggered by a second user, the installable file is copied to the computing device so that the computing device starts the Worker process to start the data input component and the data processing component.

[0074] In this embodiment, the packaging command triggered by the first user drives the compilation of topology code to generate installable files. The topology packaging operation is bound to user instructions, enabling on-demand triggering and automated execution of the packaging process. This simplifies the conversion process from topology code to deployable files and improves packaging efficiency. The installation operation triggered by the second user executes the copying and deployment of the installable files on the computing device. The deployment action is user-triggered, adapting to the flexible needs of different deployment times and computing devices. The operation is simple and easy to execute, lowering the deployment threshold for non-professionals. During deployment, the computing device automatically starts a Worker process and loads and starts the data input and data processing components. This automates and manages the component startup process, ensuring that the data input and data processing components start stably and operate collaboratively according to the preset topology structure and flow relationship. Simultaneously, the Worker process provides a dedicated execution environment for component operation, ensuring a smooth flow of the log processing chain from data reading to preprocessing. Overall, this improves the convenience, controllability, and reliability of topology architecture deployment and component operation, reducing the manual costs of system deployment and maintenance.

[0075] As one example, multiple target log texts are classified to obtain log sets corresponding to at least one log category, including: For two target log texts among multiple target log texts, calculate the similarity between the two target log texts; If the similarity between any two target log texts is greater than or equal to the similarity threshold, then the two target log texts are determined to belong to the same log category, so as to obtain at least one log set corresponding to each log category.

[0076] For multiple target log files, two target log files can be selected as a group, and the similarity between the two target log files can be calculated. Any two target log files can be enumerated from the multiple target log files.

[0077] Optionally, after obtaining new target log text, the clustering results can be updated in real time, or the clustering can be re-clustered.

[0078] Optionally, the log sets corresponding to at least one log category can be stored in the database.

[0079] Specifically, the log sets corresponding to at least one log category can be stored in the database, which may include: Create a table in the database to store clustering results, containing cluster ID, log content, timestamp, and similarity score. After log clustering is complete, write the clustering results to the database via batch insertion. For each cluster, generate a unique cluster ID and associate this ID with the target log text within the cluster. Retrieve the latest clustering results from the database in real time and perform real-time analysis. Trigger an alarm when the size of a cluster exceeds a preset threshold, the similarity score is abnormally high, or the cluster contains specific error logs. Also trigger an alarm when abnormal clusters are detected.

[0080] By storing the clustered log data in a database, persistent data storage is ensured. This provides users with the ability to query and analyze historical data, supporting long-term monitoring of system status and investigation of problems. Tables created in the database to store clustering results manage the clustering data in a structured manner, including key information such as cluster ID, log content, timestamps, and similarity scores. This structured storage facilitates data retrieval, analysis, and report generation. Batch insertion of clustering results into the database improves data writing efficiency. Simultaneously, real-time retrieval of the latest clustering results from the database for analysis ensures the real-time performance and responsiveness of the monitoring system. Real-time analysis of the clustering results retrieved from the database can quickly identify abnormal clusters, such as clusters exceeding preset thresholds, abnormally high similarity scores, or clusters containing specific error logs. This rapid identification mechanism helps to promptly discover potential problems or faults in the system. When abnormal clusters are detected and alarms are triggered, the system can promptly send alarm information to administrators or relevant personnel, reminding them to pay attention and take appropriate measures. This helps prevent potential faults from occurring or mitigate their impact on the system. This method enhances system stability and reliability through real-time monitoring and timely alerts. Administrators can respond to and handle system anomalies more quickly, thereby reducing system downtime and improving user satisfaction.

[0081] Optionally, a similarity threshold can be predetermined. Specifically, determining the similarity threshold includes: From the existing log data, a representative sample set is randomly selected, containing logs of various types. The logs in the sample set are preprocessed, and the preprocessed log samples are segmented into sequences of words or phrases. The Levenshtein distance algorithm is used to calculate the similarity between each pair of logs to generate a similarity matrix, where each element represents the similarity between the corresponding log pairs. Statistical analysis is performed on the calculated similarity data to identify the similarity distribution range. Based on the similarity distribution range and business requirements, a similarity threshold is set. The logs include normal operation logs, abnormal logs, and warning logs.

[0082] By randomly selecting a representative sample set from existing log data, this process ensures that the selected samples comprehensively reflect the diversity and complexity of the log data. This representativeness helps to more accurately determine the similarity threshold, thereby improving the accuracy of log clustering. Preprocessing and word segmentation of the logs in the sample set removes noise and irrelevant information, transforming the logs into a standardized format. This helps to more accurately calculate the similarity between logs and provides a reliable data foundation for subsequent threshold setting. By calculating the similarity between each pair of logs and generating a similarity matrix, a comprehensive view of the similarity between logs can be obtained. This allows analysts to gain a deeper understanding of the correlation between logs, providing strong support for setting reasonable similarity thresholds. Statistical analysis of the calculated similarity data identifies the similarity distribution interval, which helps to set similarity thresholds more scientifically. This data-driven decision-making method reduces the influence of subjective judgment and improves the objectivity and accuracy of threshold setting. Setting similarity thresholds based on similarity distribution intervals and business needs ensures that the threshold settings are closely aligned with actual application scenarios. This flexibility allows the method to adapt to different business needs and log data characteristics, improving the practicality and effectiveness of log clustering. Given that logs include different types such as normal operation logs, exception logs, and warning logs, setting an appropriate similarity threshold helps to more accurately distinguish these types. This is crucial for timely detection and handling of anomalies or warnings, and can improve the efficiency of system monitoring and troubleshooting.

[0083] Understandably, the calculated similarity between two target log texts is compared to a similarity threshold. If the similarity between two logs is greater than the threshold, they are considered similar. When two logs are found to be similar, a list of clusters is maintained in memory, with each cluster containing a group of similar logs, to group them into the same cluster. If a new log is similar to a log in an existing cluster, the new log is added to that cluster. If a new log is not similar to any log in any existing cluster, a new cluster is created, and the new log is the first member of the new cluster.

[0084] This method achieves automatic log clustering by automatically comparing the similarity between logs and grouping logs with similarity scores above a threshold into the same category. This significantly reduces the workload of manual log review and classification, improving the automation level of log processing. Using a similarity threshold as the judgment criterion can accurately identify logs with similar or related content. This helps to quickly discover recurring problems or patterns in large amounts of log data, providing strong support for problem localization and resolution. As new logs continuously arrive, the method can update the clustering results in real time. If a new log is similar to a log in an existing cluster, it will be automatically added to the corresponding cluster; if it is dissimilar, a new cluster will be created. This dynamic updating mechanism ensures the timeliness and accuracy of the clustering results. By grouping similar logs into the same category, this method avoids the repeated storage and processing of the same or similar log data. This helps optimize the use of storage resources, reduce data processing costs, and improve the overall operating efficiency of the system. Through cluster analysis of similar logs, abnormal behavior or potential faults in the system can be more easily identified. When the number of abnormal logs in a certain cluster suddenly increases, it may mean that a problem has occurred in the system, thus triggering timely fault warnings and troubleshooting processes. Clustered log data provides a richer perspective for system monitoring. By monitoring the quantity and trends of logs in each cluster in real time, a more comprehensive understanding of the system's operational status and performance can be achieved.

[0085] In this embodiment, the similarity calculation method between two target log texts is used to obtain the content association degree between every two target log texts in multiple target log files, and to fully explore the inherent semantic features of log texts. By comparing the similarity with a preset similarity threshold, similar log texts can be quickly classified into the same log category and form a corresponding log set, thereby achieving efficient log classification.

[0086] Optionally, the similarity between two target log texts is calculated, including: Calculate the similarity between two target log texts based on the Levenshtein distance.

[0087] Optionally, calculating the similarity between two target log texts based on the Levenshtein distance may include: segmenting the two target log texts into words to obtain the word segments corresponding to the two target log texts respectively, calculating the Levenshtein distance between the word segments corresponding to the two target log texts respectively, and using the calculated Levenshtein distance as the similarity between the two target log texts.

[0088] Optionally, logs can be segmented using Natural Language Processing (NLP). For example, NLP techniques can be used to perform semantic analysis on the log text to obtain the segmented sequence of the target log text. Using the segmented sequence of the target log text can improve the accuracy and interpretability of clustering.

[0089] Optionally, calculating the similarity between two target log texts based on the Levenshtein distance may also include: Create a function to calculate the Levenshtein distance between two strings. The function should take two strings as input and return the distance between the two strings. Create a matrix dp of size (len(A) + 1) × (len(B) + 1), and initialize the first row and the first column; Use nested loops to traverse each element of the matrix and calculate the value of each dp[i][j], where dp[i][j] represents the Levenshtein distance between the first i characters of string A and the first j characters of string B, to obtain dp[len(A)][len(B)], where dp[len(A)][len(B)] contains the Levenshtein distance between the two strings.

[0090] Understandably, Levenshtein distance, also known as edit distance, is an effective method for measuring the differences between two strings. It accurately calculates the minimum number of editing operations (inserting, deleting, or replacing a character) required to transform one string into another. In practical applications, log records may vary slightly due to factors such as system configuration, version updates, or user behavior. Using Levenshtein distance can effectively handle these subtle variations in text, thus more accurately identifying similar or related log entries. In log clustering, using Levenshtein distance as a similarity metric can more accurately group similar logs together.

[0091] Calculating the Levenshtein distance between logs makes it easier to identify anomalous logs that significantly differ from regular log patterns. This is crucial for timely detection of system anomalies and prevention of potential failures. The Levenshtein distance algorithm can be easily extended to handle large-scale datasets, and its processing efficiency can be improved through techniques such as parallel computing. Alternatively, in addition to using Levenshtein distance to calculate the similarity between two target log texts, other analytical methods, such as time series analysis and user behavior analysis, can be used to gain a more comprehensive understanding of the log data. For example, by combining timestamp information, the distribution and trends of logs in different time periods can be analyzed to identify potential periodic issues.

[0092] In this embodiment, the Levenshtein distance quantifies text differences by statistically analyzing the minimum number of single-character editing operations required to transform one text into another. This accurately captures text similarity changes caused by subtle differences in dependent variable fields, character additions or subtractions, and character errors in log text, thereby improving the accuracy and robustness of log text similarity calculation.

[0093] As an example, early warning analysis is performed on each log set to obtain early warning results, including: If the number of multiple target log files in any log set exceeds the number threshold, then clustering anomaly is determined as the warning result for the log set; Alternatively, if there are target log files with abnormal similarity in any log set, then clustering anomaly is determined as the warning result for the log set; Alternatively, if an erroneous target log file exists in any log set, then cluster anomaly is determined as the warning result for the log set.

[0094] In this embodiment, using a threshold exceeding the number of target log texts within the log set as an anomaly detection criterion enables rapid identification of large-scale anomalies caused by sudden surges in logs or the generation of numerous duplicate logs, and timely captures issues such as sudden changes in system load and log collection anomalies. Alternatively, including target log texts with abnormal similarity within the log set in the detection scope can accurately identify outlier logs with significant deviations, effectively discovering hidden anomalies at the log content level. Alternatively, using erroneous target log texts within the log set as another detection dimension directly identifies anomalies caused by erroneous logs, achieving full coverage detection of both explicit and implicit anomalies. This significantly improves the comprehensiveness and accuracy of log set anomaly identification, enables rapid output of early warning results, effectively enhances the response speed and fault warning capabilities of system log monitoring, and provides reliable log-level protection for stable system operation.

[0095] like Figure 4 The diagram shown is a flowchart of a log processing apparatus provided in an embodiment of this application. The log processing apparatus 400 may include the following units: Log acquisition unit 401 is used to acquire multiple target log texts.

[0096] Log classification unit 402 is used to classify multiple target log texts to obtain at least one log set corresponding to each log category, wherein the log set includes multiple target log texts belonging to the same log category.

[0097] The early warning analysis unit 403 is used to perform early warning analysis on each log set and obtain the early warning results of each log set. The early warning results include clustering anomalies or clustering normality.

[0098] The abnormal output unit 404 is used to output the log set whose early warning result is clustering anomaly.

[0099] As one embodiment, the log acquisition unit 401 may include: The log access module is used to connect at least one log data source to the message queue.

[0100] The log reading module is used to read log data streams from the log data source in the message queue through the data input component and send the log data streams to the data processing component.

[0101] The log processing module is used to preprocess the logs in the log data stream through the data processing component to obtain multiple preprocessed target log texts.

[0102] As another example, the log processing module is specifically used for: Filter invalid log records from the log data stream to obtain the remaining valid log records. Use regular expressions to identify variable text in each valid log record and replace it with general text to obtain the replacement log text after text replacement for each valid log record. Standardize the format of each replacement log text to obtain multiple preprocessed target log texts.

[0103] As yet another embodiment, it also includes: The topology building unit is used to establish the topology structure of the data input component and the data processing component, configure the data flow relationship between the data input component and the data processing component, and obtain the topology code; The file packaging unit is used to package the topology code into an installable file; The file installation unit is used to deploy installable files into a computing device so that the computing device can run the data input component and data processing component corresponding to the installable file installation topology.

[0104] As another embodiment, the file packaging unit is specifically used for: In response to the first user's packaging command triggered by the topology code, the topology code is compiled to obtain an installable file.

[0105] The file installation unit is specifically used for: In response to an installation operation triggered by a second user, the installable file is copied to the computing device, causing the computing device to start a Worker process to initiate the data input and data processing components.

[0106] As another embodiment, the log classification unit includes: The similarity calculation module is used to calculate the similarity between two target log texts from multiple target log texts.

[0107] The log classification module is used to determine that two target log texts belong to the same log category if the similarity between any two target log texts is greater than or equal to a similarity threshold, so as to obtain at least one log set corresponding to each log category.

[0108] As another embodiment, the similarity calculation module is specifically used to calculate the similarity between two target log texts based on the Levenshtein distance.

[0109] As another embodiment, the early warning analysis unit is specifically used for: If the number of multiple target log files in any log set exceeds the number threshold, then clustering anomaly is determined as the warning result for the log set; Alternatively, if there are target log files with abnormal similarity in any log set, then clustering anomaly is determined as the warning result for the log set; Alternatively, if an erroneous target log file exists in any log set, then cluster anomaly is determined as the warning result for the log set.

[0110] Figure 5 This application provides a hardware block diagram of a computing device. The computing device 500 according to this application includes at least a memory 501, a processor 502, and a transceiver 503. The memory 501 stores a computer program, the processor 502 executes the computer program to implement the log processing method of any of the above embodiments, and the transceiver 503 receives or sends signal requests / messages / information.

[0111] In addition, the memory 501, processor 502, and transceiver 503 are all electrically connected to the bus 504.

[0112] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program. When executed by a processor, the computer program implements the log processing method of any of the preceding embodiments of this application. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0113] This application further provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the log processing method of any of the preceding embodiments of this application.

[0114] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0115] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0116] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0117] It should also be noted that in the system and method of this application, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of this application.

[0118] Various changes, substitutions, and modifications can be made to the technology herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0119] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0120] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A log processing method characterized by, include: Retrieve multiple target log texts; The multiple target log texts are classified to obtain at least one log set corresponding to each log category, and the log set includes multiple target log texts belonging to the same log category. Early warning analysis is performed on each log set to obtain early warning results for each log set, including abnormal clustering or normal clustering. The output alert is a log set showing cluster anomalies.

2. The method of claim 1, wherein, The acquisition of multiple target log texts includes: Connect at least one log data source to the message queue; The data input component reads the log data stream from the log data source from the message queue and sends the log data stream to the data processing component. The data processing component preprocesses the logs in the log data stream to obtain the preprocessed target log texts.

3. The method of claim 2, wherein, The step of preprocessing the logs in the log data stream to obtain the preprocessed target log texts includes: Filter out invalid log records in the log data stream to obtain the remaining valid log records; Use regular expressions to identify variable text in each valid log record and replace it with general text to obtain the replacement log text after text replacement for each valid log record. The format of each replaced log text is standardized to obtain multiple preprocessed target log texts.

4. The method of claim 2, wherein, Also includes: Establish the topology of the data input component and the data processing component, configure the data flow relationship between the data input component and the data processing component, and obtain the topology code; Package the topology code into an installable file; The installable file is deployed on a computing device so that the computing device runs the installable file to install the data input component and the data processing component corresponding to the topology.

5. The method of claim 4, wherein, The step of packaging the topology code into an installable file includes: In response to a packaging command triggered by the first user on the topology code, the topology code is compiled to obtain the installable file; Deploying the installable file on a computing device includes: In response to an installation operation triggered by a second user, the installable file is copied to the computing device, causing the computing device to start a worker process to initiate the data input component and the data processing component.

6. The method of claim 1, wherein, The step of classifying the multiple target log texts to obtain log sets corresponding to at least one log category includes: For two target log texts among the plurality of target log texts, calculate the similarity between the two target log texts; If the similarity between any two target log texts is greater than or equal to the similarity threshold, then the two target log texts are determined to belong to the same log category, so as to obtain at least one log set corresponding to each log category.

7. The method of claim 6, wherein, The calculation of the similarity between two target log texts includes: The similarity between two target log texts is calculated based on the Levenshtein distance.

8. The method according to claim 1, characterized in that, The process of performing early warning analysis on each log set to obtain early warning results includes: If the number of multiple target log files in any log set exceeds the number threshold, then clustering anomaly is determined as the warning result for the log set; Alternatively, if there are target log files with abnormal similarity in any log set, then clustering anomaly is determined as the warning result for the log set; Alternatively, if an erroneous target log file exists in any log set, then cluster anomaly is determined as the warning result for the log set.

9. A communications device, characterized by The communication device includes a transceiver unit and a processing unit, wherein the transceiver unit is used to send and receive data, and the processing unit is used to execute the log processing method described in any one of 1-8 above.

10. A computing device, comprising: Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the log processing method as described in any one of claims 1-7.

Images