Log monitoring method and device, equipment, medium and program product

Abstract

The invention provides a log monitoring method which can be applied to the technical field of artificial intelligence and the technical field of distribution. The method comprises the following steps: based on a stream processing architecture, processing log data acquired in real time in a distributed manner through micro-service; wherein the micro-service comprises a preprocessing micro-service, a feature extraction micro-service and a real-time monitoring micro-service. Standardizing the log data through the preprocessing micro-service to obtain standard semantic data; processing the standard semantic data based on a deep semantic extraction model through a feature extraction micro-service to obtain deep semantic feature data; wherein the deep semantic extraction model is constructed based on an attention mechanism model, an auto-encoder and a graph neural network; and monitoring the dynamic change of the log data in real time based on an online learning mechanism by monitoring the micro-service in real time, and positioning the abnormal log based on the deep semantic feature data. The invention further provides a log monitoring device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present application relates to the field of distributed technology, specifically to the field of artificial intelligence technology, and more specifically to a log monitoring method, apparatus, device, medium, and program product. Background Art

[0002] Log analysis can be used to monitor system status, locate faults, and optimize performance to ensure stable system operation. In log analysis, preprocessing, feature extraction, and real-time monitoring and early warning are very important steps. Preprocessing improves log quality through operations such as cleaning and filtering. Feature extraction mines key information from logs for analysis. Real-time monitoring and early warning dynamically monitor the system, detect anomalies in a timely manner, and trigger responses.

[0003] However, existing log analysis relies on specific log formats and fails to provide sufficient and in-depth semantic analysis, which limits the depth of fault analysis and restricts log processing capabilities. Summary of the Invention

[0004] In view of the above problems, the present application provides a log monitoring method, apparatus, device, medium and program product that improve log processing capabilities.

[0005] According to the first aspect of the present application, a log monitoring method is provided, comprising: based on a stream processing architecture, distributed processing of log data acquired in real time is performed through microservices; wherein the microservices include a preprocessing microservice, a feature extraction microservice, and a real-time monitoring microservice; through the preprocessing microservice, the log data is standardized to obtain standard semantic data; through the feature extraction microservice, the standard semantic data is processed based on a deep semantic extraction model to obtain deep semantic feature data; wherein the deep semantic extraction model is constructed based on an attention mechanism model, an autoencoder, and a graph neural network; through the real-time monitoring microservice, the dynamic changes of the log data are monitored in real time based on an online learning mechanism, and abnormal logs are located based on the deep semantic feature data and processed based on the deep semantic extraction model.

[0006] According to an embodiment of the present application, the standardized processing of the log data to obtain standard semantic data includes: preprocessing the log data based on a pre-trained deep neural network; wherein the preprocessing includes data cleaning and / or formatting; and performing semantic standardization processing on the preprocessed log data to obtain the standard semantic data.

[0007] According to an embodiment of the present application, the semantic standardization of the preprocessed log data to obtain the standard semantic data includes: semantically standardizing the preprocessed log data through natural language processing technology to obtain the standard semantic data; wherein, the semantic standardization includes one or more of stem extraction, lemma restoration, synonym replacement and template matching.

[0008] According to an embodiment of the present application, the standard semantic data is processed based on the deep semantic extraction model to obtain deep semantic feature data, including: generating lexical semantic vectors and lexical attention weights through the attention mechanism model according to the standard semantic data; compressing the lexical semantic vectors through the autoencoder to generate deep coding features; extracting graph structure features based on the graph neural network according to the standard semantic data; and fusing the lexical semantic vectors, the lexical attention weights, the deep coding features and the graph structure features to obtain the deep semantic feature data.

[0009] According to an embodiment of the present application, generating a lexical semantic vector and a lexical attention weight based on the standard semantic data through the attention mechanism model includes: processing the standard semantic data through the attention mechanism model to generate the lexical semantic vector; wherein the lexical semantic vector contains lexical context information; and identifying key information of the standard semantic data based on the attention mechanism model to generate the lexical attention weight.

[0010] According to an embodiment of the present application, the real-time monitoring of the dynamic changes of the log data based on the online learning mechanism includes: when the log pattern of the log data is updated, based on the online learning mechanism, dynamically adjusting the feature extraction strategy of the deep semantic extraction model according to the updated log data to update the deep semantic feature data.

[0011] According to an embodiment of the present application, locating the abnormal log based on the deep semantic feature data includes: storing the deep semantic feature data in a database and establishing an index corresponding to the deep semantic feature data; and retrieving abnormal features based on the index corresponding to the deep semantic feature data to locate the abnormal log.

[0012] The second aspect of the present application provides a log monitoring device, including: a distributed processing module, which is used to distribute the log data obtained in real time through microservices based on a stream processing architecture; wherein the microservices include a preprocessing microservice, a feature extraction microservice and a real-time monitoring microservice; a preprocessing module, which is used to standardize the log data through the preprocessing microservice to obtain standard semantic data; a feature extraction module, which is used to process the standard semantic data based on a deep semantic extraction model through the feature extraction microservice to obtain deep semantic feature data; wherein the deep semantic extraction model is constructed based on an attention mechanism model, an autoencoder and a graph neural network; and a real-time monitoring module, which is used to monitor the dynamic changes of the log data in real time based on an online learning mechanism through the real-time monitoring microservice, and locate abnormal logs based on the deep semantic feature data for processing based on the deep semantic extraction model.

[0013] According to an embodiment of the present application, the preprocessing module includes: a preprocessing unit, which is used to preprocess the log data based on a pre-trained deep neural network; wherein the preprocessing includes data cleaning and / or formatting; and a semantic standardization unit, which is used to perform semantic standardization on the preprocessed log data to obtain the standard semantic data.

[0014] According to an embodiment of the present application, the semantic standardization unit includes: a natural language processing sub-unit, which is used to semantically standardize the preprocessed log data through natural language processing technology to obtain the standard semantic data; wherein, the semantic standardization includes one or more of stem extraction, morphological restoration, synonym replacement and template matching.

[0015] According to an embodiment of the present application, the feature extraction module includes: an attention mechanism unit, which is used to generate a lexical semantic vector and a lexical attention weight through the attention mechanism model according to the standard semantic data; an autoencoder unit, which is used to compress the lexical semantic vector through the autoencoder to generate deep coding features; a graph neural network unit, which is used to extract graph structure features based on the graph neural network according to the standard semantic data; and a fusion unit, which is used to fuse the lexical semantic vector, the lexical attention weight, the deep coding features and the graph structure features to obtain the deep semantic feature data.

[0016] According to an embodiment of the present application, the attention mechanism unit includes: a semantic vector sub-unit, used to process the standard semantic data through the attention mechanism model to generate the lexical semantic vector; wherein the lexical semantic vector contains lexical context information; and an attention weight sub-unit, used to identify the key information of the standard semantic data based on the attention mechanism of the attention mechanism model and generate the lexical attention weight.

[0017] According to an embodiment of the present application, the real-time monitoring module includes: an online learning unit, which is used to dynamically adjust the feature extraction strategy of the deep semantic extraction model based on the online learning mechanism and the updated log data when the log pattern of the log data is updated, so as to update the deep semantic feature data.

[0018] According to an embodiment of the present application, the real-time monitoring module also includes: an abnormality locating unit, which is used to store the deep semantic feature data in a database and establish an index corresponding to the deep semantic feature data; and retrieve abnormal features based on the index corresponding to the deep semantic feature data to locate the abnormal log.

[0019] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0020] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0021] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0022] In the embodiments of the present application, a stream processing architecture is adopted to realize real-time processing of log data. Through microservices, the scalability and fault tolerance of the system are improved, which is more suitable for log monitoring of large-scale distributed systems. The introduction of attention mechanism, autoencoder and graph neural network to extract deep features of log data enhances the understanding of the intrinsic pattern of log data and effectively improves the log processing capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0024] Figure 1 Schematically illustrates an application scenario diagram of the log monitoring method, apparatus, device, medium, and program product according to an embodiment of the present application;

[0025] Figure 2 The following schematically shows a flow chart of a log monitoring method according to an embodiment of the present application;

[0026] Figure 3The following schematically shows a standardized processing flow chart of the log monitoring method according to an embodiment of the present application;

[0027] Figure 4 Schematically shows a deep semantic feature extraction diagram of a log monitoring method according to an embodiment of the present application;

[0028] Figure 5 The following schematically shows a structure diagram of a deep semantic extraction model of a log monitoring method according to an embodiment of the present application;

[0029] Figure 6 The following schematically shows a flow chart of locating abnormal logs according to a log monitoring method according to an embodiment of the present application;

[0030] Figure 7 Schematically shows a structural block diagram of a log monitoring device according to an embodiment of the present application; and

[0031] Figure 8 A block diagram of an electronic device suitable for implementing a log monitoring method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] In the financial sector, log monitoring is a core component of risk prevention and control, ensuring stable business operations. The high-frequency transactions and massive data exchange characteristics of financial systems make them vulnerable to cyberattacks, system failures, or operational errors. Logs, acting as the black box of system operations, record key information such as transaction details, user behavior, and device status in real time. Real-time monitoring and in-depth analysis of logs can achieve the following:

[0037] Risk warning and compliance supervision: Timely capture of abnormal transaction patterns (such as high-frequency transfers and cross-regional logins), identify abnormal trading behaviors, and meet regulatory requirements for transaction traceability;

[0038] System performance optimization: By analyzing the response time and error codes in the logs, we can identify problems such as excessive server load and interface timeouts, ensuring smooth transaction links and avoiding financial losses or user churn caused by system lags.

[0039] Quick fault location: When the system crashes or a transaction fails, the complete operation chain can be traced back through logs, shortening the troubleshooting time, reducing business interruption losses, and maintaining the credibility of financial institutions.

[0040] Therefore, log monitoring is the nerve center of digital operations in the financial industry. Its accuracy and timeliness are directly related to business continuity, compliance, and user asset security.

[0041] However, although the existing technology has made certain progress in automated log analysis and fault handling, it still has the following shortcomings: (1) Log format limitations: Existing systems usually require log data to follow a specific format and have limited ability to process non-standardized logs. (2) Insufficient real-time and dynamic performance: Existing systems may not be able to respond quickly to newly generated log data and have difficulty adapting to dynamic changes in log data. (3) Limited deep semantic analysis: Existing technology may not fully utilize natural language processing technology for in-depth semantic analysis, limiting the depth of fault analysis. (4) Scalability issues: Existing systems may lack good scalability and have difficulty adapting to the log monitoring needs of large-scale or distributed systems. Therefore, this application aims to address the limitations of existing log analysis technology and provide a log recording and monitoring method that does not rely on a specific log format, has high real-time and dynamic performance, can deeply explore the deep semantic features of log data, and has good scalability.

[0042] The embodiment of the present application provides a log monitoring method, which is based on a stream processing architecture and distributes the log data obtained in real time through microservices; wherein, the microservices include a preprocessing microservice, a feature extraction microservice and a real-time monitoring microservice. The preprocessing microservice is configured to: process the log data in a standardized manner to obtain standard semantic data; the feature extraction microservice is configured to: process the standard semantic data based on a deep semantic extraction model to obtain deep semantic feature data; wherein, the deep semantic extraction model is constructed based on an attention mechanism model, an autoencoder and a graph neural network; the real-time monitoring microservice is configured to: monitor the dynamic changes of the log data in real time based on an online learning mechanism, and locate abnormal logs based on the deep semantic feature data. The stream processing architecture is used to realize real-time processing of log data. Through microservices, the scalability and fault tolerance of the system are improved, and it is more suitable for log monitoring of large-scale distributed systems. The introduction of attention mechanisms, autoencoders and graph neural networks to extract deep features of log data enhances the understanding of the inherent patterns of log data and effectively improves the log processing capabilities.

[0043] In the technical solution of this application, the log information, user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0044] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided in the embodiments of the present application all provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0045] Figure 1 The application scenario diagram of the log monitoring method, apparatus, device, medium and program product according to the embodiments of the present application is schematically shown.

[0046] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0047] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0048] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0049] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0050] It should be noted that the log monitoring method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the log monitoring device provided in the embodiment of the present application can generally be set in the server 105. The log monitoring method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the log monitoring device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0052] The following will be based on Figure 1 The scene described by Figures 2 to 6 The log monitoring method according to the embodiment of the present application is described in detail.

[0053] Figure 2 The flowchart of the log monitoring method according to an embodiment of the present application is schematically shown.

[0054] like Figure 2 As shown, the log monitoring method of this embodiment includes operations S200 to S230. The log monitoring method is not limited to a specific execution subject. The execution subject can be any electronic device, such as a terminal device or a server device, etc. The execution subject can also be any software application or client.

[0055] In operation S200 , based on a stream processing architecture, log data acquired in real time is processed in a distributed manner through microservices, wherein the microservices include a preprocessing microservice, a feature extraction microservice, and a real-time monitoring microservice.

[0056] Stream processing architecture is a technical framework for processing continuous data streams in real time. It is primarily used in scenarios where data is highly real-time, large in volume, and continuously generated (such as log monitoring). Unlike traditional batch processing (which stores data first and then processes it), stream processing emphasizes immediate analysis and response to real-time, flowing data, making it suitable for business scenarios requiring millisecond or second latency.

[0057] The stream processing architecture includes but is not limited to the log collection layer, stream processing engine, storage and output layer.

[0058] Log collection layer: Receives logs from each business node in real time through a distributed message queue, and uses log processing tools to stream the logs to the queue, supporting concurrent access of tens of thousands of logs per second.

[0059] Stream processing engine: deploys a microservice cluster, consumes log streams in the queue, uses microservices to process logs, and preliminarily aggregates (counts log frequency by minute) log data.

[0060] Storage and output layer: The processed structured logs are written to the real-time data warehouse and pushed to the visualization platform or alarm system.

[0061] The log monitoring system implements distributed processing through a microservices architecture. Each microservice is responsible for a specific aspect of log analysis. For example, the data collection microservice, preprocessing microservice, feature extraction microservice, and real-time monitoring microservice correspond to data collection, preprocessing, feature extraction, and real-time monitoring, respectively. This microservices architecture improves the system's scalability and fault tolerance.

[0062] Log data can be obtained from log files using the data collection microservice. Log files are records generated by computer systems, network services, or applications during operation. They typically contain information such as timestamps, event levels, and event descriptions. The data collection microservice collects log files and obtains the log data contained in them.

[0063] It is understandable that the microservice architecture, as a software development architectural style, decomposes the application into a series of small services. Each microservice runs in its own independent process and is usually built around specific business capabilities.

[0064] Distributed processing mechanisms include but are not limited to parallel computing, state management, and elastic scaling.

[0065] Parallel computing: The stream processing engine divides the log stream into multiple partitions and distributes them to different nodes in a microservice cluster (each microservice cluster includes microservices such as data collection, preprocessing, feature extraction, and real-time monitoring) for parallel processing. It uses multithreading or graphics processing units (GPUs) to accelerate computationally intensive tasks such as semantic feature extraction.

[0066] State management: For logs that need to be associated across batches (such as tracking the context of a user's complete operation), these temporary data are persisted and shared across multiple processing nodes / task instances (such as saving the log sequence corresponding to the user session ID) to ensure state consistency in a distributed environment and ensure fault tolerance and continuity of stream processing.

[0067] Elastic Scaling: Container orchestration tools automatically adjust the number of microservice instances based on real-time load. For example, when log traffic surges, the number of parsing nodes can be dynamically increased to avoid processing delays.

[0068] In operation S210 , the log data is standardized by preprocessing the microservice to obtain standard semantic data.

[0069] Standardization processing includes preprocessing and semantic standardization, which standardizes log data to facilitate subsequent analysis.

[0070] In operation S220, the feature extraction microservice processes the standard semantic data based on the deep semantic extraction model to obtain deep semantic feature data; wherein the deep semantic extraction model is constructed based on the attention mechanism model, the autoencoder and the graph neural network.

[0071] The deep semantic extraction model performs deep feature extraction on standard semantic data. Conventional feature extraction in data analysis refers to extracting features or attributes that can represent the essence of the data from the original data. The deep semantic feature extraction of this application aims to extract features that can represent the intrinsic meaning of log data from log data.

[0072] Deep semantic features and complex structural features in log data can be extracted using attention models, graph neural networks, or autoencoders. Attention models can accurately extract word vectors and corresponding weights that reflect contextual relationships, graph neural networks can capture entity relationships in log data, and autoencoders can learn compressed representations of data, helping to discover potential patterns and anomalies in logs.

[0073] For example, it identifies key words such as “transaction failure,” “timeout,” and “system crash,” and extracts deep semantic vectors and attention weights associated with these words.

[0074] In operation S230 , the microservice is monitored in real time, the dynamic changes of the log data are monitored in real time based on an online learning mechanism, and abnormal logs are located based on deep semantic feature data.

[0075] Real-time monitoring means that the log monitoring system can respond instantly to changes in input or status. The monitoring system can quickly provide feedback, alerts, and locate abnormal events. Real-time and dynamic performance are two key requirements for log monitoring. Real-time performance requires the log monitoring system to quickly process and respond to newly generated log data, while dynamic performance requires the system to adapt to changes in log data, such as changes in log format and content.

[0076] In the embodiments of the present application, a stream processing architecture is adopted to realize real-time processing of log data. Through microservices, the scalability and fault tolerance of the system are improved, which is more suitable for log monitoring of large-scale distributed systems. The introduction of attention mechanism, autoencoder and graph neural network to extract deep features of log data enhances the understanding of the intrinsic pattern of log data and effectively improves the log processing capability.

[0077] Figure 3 The following schematically shows a standardized processing flow chart of the log monitoring method according to an embodiment of the present application.

[0078] like Figure 3 As shown, the log data is standardized in operation S210 to obtain standard semantic data, which includes operations S310 to S320.

[0079] In operation S310 , the log data is preprocessed based on a pre-trained deep neural network; wherein the preprocessing includes data cleaning and / or formatting.

[0080] Deep learning technology is used to automatically clean and format log data for data preprocessing. By training deep neural networks, we can automatically identify and remove noise from logs, converting unstructured log data into a standardized format for subsequent analysis. Deep learning, a machine learning method based on artificial neural networks, is particularly adept at processing large amounts of data and discovering complex patterns within it.

[0081] When training a deep neural network to process log data, labeled logs are first collected, noise and valid information are annotated, and preprocessed into a unified format. Deep learning neural network architectures such as CNN (convolutional neural network) and RNN (recurrent neural network) are used. The input layer receives the log sequence, and the hidden layer extracts features through multiple layers of neurons, distinguishing between noise patterns and structured fields. The output layer outputs cleaned data through an activation function, and loss functions such as cross-entropy can be used to optimize parameters. When deploying the model, real-time log data is parsed line by line to identify and remove noise such as duplicate entries and unusual formatting. Unstructured log data is converted to a standardized format, completing the unstructured-to-standardized transformation.

[0082] In operation S320 , semantic standardization is performed on the preprocessed log data to obtain standard semantic data.

[0083] The preprocessed log data usually contains a large amount of text information, which may have differences in syntax and format. In order to make the log data suitable for subsequent in-depth analysis, semantic standardization is required.

[0084] In an embodiment of the present application, log data is cleaned and formatted based on a pre-trained deep neural network, so that subsequent log data processing is independent of the format, format differences are reduced, and it is not dependent on a specific log format, and can process various standardized and non-standardized log data.

[0085] According to an embodiment of the present application, in operation S320, the preprocessed log data is semantically standardized to obtain standard semantic data, including: using natural language processing technology to semantically standardize the preprocessed log data to obtain standard semantic data; wherein, semantic standardization includes one or more of stem extraction, morphological restoration, synonym replacement and template matching.

[0086] The pre-processed log data is semantically standardized using natural language processing (NLP) technology. NLP technology is a branch of artificial intelligence and linguistics that aims to enable computers to understand, interpret, and generate human language.

[0087] Using NLP technology, especially context-based semantic understanding models, the information in log text can be converted into a unified and standardized format. This step includes but is not limited to stemming, lemmatization, synonym replacement, and template matching.

[0088] For example, a piece of log data is: "[ERR] Connection timeout to database server at 2023-04-29 09:45:30".

[0089] Stemming: Convert “timeout” to its base form “time out” to facilitate understanding by deep semantic extraction models.

[0090] Lemmatization: Lemmatizes "Connection" and "server" to "connect" and "serve," helping deep semantic extraction models identify word roots, simplify lexical forms, and eliminate grammatical differences.

[0091] Synonym replacement: "timeout" can be marked as a synonym associated with "failure" or "error" to enhance the deep semantic extraction model's recognition of errors and achieve term normalization.

[0092] Template matching: Align the processed text with a preset standardized template (such as the structure of "event type-target component-time-status description"), i.e., event type: ERR (error), target component: database serve (database server), time: 2023-04-29 09:45:30, status description: connect failure (connection failure).

[0093] In the embodiments of this application, natural language processing technology is used to semantically standardize preprocessed log data, converting the information in the log into unified, standardized semantics, reducing grammatical differences and making the log data suitable for subsequent in-depth analysis. Combining deep learning technology with natural language processing technology for standardized processing of log data improves the automation and accuracy of log processing.

[0094] Figure 4 A diagram schematically illustrates a deep semantic feature extraction diagram of a log monitoring method according to an embodiment of the present application.

[0095] Figure 5 The schematic diagram shows a deep semantic extraction model structure diagram of the log monitoring method according to an embodiment of the present application.

[0096] like Figure 4 、 Figure 5As shown, operation S220 processes the standard semantic data based on the deep semantic extraction model to obtain deep semantic feature data, which includes operations S410 to S440.

[0097] In operation S410, a vocabulary semantic vector and a vocabulary attention weight are generated according to standard semantic data through an attention mechanism model.

[0098] In operation S410, generating a vocabulary semantic vector and a vocabulary attention weight according to standard semantic data through an attention mechanism model includes operations S4101 to S4102.

[0099] In operation S4101, standard semantic data is processed through an attention mechanism model to generate a lexical semantic vector; wherein the lexical semantic vector includes lexical context information.

[0100] The attention mechanism model can use a Transformer-based model, such as the BERT model, to generate context-aware representations of log text, namely lexical semantic vectors.

[0101] Transformer represents a deep learning architecture based on the self-attention mechanism, which dynamically captures global semantic dependencies by calculating the association weight (attention score) of each element in the sequence with all other elements.

[0102] The BERT (Bidirectional Encoder Representations from Transformers) model is a pre-trained language model based on the Transformer architecture. It generates more comprehensive and accurate text semantic representations through bidirectional context modeling.

[0103] The BERT model captures the complex relationships between words and generates a high-dimensional semantic vector for each word. For example, the BERT model processes the standard semantic data described above and generates a vector representation for each word. For example, the words "connection" and "failure" require more than just the words themselves; their contextual relationships within the sentence must also be extracted.

[0104] In operation S4102, based on the attention mechanism of the attention mechanism model, key information of the standard semantic data is identified and a vocabulary attention weight is generated.

[0105] The attention mechanism is used to identify key information in the log. The attention score can quantify the semantic contribution of a word to the entire log sentence, thereby helping the model focus on the most important part of the log.

[0106] For example, the attention mechanism can help the model identify that "failure" and "database serve" are key information in the log, which are directly related to the error type and the affected component.

[0107] In an embodiment of the present application, standard semantic data is processed through an attention mechanism model to generate vocabulary semantic vectors, and the attention mechanism model is used to capture the complex relationships between words, perceive the contextual relationships of words, capture the global association of data, and dynamically generate semantic vectors rich in contextual information to improve the accuracy of subsequent tasks.

[0108] In operation S420 , the vocabulary semantic vector is compressed by an autoencoder to generate a deep encoding feature.

[0109] An autoencoder is an unsupervised learning model that compresses input data into a low-dimensional representation through an encoder and then reconstructs the original data through a decoder. In log analysis, autoencoders can be used to extract deep features from logs that may be closely related to the normal operation or abnormal state of the system.

[0110] Use autoencoders to extract deep features from logs. For example, use autoencoders to compress "failure" and "database serve" into a vector that represents the pattern of system connection failure.

[0111] In operation S430 , graph structural features are extracted based on the graph neural network according to the standard semantic data.

[0112] Because of the relationships between entities (vocabulary) in log data, graph neural networks (GNNs) can be used to extract structural features from log data. GNNs can process graph-structured data and capture complex relationships between entities, such as causal relationships and temporal order.

[0113] For example, if the log data contains interaction information between multiple components, such as "user -> database -> server", GNN captures the dependencies and interaction relationships between these components to generate graph structure features corresponding to the entities.

[0114] In operation S440 , the vocabulary semantic vector, the vocabulary attention weight, the deep encoding feature, and the graph structure feature are integrated to obtain deep semantic feature data.

[0115] The different types of features extracted in operations S410 to S430 are fused to form a comprehensive feature representation. For example, the lexical semantic vectors, lexical attention weights, deep encoding features, and graph structure features extracted above are fused to form a comprehensive feature vector that represents the deep semantics of the log entry.

[0116] In an embodiment of the present application, based on the attention mechanism model, autoencoder and graph neural network to extract the features of standard semantic data, different types of models are introduced to extract the deep features of the log, deeply explore the deep semantic information of the log, and fuse the extracted different types of features to form a comprehensive feature representation, which comprehensively represents the deep semantics of the log.

[0117] According to an embodiment of the present application, in operation S230, the dynamic changes of log data are monitored in real time based on the online learning mechanism, including: when the log pattern of the log data is updated, based on the online learning mechanism, the feature extraction strategy of the deep semantic extraction model is dynamically adjusted according to the updated log data to update the deep semantic feature data.

[0118] Because the characteristics of log data may change over time, a method for dynamically adjusting the feature extraction process is proposed. Through an online learning mechanism (online learning technology), the deep semantic extraction model can access new log data in real time and dynamically adjust network parameters or feature weights. The deep semantic extraction model can update its feature extraction strategy in real time based on the new log data to adapt to the dynamic changes in log data. The deep semantic extraction model instantly identifies differences and updates the feature extraction layer through algorithms such as gradient descent. This strengthens the ability to capture new semantic patterns, avoids offline lag, and continuously improves the accuracy and generalization of semantic parsing of complex log data.

[0119] For example, suppose a new log pattern appears after the system has been running for a while: "[WARN] Disk space is low on server at 2023-04-29 10:00:00." The deep semantic extraction model uses an online learning mechanism to update its vocabulary and feature extraction strategy in real time to identify the new warning type. Adaptive feature extraction: The deep semantic extraction model identifies "Disk space" and "low" as new key features and uses an adaptive mechanism to strengthen the weight of these features in subsequent feature analysis.

[0120] Online learning refers to a technical framework that receives log data streams in real time and dynamically updates model parameters or detection strategies by processing new log samples individually or in batches. As log data continues to be generated, deep semantic extraction models can automatically adapt to dynamic scenarios such as changes in log formats and evolving anomaly patterns without manual intervention or offline retraining, maintaining accurate and timely monitoring.

[0121] This application uses stream processing and online learning technologies to rapidly process and respond to newly generated log data. Stream processing ensures real-time log monitoring, while online learning enables the system to adapt to dynamic changes in log data, such as format or content changes. For example, combined conditions can be defined, such as simultaneously satisfying "number of warnings exceeding a threshold" and "specific server identification." For example, an alarm will be triggered if the number of occurrences of a specific statement exceeds a preset threshold.

[0122] In an embodiment of the present application, through online learning technology, the feature extraction strategy of the deep semantic extraction model is dynamically adjusted to achieve real-time updating of the feature extraction strategy to adapt to the dynamic changes of log data, thereby realizing real-time monitoring of log data and rapid processing and response to newly generated log data.

[0123] Figure 6 The following schematically shows a flow chart of locating abnormal logs according to the log monitoring method of an embodiment of the present application.

[0124] like Figure 6 As shown, operation S230 of locating abnormal logs based on deep semantic feature data includes operations S610 to S620.

[0125] In operation S610 , the deep semantic feature data is stored in a database, and an index corresponding to the deep semantic feature data is created.

[0126] The extracted deep semantic feature data is stored in a database. These deep semantic features will be stored in an efficient data structure to facilitate retrieval and analysis, and indexing will be established to speed up retrieval. This process involves building indexes, using database management systems, or adopting other data storage technologies.

[0127] For example, first, deep semantic features are written to a database table through a data interface. The fields are designed to include feature IDs, vector data, and log metadata (time, type, etc.). Next, a database index is established for the feature IDs and key metadata (such as timestamps and log types). Tree indexes can be used to accelerate equality queries or range queries. If vector data supports semantic retrieval (such as similarity queries), a vector database can be used or a vector index module can be integrated into a traditional database to construct an index structure based on cosine similarity and Euclidean distance, ultimately achieving efficient feature storage and fast retrieval.

[0128] In operation S620 , an abnormal feature is retrieved based on an index corresponding to the deep semantic feature data to locate the abnormal log.

[0129] When you need to diagnose a specific exception problem, such as a database connection failure, you can quickly locate the exception log by retrieving feature vectors related to "timeout" and "database server."

[0130] The retrieval process can be as follows: (1) Keyword vectorization: convert abnormality-related search keywords such as "timeout" and "database server" into vectors through models such as BERT. (2) Similarity retrieval: use vector indexes (such as cosine similarity) to match similar deep semantic feature vectors in the database. (3) Log association screening: extract corresponding logs based on the matching results, and filter out abnormal logs (such as records containing keywords within the past hour) based on metadata such as time and type. (4) Result presentation: sort by similarity and display log details (such as time and error information) to help quickly locate abnormal problems such as database connection timeouts.

[0131] In an embodiment of the present application, the extracted deep semantic features are stored in a database, an index is established to speed up the retrieval, and abnormal logs with problems are quickly located by retrieving related feature vectors.

[0132] Based on the above log monitoring method, this application also provides a log monitoring device. Figure 7 The device is described in detail.

[0133] Figure 7 The structural block diagram of the log monitoring device according to an embodiment of the present application is schematically shown.

[0134] like Figure 7 As shown, the log monitoring device of this embodiment includes a distributed processing module 700 , a pre-processing module 710 , a feature extraction module 720 and a real-time monitoring module 730 .

[0135] Distributed processing module 700 is used to distribute and process real-time log data via microservices based on a stream processing architecture. The microservices include a preprocessing microservice, a feature extraction microservice, and a real-time monitoring microservice. In one embodiment, distributed processing module 700 can be used to perform operation S200 described above, which will not be further described here.

[0136] The preprocessing module 710 is used to standardize the log data by preprocessing the microservices to obtain standard semantic data. In one embodiment, the preprocessing module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0137] Feature extraction module 720 is used to process standard semantic data using a feature extraction microservice based on a deep semantic extraction model to obtain deep semantic feature data. The deep semantic extraction model is constructed based on an attention mechanism model, an autoencoder, and a graph neural network. In one embodiment, feature extraction module 720 can be used to perform operation S220 described above and will not be further described here.

[0138] The real-time monitoring module 730 is used to monitor the dynamic changes of log data in real time based on the online learning mechanism by monitoring microservices in real time, and locate abnormal logs based on deep semantic feature data. In one embodiment, the real-time monitoring module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0139] According to an embodiment of the present application, the preprocessing module 710 includes: a preprocessing unit for preprocessing the log data based on a pre-trained deep neural network; wherein the preprocessing includes data cleaning and / or formatting; and a semantic standardization unit for performing semantic standardization on the preprocessed log data to obtain standard semantic data.

[0140] According to an embodiment of the present application, the semantic standardization unit includes: a natural language processing sub-unit, which is used to semantically standardize the pre-processed log data through natural language processing technology to obtain standard semantic data; wherein, semantic standardization includes one or more of stem extraction, morphological restoration, synonym replacement and template matching.

[0141] According to an embodiment of the present application, the feature extraction module 720 includes: an attention mechanism unit, which is used to generate lexical semantic vectors and lexical attention weights through an attention mechanism model based on standard semantic data; an autoencoder unit, which is used to compress lexical semantic vectors through an autoencoder to generate deep coding features; a graph neural network unit, which is used to extract graph structure features based on the graph neural network according to standard semantic data; and a fusion unit, which is used to fuse lexical semantic vectors, lexical attention weights, deep coding features and graph structure features to obtain deep semantic feature data.

[0142] According to an embodiment of the present application, the attention mechanism unit includes: a semantic vector sub-unit, which is used to process standard semantic data through an attention mechanism model to generate a lexical semantic vector; wherein the lexical semantic vector contains lexical context information; and an attention weight sub-unit, which is used to identify key information of the standard semantic data based on the attention mechanism of the attention mechanism model and generate a lexical attention weight.

[0143] According to an embodiment of the present application, the real-time monitoring module 730 includes: an online learning unit, which is used to dynamically adjust the feature extraction strategy of the deep semantic extraction model based on the online learning mechanism and the updated log data when the log pattern of the log data is updated, so as to update the deep semantic feature data.

[0144] According to an embodiment of the present application, the real-time monitoring module 730 also includes: an anomaly locating unit, which is used to store deep semantic feature data in a database and establish an index corresponding to the deep semantic feature data; and retrieve abnormal features based on the index corresponding to the deep semantic feature data to locate abnormal logs.

[0145] According to embodiments of the present application, any multiple modules among the distributed processing module 700, preprocessing module 710, feature extraction module 720, and real-time monitoring module 730 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present application, at least one of the distributed processing module 700, preprocessing module 710, feature extraction module 720, and real-time monitoring module 730 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the distributed processing module 700, the preprocessing module 710, the feature extraction module 720 and the real-time monitoring module 730 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0146] Figure 8 A block diagram of an electronic device suitable for implementing a log monitoring method according to an embodiment of the present application is schematically shown.

[0147] like Figure 8 As shown, an electronic device 900 according to an embodiment of the present application includes a processor 901, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 902 or programs loaded from a storage unit 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0148] Various programs and data required for the operation of the electronic device 900 are stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in one or more memories.

[0149] According to an embodiment of the present application, electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to bus 904. Electronic device 900 may also include one or more of the following components connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 908 including a hard disk; and a communication section 909 including a network interface card such as a LAN card or modem. Communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 910 as needed, so that computer programs read from the removable media can be installed into storage section 908 as needed.

[0150] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0151] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0152] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the log monitoring method provided in the embodiments of the present application.

[0153] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the processor 901 executes the computer program. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0154] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0155] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0156] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

Claims

1. A log monitoring method, characterized in that: The method comprises: Based on the stream processing architecture, the real-time log data is processed in a distributed manner through microservices; wherein the microservices include pre-processing microservices, feature extraction microservices and real-time monitoring microservices; Standardize the log data through the preprocessing microservice to obtain standard semantic data; Processing the standard semantic data based on a deep semantic extraction model through the feature extraction microservice to obtain deep semantic feature data; wherein the deep semantic extraction model is constructed based on an attention mechanism model, an autoencoder, and a graph neural network; The real-time monitoring microservice is used to monitor the dynamic changes of the log data in real time based on an online learning mechanism, and locate abnormal logs based on the deep semantic feature data.

2. The method according to claim 1, characterized in that The standardization processing of the log data to obtain standard semantic data includes: Preprocessing the log data based on a pre-trained deep neural network; wherein the preprocessing includes data cleaning and / or formatting; and The preprocessed log data is subjected to semantic standardization processing to obtain the standard semantic data.

3. The method according to claim 2, characterized in that The performing semantic standardization on the preprocessed log data to obtain the standard semantic data includes: semantically standardizing the preprocessed log data using natural language processing technology to obtain the standard semantic data; The semantic standardization includes one or more of stem extraction, lemma restoration, synonym replacement and template matching.

4. The method according to claim 1, wherein The processing of the standard semantic data based on the deep semantic extraction model to obtain deep semantic feature data includes: Generate a lexical semantic vector and a lexical attention weight according to the standard semantic data through the attention mechanism model; Compressing the lexical semantic vector by the autoencoder to generate deep coding features; Extracting graph structural features based on the graph neural network according to the standard semantic data; and The lexical semantic vector, the lexical attention weight, the deep coding feature and the graph structure feature are integrated to obtain the deep semantic feature data.

5. The method according to claim 4, characterized in that The generating of a vocabulary semantic vector and a vocabulary attention weight by the attention mechanism model according to the standard semantic data includes: Processing the standard semantic data through the attention mechanism model to generate the lexical semantic vector; wherein the lexical semantic vector includes lexical context information; and Based on the attention mechanism of the attention mechanism model, key information of the standard semantic data is identified and the vocabulary attention weight is generated.

6. The method according to claim 1, characterized in that The real-time monitoring of the dynamic changes of the log data based on the online learning mechanism includes: When the log pattern of the log data is updated, based on the online learning mechanism, the feature extraction strategy of the deep semantic extraction model is dynamically adjusted according to the updated log data to update the deep semantic feature data.

7. The method according to claim 1, characterized in that The locating abnormal logs based on the deep semantic feature data includes: Storing the deep semantic feature data in a database and creating an index corresponding to the deep semantic feature data; and Based on the index corresponding to the deep semantic feature data, the abnormal feature is retrieved to locate the abnormal log.

8. A log monitoring device, characterized in that: The device comprises: A distributed processing module, which is used to process the real-time acquired log data in a distributed manner through microservices based on a stream processing architecture; wherein the microservices include a preprocessing microservice, a feature extraction microservice, and a real-time monitoring microservice; A preprocessing module, configured to standardize the log data through the preprocessing microservice to obtain standard semantic data; A feature extraction module, configured to process the standard semantic data based on a deep semantic extraction model through the feature extraction microservice to obtain deep semantic feature data; wherein the deep semantic extraction model is constructed based on an attention mechanism model, an autoencoder, and a graph neural network; and The real-time monitoring module is used to monitor the dynamic changes of the log data in real time based on the online learning mechanism through the real-time monitoring microservice, and locate abnormal logs based on the deep semantic feature data.

9. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images