A log intelligent collection and multi-dimensional abnormality alarm method and system

Abstract

The application provides a log intelligent collection and multi-dimensional abnormality alarm method and system, the method comprises the following steps: a parsing algorithm is used for structural analysis on the pretreated original log data; the log template and dynamic parameters are jointly vectorized; an abnormality detection model is established to transfer the general knowledge of the source domain log in the log semantic vector sequence to the target domain; according to the preset multi-level alarm threshold and abnormality probability score, different levels of abnormality alarm information are generated; and the abnormality alarm information, log distribution statistical analysis chart and log trend analysis chart are displayed in real time through a visual interactive interface. The application significantly reduces the sensitivity to the starting position of the log variable, avoids the loss of key information caused by traditional wildcard replacement, integrates the multi-level alarm threshold and the visual interactive interface, and supports real-time display, trend analysis and automatic work order triggering of abnormal logs.

Description

Technical Field

[0001] This invention belongs to the technical field of log management, specifically relating to a method and system for intelligent log collection and multi-dimensional anomaly alerting. Background Technology

[0002] As cloud computing and distributed systems continue to expand in scale, system logs, as a critical data source for recording operational status, fault information, and operational behaviors, have become the core foundation of Intelligent Operations and Maintenance (AIOps). Log anomaly detection helps operations personnel quickly identify system faults, pinpoint root causes, and restore services, which is of great significance for ensuring high system availability. However, the massive, heterogeneous, and rapidly growing log data makes traditional manual analysis and rule matching methods unsustainable.

[0003] In recent years, significant progress has been made in deep learning-based log anomaly detection methods. Mainstream techniques typically consist of two stages: log parsing and anomaly detection. First, unstructured logs are converted into log templates using algorithms such as Drain and Spell, and then models such as LSTM and Transformer are used to model the template sequence.

[0004] Existing technologies generally suffer from drawbacks such as log parsing being sensitive to variable positions, ignoring the semantic information of dynamic parameters during the parsing process, and poor model generalization ability in cross-system or cross-dataset scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for intelligent log collection and multi-dimensional anomaly alerting, thereby overcoming the shortcomings of the prior art.

[0006] In a first aspect, the present invention provides a method for intelligent log collection and multi-dimensional anomaly alerting, the method comprising: The raw log data from heterogeneous data sources is captured in real time by a distributed log collector, and preprocessing is performed on the raw log data. The log parsing algorithm based on semantic awareness and parsing forest performs structured parsing on the preprocessed original log data, generates log templates, and retains dynamic parameters; A pre-trained language model based on a two-layer coding structure is used to jointly vectorize the log template and the dynamic parameters to generate a log semantic vector sequence that integrates static semantic information and dynamic parameter features; An anomaly detection model is established, and the log semantic vector sequence is input into the anomaly detection model to transfer the general knowledge of the source domain logs in the log semantic vector sequence to the target domain, and an anomaly probability score for each log semantic vector sequence is output. Based on the preset multi-level alarm thresholds and the anomaly probability score, different levels of anomaly alarm information are generated; The abnormal alarm information, log distribution statistical analysis charts, and log trend analysis charts are displayed in real time through a visual interactive interface.

[0007] Compared with existing technologies, the beneficial effects of this invention are as follows: At the log parsing level, by constructing a semantically aware parsing forest and introducing a character order perturbation mechanism, the sensitivity to the starting position of log variables is significantly reduced, greatly improving parsing accuracy and robustness. At the log representation level, a joint vectorization method of templates and parameters is adopted, integrating dynamic parameter features into static template semantics, avoiding the loss of key information caused by traditional wildcard replacement, and providing richer feature inputs for anomaly detection. At the anomaly detection model level, the introduction of an anomaly detection model enables efficient transfer of source domain knowledge even when the backbone network is frozen, significantly improving the model's generalization ability in cross-system and cross-dataset scenarios and reducing the cold start cost of new business systems. At the system operation and maintenance level, multi-level alarm thresholds and a visual interactive interface are integrated, supporting real-time display, trend analysis, and automated work order triggering of anomaly logs, forming an end-to-end intelligent operation and maintenance closed loop from log collection to fault handling, effectively improving fault detection efficiency and system stability.

[0008] Furthermore, the step of the log parsing algorithm based on semantic awareness and parsing forest to perform structured parsing on the preprocessed original log data, generate log templates, and retain dynamic parameters includes: A fixed-depth parsed forest is constructed, and the preprocessed raw log data is subjected to multiple rounds of character order perturbation to generate multiple log replicas; The semantic embedding in the pre-trained BERT model is used to calculate the mixed similarity between the log replica and the candidate template, and the optimal template is selected based on the mixed similarity. The dynamic parameters in the log replica are replaced with placeholders with semantic representations, and a mapping dictionary of log templates and parameter key-value pairs is constructed.

[0009] Furthermore, the step of jointly vectorizing the log template and the dynamic parameters using a pre-trained language model based on a two-layer coding structure includes: The log template is input into a pre-trained language model to extract context-aware template semantic vectors, and the template semantic vectors are weighted using inverse document frequency to enhance the representation of rare events. The dynamic parameters are distinguished into text-type parameters and numerical-type parameters, and the text-type parameters are mapped into text-type parameter vectors and the numerical parameters are mapped into numerical parameter vectors through the pre-trained language model. The textual parameter vector and the numerical parameter vector are weighted and aggregated using an attention mechanism to obtain a parameter feature vector; The template semantic vector and the parameter feature vector are concatenated and fused.

[0010] Furthermore, the anomaly detection model is established based on the ITLog model of the improved Transformer.

[0011] Furthermore, before the step of inputting the log semantic vector sequence into the anomaly detection model, the method further includes: The log semantic vector sequences in the training set are accumulated to obtain sample points. The HDBSCAN hierarchical clustering algorithm is used to cluster the sample points, and the FastICA algorithm is combined to reduce the dimensionality of the clustered sample points. Based on the distribution of normal log sequences within the clustered sample points, classification probability labels are assigned to unlabeled log sequences, and the anomaly detection model is trained using the cross-entropy loss function combined with the probability labels.

[0012] Furthermore, the step of generating different levels of abnormal alarm information based on preset multi-level alarm thresholds and the abnormal probability score includes: Based on the anomaly probability score, the multi-level alarm thresholds are divided into fatal alarm messages, critical alarm messages, warning alarm messages, and alert alarm messages; If the anomaly probability score of the log semantic vector sequence exceeds the preset multi-level alarm threshold, a fault handling work order is generated and pushed to the operation and maintenance mobile terminal.

[0013] Furthermore, the visual interactive interface includes an information configuration panel, a log details display panel, and a statistical chart panel.

[0014] Secondly, the present invention also provides a log intelligent collection and multi-dimensional anomaly alarm system, the system comprising: The capture and processing module is used to capture raw log data from heterogeneous data sources in real time through a distributed log collector and to perform preprocessing on the raw log data. The parsing and generation module is used to perform structured parsing on the preprocessed original log data based on the log parsing algorithm of semantic awareness and parsing forest, generate log templates and retain dynamic parameters; A fusion generation module is used to jointly vectorize the log template and the dynamic parameters using a pre-trained language model based on a two-layer coding structure, so as to generate a log semantic vector sequence that fuses static semantic information and dynamic parameter features. An output module is established to build an anomaly detection model and input the log semantic vector sequence into the anomaly detection model to transfer the general knowledge of the source domain logs in the log semantic vector sequence to the target domain, and output the anomaly probability score of each log semantic vector sequence. The generation module is used to generate different levels of abnormal alarm information based on the preset multi-level alarm thresholds and the abnormal probability score. The display module is used to display the abnormal alarm information, log distribution statistical analysis charts, and log trend analysis charts in real time through a visual interactive interface.

[0015] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned intelligent log collection and multi-dimensional anomaly alarm method.

[0016] Fourthly, the present invention also provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-mentioned intelligent log collection and multi-dimensional anomaly alarm method. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the intelligent log collection and multi-dimensional anomaly alarm method in the first embodiment of the present invention; Figure 2 This is a structural block diagram of the intelligent log collection and multi-dimensional anomaly alarm system in the second embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device in the third embodiment of the present invention.

[0019] Explanation of key component symbols: 10. Capture and Processing Module; 20. Parsing and Generation Module; 30. Generation and Fusion Module; 40. Output Establishment Module; 50. Generation Module; 60. Display Module; 70. Bus; 71. Processor; 72. Memory; 73. Communication interface.

[0020] The embodiments of the present invention will be further described below with reference to the accompanying drawings. Detailed Implementation

[0021] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0022] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0024] Example 1 Please see Figure 1 The figure shows the intelligent log collection and multi-dimensional anomaly alarm method in the first embodiment of the present invention, the method including steps S1 to S6: S1, capture raw log data from heterogeneous data sources in real time through a distributed log collector, and perform preprocessing on the raw log data; It should be noted that the preprocessing includes sequentially performing noise filtering, stop word removal, compound word splitting, and timestamp normalization on the raw log data.

[0025] S2, The log parsing algorithm based on semantic awareness and parsing forest performs structured parsing on the preprocessed original log data, generates log templates and retains dynamic parameters; Specifically, step S2 includes steps S21 to S23: S21, Construct a fixed-depth parsing forest and perform multiple rounds of character order perturbation on the preprocessed original log data to generate multiple log replicas; It should be noted that a fixed-depth parse tree structure of 4 is initialized, and the sliding window size is set to 1000 log entries. When the window is full, the Fisher-Yates random shuffling algorithm is applied to the log entries within the window, generating three sequential replicas. For each log entry in each replica, a circular right shift operation is further performed: for example, the original log token sequence ["ERROR","opening","file","blk_123"] becomes ["opening","file","blk_123","ERROR"] after one right shift. These operations generate multiple parse trees, forming a parse forest.

[0026] S22, the semantic embedding in the pre-trained BERT model is used to calculate the mixed similarity between the log copy and the candidate template, and the optimal template is selected based on the mixed similarity; It should be noted that for each log copy, length matching and token-by-token matching are first performed in the parse tree. If a candidate template is matched, structural similarity (based on the Jaccard coefficient) is calculated. If the highest structural similarity does not exceed a preset threshold (set to 0.7 in this embodiment), the semantic awareness module is triggered: the pre-trained BERT-base model is loaded, and the original log and candidate template are encoded into 768-dimensional semantic vectors respectively, and the cosine similarity is calculated as the semantic similarity. The final formula for calculating the mixed similarity is: S_hybrid=β*S_semantic+(1-β)*S_struct; In the formula, S_hybrid represents the hybrid similarity, β represents the weight coefficient of semantic similarity, S_semantic represents semantic similarity, and S_struct represents structural similarity, where β is 0.6. The candidate template with the highest hybrid similarity is selected as the matching result.

[0027] S23, replace the dynamic parameters in the log replica with placeholders with semantic representations, and construct a mapping dictionary between log templates and parameter key-value pairs; It should be noted that, based on the preset parameter replacement rule table, dynamic parameters in the logs are replaced with semantic placeholders. For example, blk_5126469776250 is replaced with... <blockid>Replace 10.251.42.84 with <ip>At the same time, a JSON-formatted parameter dictionary is constructed for each log entry, such as {"blockId":"blk_5126469776250","IP":"10.251.42.84"}, and stored in a MySQL database.

[0028] S3, a pre-trained language model based on a two-layer coding structure is used to jointly vectorize the log template and the dynamic parameters to generate a log semantic vector sequence that integrates static semantic information and dynamic parameter features; Specifically, step S3 includes steps S31 to S34: S31, the log template is input into a pre-trained language model to extract context-aware template semantic vectors, and the template semantic vectors are weighted using inverse document frequency to enhance the representation of rare events; It should be noted that the log template (such as "ERROR opening file") should be used. <blockid>The input is fed into the ALBERT-base pre-trained model, and the output of the last layer labeled [CLS] is taken as the initial semantic vector (768 dimensions) of the template. Then, the inverse document frequency (IDF) of the template in the entire log dataset is calculated using the formula IDF = log(N / N_T), where N is the total number of log events and N_T is the number of events containing the template. The IDF value is mapped to the [0,1] interval using max-min normalization and multiplied with the initial semantic vector to obtain the weighted template semantic vector.

[0029] S32, distinguish between text-type parameters and numerical parameters in the dynamic parameters, and map the text-type parameters into text-type parameter vectors and the numerical parameters into numerical parameter vectors through the pre-trained language model; It should be noted that for text-type parameters (such as...) <user>The same ALBERT model is used to map it to a 768-dimensional vector. For numerical parameters (such as port number 8080), Z-score standardization is first performed (subtract the mean and divide by the standard deviation), and then it is mapped to a 768-dimensional vector space through a fully connected layer.

[0030] S33, The textual parameter vector and the numerical parameter vector are weighted and aggregated through an attention mechanism to obtain a parameter feature vector; It should be noted that the attention mechanism is used to perform a weighted summation of all parameter vectors in the same log: the dot product of each parameter vector and the learnable vector is calculated, the attention weights are obtained by softmax normalization, and then the weighted aggregation is used to obtain the final parameter feature vector. S34, the template semantic vector and the parameter feature vector are concatenated and fused; It should be noted that the template semantic vector (768 dimensions) is concatenated with the parameter feature vector (768 dimensions) to obtain a 1536-dimensional intermediate vector. This intermediate vector is then reduced to 256 dimensions through a fully connected layer, which serves as the final semantic vector for the log. Logs belonging to the same HDFS block ID are arranged in chronological order to form a sequence of log semantic vectors.

[0031] S4. Establish an anomaly detection model and input the log semantic vector sequence into the anomaly detection model to transfer the general knowledge of the source domain logs in the log semantic vector sequence to the target domain, and output the anomaly probability score of each log semantic vector sequence. It should be noted that the anomaly detection model is based on the improved Transformer ITLog model, in which the cross-domain adaptive fine-tuning module is the Dynamically Gated Low-Rank Adapter (DGLRA). DGLRA reduces and increases the dimensionality of the input features through low-rank decomposition and introduces a dynamic gating mechanism based on the input features to adaptively adjust the feature update weights of each dimension. When fine-tuning across datasets, the parameters of the Transformer backbone network are frozen, only the parameters of the DGLRA module are updated, and a selective unfreezing strategy is used to fine-tune the high-level network. Specifically, step S4 includes steps S41 to S42: S41, the log semantic vector sequences in the training set are accumulated to obtain sample points, the HDBSCAN hierarchical clustering algorithm is used to cluster the sample points, and the FastICA algorithm is combined to reduce the dimensionality of the clustered sample points. 558,782 log sequences were selected from the training set, and the semantic vectors of each sequence were summed to form sample points. The FastICA algorithm was used to reduce the 256-dimensional vectors to 32-dimensional vectors, and then HDBSCAN clustering was performed (with a minimum cluster size of 50). For each cluster, if the cluster contained known normal log sequences, the Euclidean distance between the unlabeled samples within the cluster and the cluster center was calculated, and the distance was normalized to obtain the anomaly probability label.

[0032] S42, Based on the distribution of normal log sequences within the clustered sample points, classify unlabeled log sequences with probability labels, and train the anomaly detection model using the cross-entropy loss function combined with the probability labels; It should be noted that an ITLog model based on an improved Transformer is built, inserting a Dynamically Gated Low-Rank Adapter (DGLRA) into each Transformer encoder layer. DGLRA reduces the input features from 768 dimensions to 32 dimensions (low-rank decomposition), then increases them back to 768 dimensions, while simultaneously generating dimension-wise updated weights through a Sigmoid-gated network. The model uses the cross-entropy loss function and is trained using the probability labels obtained in step S41. The optimizer is Adam, with a learning rate of 1e-5. After training, the parameters of the Transformer backbone network are frozen, while only the parameters of the last two encoder layers and all DGLRA module parameters are unfrozen, allowing for fine-tuning on the BGL dataset.

[0033] S5. Based on the preset multi-level alarm thresholds and the anomaly probability score, generate different levels of anomaly alarm information. Specifically, step S5 includes steps S51 to S52: S51, the multi-level alarm thresholds are divided into fatal alarm information, critical alarm information, warning alarm information and reminder alarm information according to the abnormal probability score; It should be noted that, based on the anomaly probability score output by the model, four alarm thresholds are set: Alert (0.5-0.6), Warning (0.6-0.75), Critical (0.75-0.9), and Fatal (0.9-1.0). Different levels correspond to different color codes (blue, yellow, orange, and red) and push notification strategies.

[0034] S52, if the anomaly probability score of the log semantic vector sequence exceeds the preset multi-level alarm threshold, a fault handling work order is generated and pushed to the operation and maintenance mobile terminal. It should be noted that when the anomaly score of a log sequence exceeds 0.9, the system automatically determines it as a fatal alarm. In this case, the system determines the alarm based on parameter fields in the log (such as...). <blockid>The system queries the associated fault handling script library. If a matching Python or Shell script exists, it is executed automatically (e.g., restarting a DataNode or migrating a replica). If no matching script exists, the system calls the WeChat Work API via RestTemplate to generate a fault handling work order containing exception details, occurrence time, and host IP address, which is then pushed to the operations and maintenance personnel's mobile device. After confirmation on the mobile device, the operations and maintenance personnel can edit and bind the script in the platform's web interface to achieve continuous optimization of fault handling.

[0035] S6, The abnormal alarm information, log distribution statistical analysis charts and log trend analysis charts are displayed in real time through a visual interactive interface; It should be noted that the visual interactive interface includes an information configuration panel, a log details display panel, and a statistical chart panel. The information configuration panel is used to set the log source, data display period, and alarm threshold; the log details display panel is used to display the original log content, parsed template, parameter information, and anomaly highlighting marks, and supports filtering by multiple dimensions such as time, log level, and source IP; the statistical chart panel is used to display trend charts, distribution pie charts, and historical comparison charts of abnormal logs, and the charts support dynamic refreshing and drill-down queries.

[0036] In summary, the intelligent log collection and multi-dimensional anomaly alarm method in the above embodiments of the present invention significantly reduces the sensitivity to the starting position of log variables and greatly improves parsing accuracy and robustness at the log parsing level by constructing a semantically aware parsing forest and introducing a character order perturbation mechanism. At the log representation level, a joint vectorization method of templates and parameters is adopted to integrate dynamic parameter features into static template semantics, avoiding the loss of key information caused by traditional wildcard replacement and providing richer feature inputs for anomaly detection. At the anomaly detection model level, the introduction of an anomaly detection model enables efficient transfer of source domain knowledge even when the backbone network is frozen, significantly improving the model's generalization ability in cross-system and cross-dataset scenarios and reducing the cold start cost of new business systems. At the system operation and maintenance level, multi-level alarm thresholds and a visual interactive interface are integrated, supporting real-time display, trend analysis, and automated work order triggering of anomaly logs, forming an end-to-end intelligent operation and maintenance closed loop from log collection to fault handling, effectively improving fault detection efficiency and system stability.

[0037] Example 2 Please see Figure 2 The figure shows a log intelligent collection and multi-dimensional anomaly alarm system according to the second embodiment of the present invention. The system includes: The capture and processing module 10 is used to capture raw log data from heterogeneous data sources in real time through a distributed log collector, and to perform preprocessing on the raw log data. The parsing and generation module 20 is used to perform structured parsing on the preprocessed original log data based on the log parsing algorithm of semantic awareness and parsing forest, generate log templates and retain dynamic parameters; The generation fusion module 30 is used to jointly vectorize the log template and the dynamic parameters using a pre-trained language model based on a two-layer coding structure, so as to generate a log semantic vector sequence that integrates static semantic information and dynamic parameter features. An output module 40 is established to build an anomaly detection model and input the log semantic vector sequence into the anomaly detection model to transfer the general knowledge of the source domain logs in the log semantic vector sequence to the target domain and output the anomaly probability score of each log semantic vector sequence. The anomaly detection model is an ITLog model based on an improved Transformer. The generation module 50 is used to generate different levels of abnormal alarm information based on the preset multi-level alarm thresholds and the abnormal probability score. The display module 60 is used to display the abnormal alarm information, log distribution statistical analysis charts and log trend analysis charts in real time through a visual interactive interface, wherein the visual interactive interface includes an information configuration panel, a log details display panel and a statistical chart panel.

[0038] In some optional embodiments, the parsing generation module 20 includes: A processing unit is constructed to build a fixed-depth parsed forest and to perform multiple rounds of character order perturbation on the preprocessed raw log data to generate multiple log replicas. The calculation selection unit is used to calculate the mixed similarity between the log copy and the candidate template using the semantic embedding in the pre-trained BERT model, and select the optimal template based on the mixed similarity. The replacement building unit is used to replace the dynamic parameters in the log replica with placeholders with semantic representations and to build a mapping dictionary of log templates and parameter key-value pairs.

[0039] In some alternative embodiments, the generation fusion module 30 includes: The input extraction unit is used to input the log template into a pre-trained language model to extract context-aware template semantic vectors, and to weight the template semantic vectors using inverse document frequency to enhance the representation of rare events. The differentiation mapping unit is used to differentiate between text-type parameters and numerical parameters in the dynamic parameters, and to map the text-type parameters into text-type parameter vectors and the numerical parameters into numerical parameter vectors through the pre-trained language model. The weighted aggregation unit is used to weight and aggregate the textual parameter vector and the numerical parameter vector through an attention mechanism to obtain a parameter feature vector; The splicing and fusion is used to splice and fuse the template semantic vector and the parameter feature vector.

[0040] In some alternative embodiments, the output creation module 40 includes: The accumulating clustering unit is used to accumulate the log semantic vector sequence in the training set to obtain sample points. The HDBSCAN hierarchical clustering algorithm is used to cluster the sample points, and the FastICA algorithm is combined to reduce the dimensionality of the clustered sample points. The classification training unit is used to classify unlabeled log sequences with probability labels based on the distribution of normal log sequences within the clustered sample points, and to train the anomaly detection model using the cross-entropy loss function in combination with the probability labels.

[0041] In some alternative embodiments, the generation module 50 includes: The segmentation unit is used to divide the multi-level alarm threshold into fatal alarm information, critical alarm information, warning alarm information, and alert alarm information based on the anomaly probability score; A push unit is used to generate a fault handling work order and push it to the operation and maintenance mobile terminal if the anomaly probability score of the log semantic vector sequence exceeds the preset multi-level alarm threshold.

[0042] The functions or operation steps implemented by the above modules and units are largely the same as those in the above method embodiments, and will not be repeated here.

[0043] The intelligent log collection and multi-dimensional anomaly alarm system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0044] Example 3 The third embodiment of the present invention also proposes an electronic device, please refer to [link / reference]. Figure 3 The image shows an electronic device according to a third embodiment of the present invention.

[0045] The electronic device may include a processor 71 and a memory 72 storing computer program instructions.

[0046] Specifically, the processor 71 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement this application.

[0047] The memory 72 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 72 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 72 may include removable or non-removable (or fixed) media. Where appropriate, the memory 72 may be internal or external to a data processing device. In a particular embodiment, the memory 72 is non-volatile memory. In a particular embodiment, the memory 72 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0048] The memory 72 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 71.

[0049] The processor 71 reads and executes the computer program instructions stored in the memory 72 to implement the intelligent log collection and multi-dimensional anomaly alarm method of the above embodiment 1.

[0050] In some embodiments, the electronic device may further include a communication interface 73 and a bus 70. For example, Figure 3 As shown, the processor 71, memory 72, and communication interface 73 are connected through bus 70 and complete communication with each other.

[0051] The communication interface 73 is used to enable communication between the various modules, devices, units, and / or equipment in this application. The communication interface 73 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0052] Bus 70 includes hardware, software, or both, that couples the components of a device together. Bus 70 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 70 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 70 may include one or more buses. Although this application describes and illustrates a specific bus, this application considers any suitable bus or interconnection.

[0053] The electronic device can acquire the log intelligent collection and multi-dimensional anomaly alarm system and execute the log intelligent collection and multi-dimensional anomaly alarm method of this embodiment.

[0054] Furthermore, in conjunction with the intelligent log collection and multi-dimensional anomaly alarm method in Embodiment 1 above, this application can provide a storage medium for implementation. This storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement the intelligent log collection and multi-dimensional anomaly alarm method of Embodiment 1 above.

[0055] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0056] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.< / blockid> < / user> < / blockid> < / ip> < / blockid>

Claims

1. A method for intelligent log collection and multi-dimensional anomaly alerting, characterized in that, include: The raw log data from heterogeneous data sources is captured in real time by a distributed log collector, and preprocessing is performed on the raw log data. The log parsing algorithm based on semantic awareness and parsing forest performs structured parsing on the preprocessed raw log data, generates log templates, and retains dynamic parameters; A pre-trained language model based on a two-layer coding structure is used to jointly vectorize the log template and the dynamic parameters to generate a log semantic vector sequence that integrates static semantic information and dynamic parameter features; An anomaly detection model is established, and the log semantic vector sequence is input into the anomaly detection model to transfer the general knowledge of the source domain logs in the log semantic vector sequence to the target domain, and an anomaly probability score for each log semantic vector sequence is output. Based on the preset multi-level alarm thresholds and the anomaly probability score, different levels of anomaly alarm information are generated; The abnormal alarm information, log distribution statistical analysis charts, and log trend analysis charts are displayed in real time through a visual interactive interface.

2. The method for intelligent log collection and multi-dimensional anomaly alarm according to claim 1, characterized in that, The log parsing algorithm based on semantic awareness and parsing forest performs structured parsing on the preprocessed original log data, generates log templates, and retains dynamic parameters. The steps include: A fixed-depth parsed forest is constructed, and the preprocessed raw log data is subjected to multiple rounds of character order perturbation to generate multiple log replicas; The semantic embedding in the pre-trained BERT model is used to calculate the mixed similarity between the log copy and the candidate template, and the optimal template is selected based on the mixed similarity. The dynamic parameters in the log replica are replaced with placeholders with semantic representations, and a mapping dictionary of log templates and parameter key-value pairs is constructed.

3. The method for intelligent log collection and multi-dimensional anomaly alarm according to claim 1, characterized in that, The step of jointly vectorizing the log template and the dynamic parameters using a pre-trained language model based on a two-layer coding structure includes: The log template is input into a pre-trained language model to extract context-aware template semantic vectors, and the template semantic vectors are weighted using inverse document frequency to enhance the representation of rare events. The dynamic parameters are distinguished into text-type parameters and numerical-type parameters, and the text-type parameters are mapped into text-type parameter vectors and the numerical parameters are mapped into numerical parameter vectors through the pre-trained language model. The textual parameter vector and the numerical parameter vector are weighted and aggregated using an attention mechanism to obtain a parameter feature vector; The template semantic vector and the parameter feature vector are concatenated and fused.

4. The method for intelligent log collection and multi-dimensional anomaly alarm according to claim 1, characterized in that, The anomaly detection model is based on the improved Transformer ITLog model.

5. The method for intelligent log collection and multi-dimensional anomaly alarm according to claim 1, characterized in that, Before the step of inputting the log semantic vector sequence into the anomaly detection model, the method further includes: The log semantic vector sequences in the training set are accumulated to obtain sample points. The HDBSCAN hierarchical clustering algorithm is used to cluster the sample points, and the FastICA algorithm is combined to reduce the dimensionality of the clustered sample points. Based on the distribution of normal log sequences within the clustered sample points, classification probability labels are assigned to unlabeled log sequences, and the anomaly detection model is trained using the cross-entropy loss function combined with the probability labels.

6. The method for intelligent log collection and multi-dimensional anomaly alarm according to claim 1, characterized in that, The step of generating different levels of abnormal alarm information based on preset multi-level alarm thresholds and the abnormality probability score includes: Based on the anomaly probability score, the multi-level alarm thresholds are divided into fatal alarm messages, critical alarm messages, warning alarm messages, and alert alarm messages; If the anomaly probability score of the log semantic vector sequence exceeds the preset multi-level alarm threshold, a fault handling work order is generated and pushed to the operation and maintenance mobile terminal.

7. The method for intelligent log collection and multi-dimensional anomaly alarm according to claim 1, characterized in that, The visual interactive interface includes an information configuration panel, a log details display panel, and a statistical chart panel.

8. A log intelligent collection and multi-dimensional anomaly alarm system, characterized in that, The system includes: The capture and processing module is used to capture raw log data from heterogeneous data sources in real time through a distributed log collector and to perform preprocessing on the raw log data. The parsing and generation module is used to perform structured parsing on the preprocessed original log data based on the log parsing algorithm of semantic awareness and parsing forest, generate log templates and retain dynamic parameters; A fusion generation module is used to jointly vectorize the log template and the dynamic parameters using a pre-trained language model based on a two-layer coding structure, so as to generate a log semantic vector sequence that fuses static semantic information and dynamic parameter features. An output module is established to build an anomaly detection model and input the log semantic vector sequence into the anomaly detection model to transfer the general knowledge of the source domain logs in the log semantic vector sequence to the target domain, and output the anomaly probability score of each log semantic vector sequence. The generation module is used to generate different levels of abnormal alarm information based on the preset multi-level alarm thresholds and the abnormal probability score. The display module is used to display the abnormal alarm information, log distribution statistical analysis charts, and log trend analysis charts in real time through a visual interactive interface.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the log intelligent collection and multi-dimensional anomaly alarm method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the intelligent log collection and multi-dimensional anomaly alarm method as described in any one of claims 1 to 7.

Images