A system detection method and device based on multi-source heterogeneous data
By employing a system detection method based on multi-source heterogeneous data, utilizing a hierarchical architecture and modal attention fusion mechanism, and combining FastText and Transformer encoders, the problem of insufficient detection accuracy caused by a single data source is solved, achieving more accurate system anomaly detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM DIGITAL INTELLIGENCE TECH CO LTD
- Filing Date
- 2023-01-13
- Publication Date
- 2026-06-09
AI Technical Summary
Existing automated anomaly detection methods rely on a single data source, resulting in insufficient accuracy and erroneous predictions in large-scale distributed systems.
A system detection method using multi-source heterogeneous data is proposed. It captures intramodal dependencies through a hierarchical architecture and generates global representations of log and metric data using a modal attention fusion mechanism. It combines FastText and Transformer encoders to model log semantics and metric patterns and uses causal convolutional networks to fuse heterogeneous data.
It achieves more accurate system anomaly detection by capturing different features and meaningful interactions of multimodal data through cross-modal representation learning, thereby improving detection accuracy.
Smart Images
Figure CN116089289B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a system detection method and apparatus based on multi-source heterogeneous data. Background Technology
[0002] In recent years, the scale and complexity of software systems have been growing rapidly, leading to a rise in the frequency of system anomalies. In real-world scenarios, service providers use automated anomaly detection to ensure the reliability of software systems. The basic data for software system anomaly detection comes from monitoring data of various functions, such as business metrics, logs, alerts, and traces. Metrics are real-valued time series that measure system state, such as response time and thread count, while logs are text messages used to record the runtime state of the system.
[0003] Existing automated anomaly detection relies on single metrics or log data, resulting in insufficient accuracy and numerous erroneous predictions. This is particularly true in large-scale distributed systems, where the accuracy of anomaly detection based on a single data source is even worse. Therefore, combining multiple monitoring data sources allows for more efficient utilization of runtime information to analyze system status. Summary of the Invention
[0004] In view of this, the present invention aims to provide a system detection method and apparatus based on multi-source heterogeneous data, which captures intramodal dependencies through a hierarchical architecture and generates a global representation of log and indicator data through a modal attention fusion mechanism to achieve more accurate anomaly judgment.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] In a first aspect, the present invention provides a system detection method based on multi-source heterogeneous data, comprising the following steps:
[0007] Obtain log text data, parse the log text data to extract log events, convert the log events into numerical vectors and represent them as log vectors;
[0008] Obtain time-series data of indicators, model the segment-level indicators in a hierarchical manner, and embed the extracted indicators into a D-dimensional feature representation.
[0009] Based on heterogeneous representation fusion, the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data are input into the fusion module to perform heterogeneous data fusion.
[0010] The results of system anomaly detection are obtained by calculating and inferring prediction results through fully connected layer and Softmax layer functions.
[0011] As a further aspect of the present invention, before acquiring log text data and indicator time-series data, log text data and indicator time-series data are acquired from the current heterogeneous monitoring data based on the historical extraction mode, and features are captured from the current heterogeneous monitoring data for anomaly detection.
[0012] As a further aspect of the present invention, parsing the log text data to extract log events includes:
[0013] Convert unstructured log messages in log text data into structured log events;
[0014] The Drain parser is used to extract log events, which are then sorted according to their timestamps to obtain log events arranged in chronological order.
[0015] Log events are converted into numerical vectors with lexical and semantic information, and FastText is used to capture the inherent semantic relationships of log words.
[0016] The obtained log context semantics are modeled and a log representation is generated, which is then represented as a log vector.
[0017] As a further aspect of the present invention, the trained FastText is used to map each tag to an E-dimensional vector, converting the logarithmic event x into a list of tag embeddings. Where w is the number of event markers;
[0018] FastText is also used to average all elements to obtain an embedding vector. The logarithmic sequence x 1:L Represented by sentence embedding
[0019] As a further aspect of the present invention, the embedding vector obtained based on FastText is used as the input of the sequence encoder. The input of the sequence encoder consists of two Transformer encoder layers, and the output is mapped to a D-dimensional feature space through a fully connected layer to obtain a logarithmic representation of a block.
[0020] As a further aspect of the present invention, time series data of indicators are obtained, and the indicators of the segment-level pattern are modeled in a hierarchical manner, including an intra-aspect coding and an inter-aspect coding.
[0021] The intra-aspect encoding includes: decomposing the indicators into Y groups according to their respective aspects; inputting indicators of the same aspect as a single MTS into an intra-aspect encoder composed of a multi-layer causal convolutional network; and after padding and segmentation, the intra-aspect encoder outputs the feature vector h of Y. m Max pooling is performed on the features, and the outputs are stacked to form a latent feature vector.
[0022] The inter-aspect encoding includes: converting the H output from the in-direction encoder... m As an MTS input to the inter-direction encoder, the index X within the block m Embedded into D-dimensional representation
[0023] As a further aspect of the present invention, heterogeneous data fusion is performed by inputting the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data into a fusion module, including:
[0024] Both log text data and time-series indicator data are embedded into a D-dimensional feature space and input into the fusion module;
[0025] The first attention layer, Attn-α, uses log representation R. l As a query, the metric represents R. m Used as Key and Value to match log events of metric changes;
[0026] The second attention layer Attn-β, R m For Query, R l Use key and value to find metric discrepancies consistent with log content;
[0027] The outputs from Attn-α and Attn-β are concatenated in D-dimensional space, with each data block forming a global representation. Cross-attention mechanism explicitly preserves meaningful internal connections by directly connecting Query and Value, enabling heterogeneous data fusion.
[0028] As a further aspect of the present invention, the inference prediction results are calculated through fully connected layers and Softmax layer functions, including: representing the intra-process block R... g The results are provided to the fully connected layer and the softmax layer to calculate the inference prediction results, using the following formula:
[0029]
[0030] Among them, the output The states are normal or abnormal, U and V are the weight matrices for learning, b and c are the bias terms, and σ is the activation function.
[0031] Secondly, in one aspect of the present invention, a system detection device based on multi-source heterogeneous data is provided, the system detection device based on multi-source heterogeneous data being used to execute the above-mentioned system detection method based on multi-source heterogeneous data; the system includes:
[0032] The data acquisition module is used to acquire log text data and indicator time series data from the current heterogeneous monitoring data based on historical extraction patterns;
[0033] The log data modeling module is used to model the lexical semantics and sequence dependencies of logs using the FastText algorithm and Transformer encoder, embedding log text data into a D-dimensional feature space;
[0034] The indicator data modeling module is used to learn a representation based on a causal convolutional network using a hierarchical encoder, to model the segment-level pattern indicators in a hierarchical manner, and to embed the extracted indicators into a D-dimensional feature representation.
[0035] The heterogeneous representation fusion module is used to input the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data into the fusion module for heterogeneous data fusion based on heterogeneous representation fusion.
[0036] The inference and prediction module is used to calculate the inference and prediction results through fully connected layer and Softmax layer functions to obtain the results of system anomaly detection.
[0037] As a further embodiment of the present invention, the log data modeling module further includes a logarithmic vector representation module, which is used to parse the acquired log text data to extract log events, convert the log events into numerical vectors, and perform logarithmic vector representation.
[0038] Thirdly, in another aspect provided by the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor loads and executes the computer program to implement the steps of a system detection method based on multi-source heterogeneous data.
[0039] Fourthly, in another aspect provided by the present invention, a storage medium is provided storing a computer program, which, when loaded and executed by a processor, implements the steps of the system detection method based on multi-source heterogeneous data.
[0040] The technical solution provided by this invention has the following beneficial effects:
[0041] The system detection method and apparatus based on multi-source heterogeneous data provided by this invention utilizes a hierarchical structure to fuse log semantics and indicator patterns to learn a global representation of the system state. It uses a cross-modal attention mechanism to capture different features and meaningful interactions of multi-modal data, thereby achieving accurate system anomaly detection. This invention performs anomaly detection by capturing meaningful features from heterogeneous data. It not only utilizes the semantic information of log data and the temporal dependence of indicator data, but also learns cross-modal representations through an attention fusion mechanism to narrow the gap and provide a more reasonable detection judgment.
[0042] These or other aspects of the invention will become more apparent from the following description of embodiments. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. In the drawings:
[0044] Figure 1 This is a flowchart of a system detection method based on multi-source heterogeneous data according to an embodiment of the present invention.
[0045] Figure 2 This is a structural block diagram of the ADASIS high-precision map system in a system detection method based on multi-source heterogeneous data according to an embodiment of the present invention.
[0046] Figure 3 This is a flowchart of a system detection method based on multi-source heterogeneous data for testing high-precision map reconstruction software, according to an embodiment of the present invention.
[0047] Figure 4 This is a structural block diagram of a computer device in a system detection method based on multi-source heterogeneous data according to an embodiment of the present invention. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0049] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.
[0050] The technical solutions in the exemplary embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described exemplary embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] Existing automated anomaly detection relies on single metrics or log data, resulting in insufficient accuracy and numerous erroneous predictions. This is particularly true in large-scale distributed systems, where the accuracy of anomaly detection based on a single data source is even worse. Therefore, combining multiple monitoring data sources allows for more efficient utilization of runtime information to analyze system status.
[0052] The present invention provides a system detection method and apparatus based on multi-source heterogeneous data. It realizes software system anomaly detection based on the differentiated representation of multi-source data, captures intramodal dependencies through a layered architecture, and generates a global representation of log and indicator data through a modal attention fusion mechanism to achieve more accurate anomaly judgment.
[0053] Specifically, the embodiments of this application will be further described below with reference to the accompanying drawings.
[0054] See Figure 1 As shown, one embodiment of the present invention provides a system detection method based on multi-source heterogeneous data, comprising the following steps:
[0055] Step S10: Obtain log text data, parse the log text data to extract log events, convert the log events into numerical vectors and perform log vector representation;
[0056] Step S20: Obtain time series data of indicators, model the segment-level indicators in a hierarchical manner, and embed the extracted indicators into D-dimensional feature representation.
[0057] Step S30: Based on heterogeneous representation fusion, the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data are input into the fusion module for heterogeneous data fusion;
[0058] Step S40: Calculate the inference prediction results through the fully connected layer and Softmax layer functions to obtain the system anomaly detection results.
[0059] This invention presents a system detection method based on multi-source heterogeneous data. It learns discriminative representations from heterogeneous system state data based on logs and metrics, uses a hierarchical architecture to capture intra-modal dependencies, and then generates the most discriminative representation through modal attention fusion. Specifically, for log data, the network employs the FastText algorithm and a Transformer encoder to model the lexical semantics and sequential dependencies of the logs. For metric data, the network uses a hierarchical encoder to learn a representation based on a causal convolutional network to abstract intra-aspect temporal dependencies, cross-relationships, and inter-aspect correlations. For the vector representations obtained in these two steps, a modal attention mechanism is designed in the network to learn global representations, preserving meaningful intra-modal and inter-modal information. Finally, the system anomaly detection results are obtained through fully connected layers and a Softmax function.
[0060] In this embodiment, before acquiring log text data and indicator time-series data, log text data and indicator time-series data are obtained from the current heterogeneous monitoring data based on the historical extraction mode. Features are captured from the current heterogeneous monitoring data for anomaly detection. See [link to previous document]. Figure 1 and Figure 2 As shown, the overall process of the system detection method based on multi-source heterogeneous data in this embodiment of the invention includes the following steps:
[0061] (1) For the acquired log text data, there are three steps in sequence, which aim to learn the log information representation from both lexical and semantic aspects and map each original log sequence to a low-dimensional representation.
[0062] The first step is log parsing. Since the raw log messages may contain variables that hinder subsequent analysis, it's necessary to convert the unstructured log messages into structured log events. Here, the Drain parser is used to extract the log events, and then they are sorted according to the log timestamps to obtain log events arranged in chronological order.
[0063] Then, log vectorization is performed, converting log events into numerical vectors with lexical and semantic information. FastText is used to capture the inherent semantic relationships of log words, and the obtained log context semantics are modeled and a log representation is generated, which is then represented as a log vector. In this embodiment, after training, FastText maps each tag to an E-dimensional vector, converting the log event x into a tag embedding list. Where w is the number of event markers. Then, the average of all elements is taken to obtain an embedding. Thus the logarithmic sequence x 1:L Sentence embedding can be used to represent
[0064] Finally, logarithmic vector representation is performed to model the log context semantics obtained in the previous step and generate a log representation. The embedding vector obtained in the previous step is used as the input to the sequence encoder. In this embodiment, the embedding vector obtained based on FastText is used as the input to the sequence encoder, which consists of two Transformer encoder layers to capture contextual dependencies across events. Then, a fully connected layer maps the output to a D-dimensional feature space to obtain a logarithmic representation of a block. If the sequence is too long, it will be divided into subsequences of fixed size; if it is too short, it will be padded with zeros.
[0065] (2) For acquiring time series data of indicators, the indicators of segment-level pattern are modeled in a hierarchical manner. The processing steps include intra-aspect coding and inter-aspect coding.
[0066] In the intra-aspect encoding step, since the indicator patterns describing the same aspect of the system have certain similarities, they should be analyzed together as multivariate time series (MTS). Indicator patterns describing different aspects may differ and should be input into separate models for processing. That is, modeling the indicator standards for the same aspect while simultaneously modeling the indicator standards for different aspects separately. Here, the model employs one-dimensional causal convolution, addressing the issues of information leakage and the inability to model sequential dependencies in traditional convolutional networks through parallelization, lightweight design, and accuracy.
[0067] Therefore, intra-aspect encoding includes: decomposing the metrics into Y groups according to their respective aspects; inputting metrics of the same aspect as a single MTS into an intra-aspect encoder composed of a multi-layer causal convolutional network; after padding and segmentation, the intra-aspect encoder outputs the feature vector h of Y. m Max pooling is performed on the features, and the outputs are stacked to form a latent feature vector.
[0068] In the inter-aspect encoding step, when anomalies occur, metrics from different aspects still exhibit some inter-aspect correlations. Therefore, this module utilizes causal convolution to learn inter-aspect characteristics. Complex patterns are modeled by capturing multi-level information. Thus, inter-aspect encoding includes: converting the H output from the in-direction encoder... m As an MTS input to the inter-direction encoder, it models the correlation between aspects, with the in-block index X... m Embedded into D-dimensional representation
[0069] (3) In order to make up for the time and semantic differences between log representation and indicator representation, this embodiment also has a fusion step with a cross-attention mechanism to alleviate the problem of lack of information or oversensitivity of single-source data, i.e. heterogeneous representation fusion.
[0070] In steps (1) and (2), both log and metric data are embedded into a D-dimensional feature space, and these representations are input into the fusion module. In this embodiment, heterogeneous representation fusion is performed by inputting the log vector representation of the log text data and the D-dimensional feature representation of the metric time series data into the fusion module for heterogeneous data fusion, including:
[0071] Both log text data and time-series indicator data are embedded into a D-dimensional feature space and input into the fusion module;
[0072] The first attention layer, Attn-α, uses log representation R. l As a query, the metric represents R. m Used as Key and Value to match log events of metric changes;
[0073] The second attention layer Attn-β, R m For Query, R l Using the key and value, to find metric discrepancies consistent with log content, the formula is as follows:
[0074] Fuse(Q,K,V)=tanh([softmax(QW s K T )V;Q]W a )
[0075] Among them, W a and W s These are learnable parameters;
[0076] The outputs from Attn-α and Attn-β are concatenated in D-dimensional space, with each data block forming a global representation. The definition is as follows:
[0077] R g=[Fuse(R l ,R m ,R m Fuse(R) m ,R l ,R l )]
[0078] Cross-attention mechanism explicitly preserves meaningful internal connections by directly connecting Query and Value, enabling heterogeneous data fusion.
[0079] In this embodiment, this cross-attention mechanism explicitly preserves meaningful internal connections by directly connecting Query and Value, allowing the global representation to not only retain shared information and cross-modal interactions, but also to preserve intra-modal dependencies and inference features due to the complementary relationship between logs and metrics.
[0080] (4) Finally, the inference prediction results are calculated using the fully connected layer and the Softmax layer functions, including: representing the in-process block R g The results are provided to the fully connected layer and the softmax layer to calculate the inference prediction results, using the following formula:
[0081]
[0082] Among them, the output The states are normal or abnormal, U and V are the weight matrices for learning, b and c are the bias terms, and σ is the activation function.
[0083] The system detection method based on multi-source heterogeneous data provided by this invention utilizes a hierarchical structure to fuse log semantics and indicator patterns to learn a global representation of the system state. It uses a cross-modal attention mechanism to capture different features and meaningful interactions of multi-modal data, thereby achieving accurate system anomaly detection. This invention performs anomaly detection by capturing meaningful features from heterogeneous data. It not only utilizes the semantic information of log data and the temporal dependence of indicator data, but also learns cross-modal representations through an attention fusion mechanism to narrow the gap and provide a more reasonable detection judgment.
[0084] See Figure 3 As shown, one embodiment of the present invention provides a system detection device based on multi-source heterogeneous data, which is used to execute the above-described system detection method based on multi-source heterogeneous data; the system includes:
[0085] The data acquisition module 100 is used to acquire log text data and indicator time-series data from the current heterogeneous monitoring data based on historical extraction patterns. Prior to acquiring the log text data and indicator time-series data, it acquires log text data and indicator time-series data from the current heterogeneous monitoring data based on historical extraction patterns, and captures features from the current heterogeneous monitoring data for anomaly detection.
[0086] The log data modeling module 200 is used to model the lexical semantics and sequential dependencies of logs using the FastText algorithm and Transformer encoder, embedding log text data into a D-dimensional feature space.
[0087] In some embodiments, the log data modeling module 200 further includes a log vector representation module 201, which is used to parse the acquired log text data to extract log events, convert the log events into numerical vectors, and perform log vector representation.
[0088] When using the log data modeling module 200 to parse the log text data and extract log events, the process is as follows: convert unstructured log messages in the log text data into structured log events; use the Drain parser to extract log events, sort them according to the log timestamps, and obtain log events arranged in chronological order; convert the log events into numerical vectors with lexical and semantic information, and use FastText to capture the inherent semantic relationships of log words; model the obtained log context semantics and generate a log representation, which is then represented as a log vector.
[0089] The indicator data modeling module 300 is used to learn a representation based on a causal convolutional network using a hierarchical encoder, to model the segment-level pattern indicators in a hierarchical manner, and to embed the extracted indicators into a D-dimensional feature representation.
[0090] The heterogeneous representation fusion module 400 is used to input the logarithmic vector representation of the log text data and the D-dimensional feature representation of the indicator time series data into the fusion module for heterogeneous data fusion based on heterogeneous representation fusion. In this embodiment, the process of inputting the logarithmic vector representation of the log text data and the D-dimensional feature representation of the indicator time series data into the fusion module for heterogeneous data fusion based on heterogeneous representation fusion is as follows: both the log text data and the indicator time series data are embedded into a D-dimensional feature space and input into the fusion module; the first attention layer Attn-α uses the log representation R l As a query, the metric represents R. m Used as Key and Value to match log events of metric changes; Second attention layer Attn-β, R m For Query, R lUse key and value to find metric discrepancies consistent with log content; concatenate the outputs from Attn-α and Attn-β in D-dimensional space, with each data block forming a global representation. Cross-attention mechanism explicitly preserves meaningful internal connections by directly connecting Query and Value, enabling heterogeneous data fusion.
[0091] The inference prediction module 500 is used to calculate the inference prediction result through fully connected layer and Softmax layer functions to obtain the system anomaly detection result. In this embodiment, calculating the inference prediction result through fully connected layer and Softmax layer functions includes: representing the intra-process block R... g The results are provided to the fully connected layer and the softmax layer to calculate the inference prediction results, using the following formula:
[0092]
[0093] Among them, the output The states are normal or abnormal, U and V are the weight matrices for learning, b and c are the bias terms, and σ is the activation function.
[0094] It should be understood that although the above description follows a certain order, these steps are not necessarily executed in that order. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, some steps in this embodiment may include multiple steps or multiple stages, which are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of the steps or stages in other steps.
[0095] In one embodiment, see Figure 4 As shown, an embodiment of the present invention also provides a computer device 1000, including at least one processor 1002 and a memory 1001 communicatively connected to the at least one processor 1002. The memory 1001 stores instructions executable by the at least one processor 1002. The instructions are executed by the at least one processor 1002 to cause the at least one processor 1002 to perform the system detection method based on multi-source heterogeneous data. When the processor 1002 executes the instructions, it implements the steps in the above-described method embodiments:
[0096] Obtain log text data, parse the log text data to extract log events, convert the log events into numerical vectors and represent them as log vectors;
[0097] Obtain time-series data of indicators, model the segment-level indicators in a hierarchical manner, and embed the extracted indicators into a D-dimensional feature representation.
[0098] Based on heterogeneous representation fusion, the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data are input into the fusion module to perform heterogeneous data fusion.
[0099] The results of system anomaly detection are obtained by calculating and inferring prediction results through fully connected layer and Softmax layer functions.
[0100] In exemplary embodiments of the present invention, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention can also be implemented as a program product comprising program code, wherein when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the system detection method based on multi-source heterogeneous data according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above, the system detection method based on multi-source heterogeneous data comprising:
[0101] Obtain log text data, parse the log text data to extract log events, convert the log events into numerical vectors and represent them as log vectors;
[0102] Obtain time-series data of indicators, model the segment-level indicators in a hierarchical manner, and embed the extracted indicators into a D-dimensional feature representation.
[0103] Based on heterogeneous representation fusion, the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data are input into the fusion module to perform heterogeneous data fusion.
[0104] The results of system anomaly detection are obtained by calculating and inferring prediction results through fully connected layer and Softmax layer functions.
[0105] In exemplary embodiments of the present invention, a program product for implementing the above-described method according to embodiments of the present invention is described, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0106] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0107] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.
[0108] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0109] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.
[0111] In summary, the system detection method and apparatus based on multi-source heterogeneous data provided by this invention utilizes a hierarchical structure to fuse log semantics and indicator patterns to learn a global representation of the system state, and employs a cross-modal attention mechanism to capture different features and meaningful interactions of multi-modal data, thereby achieving accurate system anomaly detection. This invention performs anomaly detection by capturing meaningful features from heterogeneous data, utilizing not only the semantic information of log data and the temporal dependence of indicator data, but also learning cross-modal representations through an attention fusion mechanism to narrow the gap and provide more reasonable detection judgments.
[0112] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A system detection method based on multi-source heterogeneous data, characterized in that, Includes the following steps: Obtain log text data, parse the log text data to extract log events, convert the log events into numerical vectors and represent them as log vectors; Obtain time-series data of indicators, model the segment-level indicators in a hierarchical manner, and embed the extracted indicators into a D-dimensional feature representation. Based on heterogeneous representation fusion, the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data are input into the fusion module to perform heterogeneous data fusion. The inference and prediction results are calculated using fully connected layer and Softmax layer functions to obtain the results of system anomaly detection; Parsing the log text data to extract log events includes: Convert unstructured log messages in log text data into structured log events; The Drain parser is used to extract log events, which are then sorted according to their timestamps to obtain log events arranged in chronological order. Log events are converted into numerical vectors with lexical and semantic information, and FastText is used to capture the inherent semantic relationships of log words. The obtained log context semantics are modeled and a log representation is generated, which is then represented as a log vector. The trained FastText is used to map each token to an E-dimensional vector, transforming the logarithmic event x into a list of token embeddings. ,in, The number of markers for the event; FastText is also used to average all elements to obtain an embedding vector. , logarithmic sequence Represented by sentence embedding .
2. The system detection method based on multi-source heterogeneous data as described in claim 1, characterized in that, Before acquiring log text data and time-series indicator data, log text data and time-series indicator data are obtained from the current heterogeneous monitoring data based on the historical extraction mode, and features are captured from the current heterogeneous monitoring data for anomaly detection.
3. The system detection method based on multi-source heterogeneous data as described in claim 2, characterized in that, The embedding vector obtained from FastText is used as the input to the sequence encoder. The input to the sequence encoder consists of two Transformer encoder layers. The output is mapped to a D-dimensional feature space through a fully connected layer, resulting in a logarithmic representation of a block. .
4. The system detection method based on multi-source heterogeneous data as described in claim 3, characterized in that, Acquire time-series data of indicators and model the segment-level indicators in a hierarchical manner, including intra-aspect coding and inter-aspect coding; The intra-aspect encoding includes: decomposing the indicators into Y groups according to their respective aspects; inputting indicators of the same aspect as a single MTS into an intra-aspect encoder composed of a multi-layer causal convolutional network; and after padding and segmentation, the intra-aspect encoder outputs the feature vector of Y. Max pooling is performed on the features, and the outputs are stacked to form a latent feature vector. ; The inter-aspect encoding includes: converting the output of the intra-aspect encoder... As an MTS input to the inter-aspect encoder, the intra-block index Embedded into D-dimensional representation .
5. The system detection method based on multi-source heterogeneous data as described in claim 4, characterized in that, Based on heterogeneous representation fusion, the logarithmic vector representation of the log text data and the D-dimensional feature representation of the indicator time series data are input into the fusion module for heterogeneous data fusion, including: Both log text data and time-series indicator data are embedded into a D-dimensional feature space and input into the fusion module; The first attention layer, Attn-α, uses log representation. As a query, the metric represents Used as Key and Value to match log events of metric changes; The second attention layer, Attn-β, For Query, Use key and value to find metric discrepancies consistent with log content; The outputs from Attn-α and Attn-β are concatenated in D-dimensional space, with each data block forming a global representation. The cross-attention mechanism explicitly preserves meaningful internal connections by directly connecting the query and value, enabling the fusion of heterogeneous data.
6. The system detection method based on multi-source heterogeneous data as described in claim 5, characterized in that, The inference prediction results are calculated using fully connected layers and Softmax layer functions, including: representing in-process blocks. The results are provided to the fully connected layer and the softmax layer to calculate the inference prediction results, using the following formula: Among them, the output This indicates whether the state is normal or abnormal. U and V are the weight matrices for learning, and b and c are the bias terms. It is an activation function.
7. A system detection device based on multi-source heterogeneous data, characterized in that, The system detection device based on multi-source heterogeneous data is used to execute the system detection method based on multi-source heterogeneous data according to any one of claims 1-6; The system includes: The data acquisition module is used to acquire log text data and indicator time series data from the current heterogeneous monitoring data based on historical extraction patterns; The log data modeling module is used to model the lexical semantics and sequence dependencies of logs using the FastText algorithm and Transformer encoder, embedding log text data into a D-dimensional feature space; The indicator data modeling module is used to learn a representation based on a causal convolutional network using a hierarchical encoder, to model the segment-level pattern indicators in a hierarchical manner, and to embed the extracted indicators into a D-dimensional feature representation. The heterogeneous representation fusion module is used to input the log vector representation of the log text data and the D-dimensional feature representation of the indicator time series data into the fusion module for heterogeneous data fusion based on heterogeneous representation fusion. The inference and prediction module is used to calculate the inference and prediction results through fully connected layer and Softmax layer functions to obtain the results of system anomaly detection.
8. The system detection device based on multi-source heterogeneous data as described in claim 7, characterized in that, The log data modeling module also includes a logarithmic vector representation module, which is used to parse the acquired log text data to extract log events, convert the log events into numerical vectors, and perform logarithmic vector representation.