Micro-service anomaly detection method and device

Abstract

The invention relates to the field of microservice system anomaly detection, in particular to a microservice anomaly detection method and device. The method comprises the steps of obtaining historical data of a micro-service system, and obtaining first modal data and a log time sequence matrix based on historical service index data, historical log data and historical call chain data in the historical data; obtaining a first modal feature matrix and a second modal feature matrix according to the first modal data and the log time sequence matrix; obtaining a historical anomaly detection result according to the first modal feature matrix and the second modal feature matrix; obtaining first and second modal reconstruction data according to the first and second modal feature matrixes; optimizing the language model and the classification network to obtain an optimized language model and an optimized classification network; and preprocessing the collected real-time data, and inputting the preprocessed real-time data into the optimized language model and the classification network for anomaly detection to obtain a real-time anomaly detection result. The method is used for realizing accurate positioning and detection of the abnormity of the micro-service system.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection in microservice systems, and more specifically, to a method and apparatus for anomaly detection in microservice systems. Background Technology

[0002] Microservice architecture is a design philosophy that decouples complex applications into independent services through functional decoupling. It breaks down traditional monolithic architectures into loosely coupled, fine-grained service units, each running in an independent process and interacting through lightweight communication mechanisms. Compared to traditional monolithic architectures, microservice architectures significantly improve the flexibility, scalability, and fault isolation capabilities of the entire microservice system due to their independent deployment, elastic scaling, and rapid iteration, and have become the mainstream approach for building large-scale enterprise applications. However, the dynamic nature of this microservice architecture also brings new reliability challenges: due to the complex call dependencies between service units, anomalies in a single service can cascade along the call chain, affecting multiple downstream components, making fault localization and root cause analysis particularly difficult.

[0003] To ensure the health of microservice systems, existing technologies commonly employ multimodal operational data for monitoring and diagnosis. This primarily includes three core data sources: service metrics reflecting resource utilization and performance, log data recording runtime events and states, and call chain data tracing request paths. These different modalities characterize microservice system behavior from various dimensions, naturally possessing complementary information. Current methods for anomaly detection in microservice systems mainly fall into two categories: manual detection and automated detection. Manual detection relies on operations personnel to comprehensively analyze multimodal data; however, as the scale of microservice systems expands and service topology complexity increases, manual detection struggles to balance efficiency and accuracy. Automated detection, while leveraging machine learning models to improve efficiency, generally suffers from reliance on single-modal data, i.e., isolated analysis based solely on service metrics, log data, or call chain data, lacking cross-modal verification capabilities and prone to false positives and false negatives. Although recent research has attempted to integrate multimodal data, early and / or mid-stage integration strategies often lead to noise diffusion, actually reducing detection accuracy. Summary of the Invention

[0004] This invention provides a microservice anomaly detection method, system, electronic device, and storage medium, enabling accurate location and detection of anomalies in microservice systems.

[0005] According to a first aspect of this application, a microservice anomaly detection method is provided, the method comprising: Obtain historical data from the microservice system, including historical service metric data, historical log data, historical call chain data, and historical anomaly tags corresponding to the historical data; The historical service indicator data and the historical call chain data are preprocessed to obtain the first modality data, and the historical log data are preprocessed to obtain the log time series matrix; The first modality data and the log time series matrix are respectively input into a preset language model for feature extraction to obtain the corresponding first modality feature matrix and second modality feature matrix; The first modality feature matrix and the second modality feature matrix are input into a preset classification network model for anomaly detection to obtain historical anomaly detection results; The first modal feature matrix and the second modal feature matrix are respectively input into a preset modal reconstruction model for modal reconstruction to obtain the corresponding first modal reconstruction data and second modal reconstruction data; The language model and the classification network are optimized based on the historical anomaly detection results, the historical anomaly labels, the first modality data, the log time series matrix, the first modality feature matrix, the second modality feature matrix, and the preset total loss function to obtain an optimized language model and an optimized classification network. The real-time service metric data, real-time log data, and real-time call chain data obtained from the microservice system are preprocessed and then input into the optimized language model and the optimized classification network for anomaly detection to obtain real-time anomaly detection results.

[0006] Understandably, by integrating multi-source data such as service metrics, logs, and call chains to construct a dual-modal feature representation, and combining language model feature extraction, classification network anomaly detection, and modal reconstruction to jointly optimize the models and networks used, high-precision real-time detection of fine-grained anomalies in microservice systems is achieved, effectively improving detection accuracy and generalization ability.

[0007] Optionally, obtaining historical data from the microservice system includes: A preset time window, wherein the time window comprises a number of time steps with equal time intervals; Obtain historical service indicator data, historical log data, and historical call chain data of the time window corresponding to the time step in the historical data, and obtain historical anomaly tags corresponding to the time step in the historical data.

[0008] Understandably, by pre-setting a time window containing equal time intervals, the system can accurately synchronize historical service indicator data, historical log data, and historical call chain data in time, and associate the anomaly tags corresponding to the historical data to build a high-resolution, multimodal training sample set, effectively supporting fine-grained anomaly detection in microservice systems.

[0009] Optionally, the preprocessing of the historical service indicator data and the historical call chain data to obtain the first modal data includes: The historical service indicator data is preprocessed to obtain the service indicator time series matrix; The historical call chain data is preprocessed to obtain the call chain timing matrix; The service metric time series matrix and the call chain time series matrix are fused to obtain the first modality data.

[0010] Understandably, by performing time-series matrix processing on historical service indicator data and historical call chain data respectively to obtain service indicator time-series matrix and call chain time-series matrix, and then merging the service indicator time-series matrix and call chain time-series matrix, multi-source runtime data can be efficiently integrated, significantly improving data dimensionality and consistency, and providing more comprehensive and accurate feature input for subsequent anomaly detection.

[0011] Optionally, the historical service indicator data is preprocessed to obtain a service indicator time series matrix, including: Obtain the corresponding triplet data based on the historical service indicator data. The triplet data includes service type, service indicator, and time step. Arrange all the triplet data in order according to the corresponding time step. Construct an initial service indicator time series matrix based on the ordered triplet data. Normalize the initial service indicator time series matrix to obtain the service indicator time series matrix. And / or, preprocessing the historical call chain data to obtain a call chain time sequence matrix, including: Obtain corresponding quadruple data based on the historical call chain data. The quadruple data includes the caller service type, the callee service type, the call type, and a timestamp. Arrange all the quadruple data in order according to their corresponding timestamps. Calculate the average span time interval in units of the time steps based on the timestamps corresponding to the quadruple data. Construct an initial call chain timing matrix based on the quadruple data, the time steps, and the average span time interval. Normalize the initial call chain timing matrix to obtain the final call chain timing matrix.

[0012] Understandably, by analyzing historical service indicator data and historical call chain data, the original operational information is transformed into a standardized time series matrix, accurately preserving the dynamic behavioral characteristics of the microservice system. Furthermore, normalization effectively eliminates dimensional differences, significantly improving data purity and comparability. This provides high-resolution, low-noise feature inputs for subsequent multimodal fusion and anomaly detection, thereby enhancing the model's accuracy in identifying hidden anomalies.

[0013] Optionally, the historical log data is preprocessed to obtain a log template time-series matrix, including: Based on a preset parsing algorithm, the corresponding variable fields, anomaly risk levels, and description fields in the historical log data are obtained. The variable fields are converted into placeholders, and formatted historical log data is constructed based on the placeholders, anomaly risk levels, and description fields. The formatted historical log data is vectorized to obtain a log vector set, wherein the log vector set includes at least one log vector; Each log vector in the log vector set is input into a preset word vector model for processing to obtain distributed word vectors; Aggregate all the distributed word vectors to obtain the log semantic vector corresponding to the historical log data; The log semantic vectors corresponding to all the historical log data are stacked to obtain the log semantic matrix; Obtain the log occurrence matrix based on the historical log data and the time steps corresponding to the historical log data; The initial log time series matrix is ​​obtained by multiplying the transpose of the log semantic matrix and the log occurrence matrix; The initial log time series matrix is ​​normalized to obtain the log time series matrix.

[0014] Understandably, by using templated parsing and vectorized encoding, unstructured log data is transformed into a temporal representation that combines semantic depth and spatiotemporal correlation. That is, by fusing the log semantic matrix and the log occurrence matrix, a high-dimensional temporal structure reflecting the evolution of log patterns is constructed. Combined with normalization operations, the data standardization is significantly improved, thereby enhancing the model's ability to perceive hidden abnormal behaviors in log sequences, such as sudden error logs and semantic mutations, and providing high-value feature inputs for fine-grained anomaly detection in microservice systems.

[0015] Optionally, the step of inputting the first modality data and the log time series matrix into a preset language model for feature extraction to obtain the corresponding first modality feature matrix and second modality feature matrix includes: The first modality data is input into the first neural network to obtain the first embedding matrix, and the first embedding matrix is ​​input into the preset first language model to extract features and obtain the first modality feature matrix; The log time series matrix is ​​input into the second neural network to obtain the second embedding matrix, and the second embedding matrix is ​​input into the preset second language model to extract features and obtain the second modality feature matrix; The first neural network and the second neural network each include a linear neural network layer and a ReLU activation function connected in sequence.

[0016] Understandably, by cascading a lightweight neural network and a language model, deep features of the first modality data and log time-series matrix are efficiently extracted, significantly compressing the dimensionality of the original data while retaining key anomaly features. Specifically, a single-layer linear network combined with the ReLU activation function quickly generates low-dimensional embeddings, while the language model delves into temporal dependencies, ensuring that feature representation possesses both semantic depth and dynamic continuity. This significantly enhances the input quality for subsequent anomaly detection, enabling accurate capture of subtle anomalies in the microservice system's operation, while also optimizing computational efficiency and improving the real-time performance and robustness of detection.

[0017] Optionally, the step of inputting the first modal feature matrix and the second modal feature matrix into a preset modal reconstruction model for modal reconstruction to obtain corresponding first modal reconstruction data and second modal reconstruction data includes: The first modal feature matrix is ​​input into a preset first modal reconstruction model for modal reconstruction to obtain first modal reconstruction data, wherein the modal size of the first modal reconstruction data is the same as that of the first modal data. The second modal feature matrix is ​​input into a preset second modal reconstruction model to perform modal reconstruction, thereby obtaining second modal reconstruction data, wherein the modal size of the second modal reconstruction data is the same as that of the second modal data.

[0018] Understandably, by reconstructing the feature matrix in reverse using a bimodal reconstruction model, the reconstruction model is forced to learn a low-dimensional and compact representation, which effectively constrains the feature space and preserves the key data distribution characteristics, significantly reduces the model's sensitivity to training noise, greatly alleviates the overfitting problem, and improves the robustness of feature representation.

[0019] Optionally, the step of inputting the first modality feature matrix and the second modality feature matrix into a preset classification network model for anomaly detection to obtain historical anomaly detection results includes: The first modality feature matrix and the second modality feature matrix are concatenated column-wise to obtain the fusion matrix; The fusion matrix is ​​divided based on a preset ratio to obtain the pre-data and detection data of the fusion matrix. The pre-processed data and the detection data are input into a preset classification network model for detection to obtain the probability of an anomaly occurring at each time step of the microservice system. If the probability of an anomaly occurring at the time step exceeds a preset anomaly threshold, the microservice system is determined to have an anomaly at that time step. If the probability of an anomaly occurring at the time step does not exceed the preset anomaly threshold, the microservice system is determined to have no anomaly at that timestamp.

[0020] Understandably, by dividing the data into pre-defined data and detection data according to a preset ratio, the pre-defined data serves only as contextual input to capture early warning patterns of anomalies without participating in real-time probability calculations, effectively enhancing the model's ability to perceive subtle anomaly signals. The detection data, after processing by a classification network, outputs refined anomaly probabilities. Combined with an anomaly threshold determination mechanism, this accurately distinguishes between real faults and environmental noise, significantly reducing the false alarm rate.

[0021] Optionally, the total loss function is constructed based on the first modality reconstruction loss function, the second modality reconstruction loss function, and the cross-entropy classification loss value; The total loss function for: Wherein, the first mode reconstruction loss function for: The second mode reconstruction loss function for: This represents the first modality data. This represents the reconstructed data of the first mode. Represents the log time series matrix. This represents the reconstructed data from the second mode; This represents the cross-entropy classification loss value, which is calculated by performing cross-entropy loss on the historical anomaly detection results and the historical anomaly labels. This indicates the number of training rounds.

[0022] Understandably, the modality reconstruction and classification tasks are cleverly integrated through a dynamically weighted total loss function. The reconstruction loss function constrains the feature space to preserve the original data distribution, effectively mitigating overfitting. In the early stages of training, the reconstruction loss is dominant, strengthening unsupervised feature learning capabilities. Later, the weights of the classification loss values ​​are gradually increased, achieving a smooth transition to supervised anomaly detection. This progressive optimization strategy significantly enhances the model's ability to capture subtle anomaly patterns, greatly improving the detection accuracy and generalization robustness of the microservice system in complex runtime environments.

[0023] According to a second aspect of this application, a microservice anomaly detection device is provided, the device comprising: The data acquisition module is used to acquire historical data of the microservice system. The historical data includes historical service indicator data, historical log data, historical call chain data, and historical anomaly tags corresponding to the historical data. The preprocessing module is used to obtain first modal data based on the historical service indicator data and the historical call chain data, and to obtain a log time series matrix based on the historical log data. The feature extraction module is used to input the first modality data and the log time series matrix into a preset language model for feature extraction, so as to obtain the corresponding first modality feature matrix and second modality feature matrix; The historical prediction module is used to input the first modality feature matrix and the second modality feature matrix into a preset classification network model for anomaly detection and to obtain historical anomaly detection results. The reconstruction module is used to input the first modal feature matrix and the second modal feature matrix into a preset modal reconstruction model to perform modal reconstruction, thereby obtaining the corresponding first modal reconstruction data and second modal reconstruction data. The optimization module is used to optimize the language model and the classification network based on the historical anomaly detection results, the historical anomaly labels, the first modality data, the log time series matrix, the first modality feature matrix, the second modality feature matrix, and a preset total loss function, so as to obtain an optimized language model and an optimized classification network. The anomaly detection module is used to preprocess real-time service indicator data, real-time log data, and real-time call chain data obtained from real-time collection of the microservice system, and then input them into the optimized language model and the optimized classification network for anomaly detection to obtain real-time anomaly detection results.

[0024] According to a third aspect of this application, an electronic device is provided, comprising: Memory, used to store one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the microservice anomaly detection method described in the first aspect above.

[0025] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the microservice anomaly detection method described in the first aspect above.

[0026] Based on any of the above aspects, the microservice anomaly detection method, system, electronic device, and storage medium provided in this application embodiment can achieve the following technical effects: By fully leveraging service metrics data, log data, and call chain data for anomaly detection in microservice systems, the dimensionality of the detection data is increased, thereby improving the accuracy of anomaly detection. Service metrics data reflects information such as response time and throughput of the microservice system; call chain data reflects information such as call relationships and cross-service latency; and log data reflects the operational flow of the microservice system. Using all three types of data for anomaly detection in microservice systems can capture anomaly characteristics from multiple perspectives, including performance fluctuations, call topology anomalies, and log semantic mutations. Compared to a single data source, the joint analysis of multi-dimensional data effectively reduces false positives or false negatives caused by data incompleteness, especially at fine time granularity, enabling more accurate location of hidden anomalies caused by complex interactions.

[0027] This paper describes a method for fusing service metric data and call chain data, and obtaining a log semantic matrix and log occurrence matrix based on log data to generate a log time-series matrix. This approach improves the efficiency and accuracy of data preprocessing. For service metric data and call chain data, time-series matrix construction and normalization are employed. Deep fusion of these two types of data preserves the dynamic characteristics of the time series while eliminating dimensional differences, thus enhancing data consistency. For log data, the log text is transformed into semantic vectors using a word vector model based on the log semantic matrix. The log occurrence matrix records the frequency of log occurrences at each time step. This not only compresses redundant information in unstructured logs but also efficiently correlates log semantics with occurrence patterns through matrix operations, enabling subsequent language models to quickly extract key anomaly signals. This preprocessing workflow balances computational efficiency and information fidelity, providing high-purity feature input for downstream models.

[0028] Using the reconstruction loss function as one of the building blocks of the total loss function can preserve the feature distribution of the modal data and the input data, reducing model overfitting. By introducing the modal reconstruction loss function and the classification loss value to jointly construct the total loss function, the reconstruction loss function can constrain the distance between the modal feature matrix and its reconstructed data, forcing the model to learn a compact representation of the data and forcibly preserving the core distribution characteristics of the original data. This effectively suppresses the model's overfitting to training set noise and enhances the robustness of feature extraction. Simultaneously, the total loss function employs a dynamic weighting strategy, emphasizing the reconstruction task in the early stages of training to stabilize feature learning, and gradually strengthening classification performance in the later stages. This achieves a balance between feature generality and task specificity, significantly reducing the model's variance on the test set while improving anomaly detection accuracy and enhancing its resistance to overfitting. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This diagram illustrates an application scenario of a microservice anomaly detection method provided in this embodiment.

[0031] Figure 2 This is a flowchart of a microservice anomaly detection method provided in this embodiment.

[0032] Figure 3 This is a flowchart for acquiring the first modal data provided in this embodiment.

[0033] Figure 4 This is a flowchart for preprocessing historical service indicator data provided in this embodiment.

[0034] Figure 5 This is a flowchart for preprocessing the historical log data provided in this embodiment.

[0035] Figure 6 This is a flowchart for preprocessing the historical call chain data provided in this embodiment.

[0036] Figure 7 This is a flowchart of an anomaly detection method provided in this embodiment.

[0037] Figure 8 This is a performance comparison chart between the method provided in this embodiment and existing methods.

[0038] Figure 9 This is a schematic diagram of the functional modules of a microservice anomaly detection device provided in this embodiment.

[0039] Figure 10 This is a schematic diagram of the structure of the electronic device provided in this embodiment. Detailed Implementation

[0040] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application. To better illustrate the following embodiments, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] In anomaly detection of microservice systems, existing methods suffer from false positives and false negatives due to isolated analysis of single-modal data. Furthermore, multimodal fusion strategies are prone to spreading noise in the early and / or middle stages, which, combined with the low efficiency of model training and inference, makes it difficult to meet the dual requirements of real-time monitoring and continuous delivery.

[0044] This embodiment provides a technical solution that can solve the above problems. The specific implementation of this application will be described in detail below with reference to the accompanying drawings.

[0045] An exemplary diagram illustrating an application scenario of a microservice anomaly detection method provided in this application embodiment is shown below. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100.

[0046] Understandably, the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smartphone terminal, personal computer, tablet computer, vehicle terminal, etc., but is not limited to these.

[0047] In one feasible implementation, server 100 and terminal 200 may each execute a microservice anomaly detection method provided in the embodiments of this application. Alternatively, the microservice anomaly detection method provided in the embodiments of this application may be executed partly in server 100 and partly in terminal 200.

[0048] like Figure 2 As shown, this embodiment provides a microservice anomaly detection method, which can be further divided into the following steps: S100. Obtain historical data of the microservice system, including historical service indicator data, historical log data, historical call chain data, and historical anomaly tags corresponding to the historical data; In this embodiment, the historical data is obtained from data generated during the historical operation of the microservice system or data generated during the trial operation of the microservice system. Specifically, the historical service metric data reflects the performance status and resource utilization of the microservice system, such as throughput, latency, error rate, and processor and memory usage; the historical log data reflects specific events, behavior records, exception information, and debugging details during the historical operation of the microservice system; and the historical call chain data reflects inter-service call dependencies, request path, and the time consumed at each stage.

[0049] Understandably, anomalies in a microservice system may occur at a specific timestamp. Historical data shows that the microservice system has been running for some time, and each historical timestamp contains corresponding anomaly information—for example, whether an anomaly occurred or not at that historical timestamp. Using these anomaly information as historical anomaly labels helps subsequent models better learn the data characteristics corresponding to microservice system anomalies, thereby improving the accuracy of microservice system anomaly detection.

[0050] Specifically, obtaining historical data from the microservice system includes: A preset time window, wherein the time window comprises a number of time steps with equal time intervals; Obtain historical service indicator data, historical log data, and historical call chain data of the time window corresponding to the time step in the historical data, and obtain historical anomaly tags corresponding to the time step in the historical data.

[0051] In this embodiment, it is necessary to obtain historical service indicator data, historical log data, historical call chain data of the time window, and corresponding historical anomaly tags for each time step of the time window in order to detect whether the microservice system has anomalies in each time step of the time window.

[0052] For example, the time window can be set to 20 seconds, containing 20 time steps, with a 1-second time interval between every two time steps. It is understood that calls in a microservice system do not necessarily occur at integer timestamps corresponding to the set time steps. That is, a collected historical call chain may appear a number of times within the time window that is not equal to the number of time steps, and the times of occurrence may not necessarily correspond to the timestamps of those time steps. Specifically, within a time window of 0-20 seconds, historical call chain data occurring within 1.5 seconds may be collected. In this embodiment, the 1.5 seconds can be used as a timestamp to represent historical call chain data collected at times other than the hour.

[0053] S200: Preprocess the historical service indicator data and the historical call chain data to obtain the first modality data, and preprocess the historical log data to obtain the log time series matrix; Specifically, such as Figure 3 As shown, the preprocessing of the historical service indicator data and the historical call chain data to obtain the first modality data may include the following steps: S211. Preprocess the historical service indicator data to obtain the service indicator time series matrix; Specifically, such as Figure 4 As shown, preprocessing the historical service indicator data to obtain the service indicator time series matrix may include the following steps: A1. Obtain the corresponding triplet data based on the historical service indicator data. The triplet data includes service type, service indicator, and time step. In this embodiment, the service type, service metric, and corresponding time step can be obtained from the historical service metric data. The service type refers to the service requested or required by each node in the microservice system; for example, the service type includes user service, inventory service, order service, etc. The service metric refers to preset metric values ​​corresponding to various monitoring metrics, such as a delay of a second or an error rate of b%.

[0054] For example, the triplet data can be [user service, delayed by a seconds, c seconds], [user service, error rate b%, b seconds], etc. Multiple triplet data representing different service metrics at the same time step can be obtained, and the triplet data corresponding to all time steps within the time window serve as the data foundation for service metrics in this application.

[0055] A2. Arrange all the triplet data in order according to the corresponding time steps; For example, the triplet data can be arranged in ascending order of time steps to conform to a certain time development pattern.

[0056] A3. Construct an initial service indicator time series matrix based on the triplet data arranged in the order of completion; In this embodiment, the initial service indicator time series matrix is ​​a matrix with time steps as rows, all [service type - service indicator] as columns, and the corresponding indicator values ​​as matrix elements. All triplet data are concatenated according to the construction method specified above to obtain the initial service indicator time series matrix.

[0057] A4. Normalize the initial service indicator time series matrix to obtain the service indicator time series matrix.

[0058] In this embodiment, the data in each column of the initial service indicator time series matrix is ​​normalized and mapped to the [0,1] interval to obtain the service indicator time series matrix.

[0059] Specifically, such as Figure 5 As shown, preprocessing the historical call chain data to obtain the call chain time sequence matrix may include the following steps: B1. Obtain the corresponding four-tuple data based on the historical call chain data. The four-tuple data includes the caller service type, the callee service type, the call type, and the timestamp. In this embodiment, the caller service type, the callee service type, the call type, and the timestamp can be obtained from historical call chain data. The caller service type refers to the business or functional type of the service initiating the call, such as user service, order service, payment service, etc.; the callee service type refers to the business or functional type of the service receiving the call, such as product service, inventory service, risk control service, etc.; the call type refers to the communication method or protocol for inter-service interaction, such as synchronous calls based on Hypertext Transfer Protocol (HTTP), asynchronous messages based on Message Queue (MQ), database queries, etc.

[0060] For example, the quadruple data can be [user service, product service, HTTP, c seconds], [order service, inventory service, database query, c seconds], etc. Multiple quadruple data sets from different call chains with the same timestamp can be obtained, and the quadruple data sets corresponding to all timestamps within a time window serve as the data foundation for call chain data in this application.

[0061] Understandably, a call action may not occur at every time step, nor may it necessarily occur at the exact time corresponding to that time step. Therefore, the time dimension of the collected call chain data is represented by timestamps. Timestamps have higher time precision than time steps. For example, the time interval between the first time step (1 second) and the second time step (2 seconds) could include multiple timestamps, such as a 1.5-second timestamp.

[0062] B2. Arrange all the quadruple data in order according to their corresponding timestamps; For example, the quadruple data can be arranged in ascending order of timestamps to conform to a certain time development pattern.

[0063] B3. Calculate the average span time interval in units of time steps based on the timestamps corresponding to the quadruple data; In this embodiment, since the data ultimately input into the model for processing is in time steps, it is necessary to convert the timestamps of the quadruple data into time steps. Specifically, the quadruple data contained in each time step can be obtained in units of time steps, and the timestamps of the contained quadruple data can be averaged to obtain the average span time interval of each time step. For example, assuming that the time step of 0 seconds is the beginning of the time window, at 0 seconds, the microservice system has no call action; while in the time step of 1 second, the quadruple data of 0-1 seconds is obtained. For example, if 0-1 seconds contains quadruple data with timestamps of d seconds and e seconds, the average span time interval can be (d+e) / 2 seconds. For example, d can be 0.2, and e can be 0.4, specifically obtained according to the actual call chain data.

[0064] B4. Construct an initial call chain timing matrix based on the quadruple data, the time step, and the average span time interval; In this embodiment, the initial call chain time series matrix can be a matrix with time steps as rows, all [caller service, called service, call type] as columns, and the average span time interval as matrix elements. All the quadruplet data and the corresponding average span time intervals are concatenated according to the construction method specified above to obtain the initial service indicator time series matrix. For example, in a 1-second time step, if the call chain in the first column appears in the 0-1 second time step, the average span time interval is used as the matrix element of the first row of that column; if the call chain in the first column does not appear in the 0-1 second time step, f is used as the matrix element of the first row of that column, where f can be 0. It is understood that the matrix elements corresponding to call chains appearing in the 0-1 second time step are all the corresponding average span time intervals.

[0065] B5. Normalize the initial call chain timing matrix to obtain the call chain timing matrix.

[0066] In this embodiment, the data in each column of the initial call chain timing matrix is ​​normalized and mapped to the [0,1] interval to obtain the call chain timing matrix.

[0067] S213. The service indicator time series matrix and the call chain time series matrix are fused to obtain the first modal data.

[0068] It is understood that both the service metric time series matrix and the call chain time series matrix are constructed according to the time steps within a time window, so the two matrices have the same number of rows. Therefore, in this embodiment, the service metric time series matrix and the call chain time series matrix can be concatenated and fused according to the number of columns to obtain the first modality data.

[0069] Specifically, such as Figure 6 As shown, preprocessing the historical log data to obtain the log template time series matrix may include the following steps: S221. Based on a preset parsing algorithm, obtain the corresponding variable fields, abnormal risk levels and description fields in the historical log data, convert the variable fields into placeholders, and construct formatted historical log data based on the placeholders, abnormal risk levels and description fields; For example, the log data can be [anomaly risk level, network address, username, description field]. It is understood that historical log data may contain a lot of redundant information. In this embodiment, it is necessary to find information related to anomaly detection as the object of processing; therefore, it is necessary to parse the historical log data based on a preset parsing algorithm. Preferably, the parsing algorithm can be an online log parsing algorithm (Drain).

[0070] For example, network addresses and usernames differ depending on the user performing actions within the microservice system. However, these information is not directly related to anomaly detection and can be discarded. On the other hand, the anomaly risk level and description fields provide a clear assessment of the risk level and describe the actions performed on the log entry. These are crucial information for anomaly detection and should be retained to reduce redundant information and highlight the log's key semantics.

[0071] Therefore, in this embodiment, it is necessary to convert variable fields such as network address and username into placeholders. These placeholders have no specific feature information and are used to preserve the basic structure of historical log data while disrupting it. Thus, the obtained formatted historical log data can be [anomaly risk level, placeholder, placeholder, description field], where the placeholder can be "*" and can be set according to the actual situation.

[0072] For example, a historical log data can be [error,192.168.1.105,user_zhang,attempted an unauthorized access operation], and formatted historical log data can be [error,*,*,attempted an unauthorized access operation].

[0073] S222. Perform vector transformation on the formatted historical log data to obtain a log vector set, wherein the log vector set includes at least one log vector; In this embodiment, for a formatted historical log data, including multiple phrases such as anomaly risk level, placeholders, and description fields, where the description field may also include multiple words, it is necessary to perform space-based word segmentation on the anomaly risk level or description field of these multiple words. For example, the description field "attempted an unauthorized accessoperation" can be segmented into five words: "attempted", "an", "unauthorized", "access", and "operation," depending on the specific historical log data.

[0074] In this embodiment, the segmented words are vectorized to obtain log vectors. All log vectors converted from all words in a formatted historical log data form a log vector set.

[0075] For example, the log vector set It can be represented as: in, This indicates the formatted historical log data. Log vectors This represents the total number of log vectors in the formatted historical log data.

[0076] Preferably, the vector conversion process requires using a pre-defined word vector lexicon to perform the word vector conversion. Words not appearing in the word vector lexicon are marked as unknown words. For these unknown words, corresponding log vectors can be obtained through random initialization or substitution with similar words.

[0077] S223. Input each log vector in the log vector set into a preset word vector model for processing to obtain distributed word vectors; In this embodiment, each log vector in the log vector set needs to be input into a preset word vector model for processing, so that each log vector can be mapped to a g-dimensional distributed word vector, where g can be set according to the actual situation. Preferably, the preset word vector model can be the Word2Vec word vector model.

[0078] For example, the distributed word vector corresponding to each log vector can be represented as follows: .

[0079] S224. Aggregate all the distributed word vectors to obtain the log semantic vector corresponding to the historical log data; In this embodiment, a fixed-dimensional log semantic vector is obtained by using a distributed word vector averaging method.

[0080] For example, the first Log semantic vector corresponding to each historical log data It can be represented as: in, This indicates the total number of historical log data.

[0081] S225. Stack the log semantic vectors corresponding to all the historical log data to obtain a log semantic matrix; Preferably, all the historical log data are stacked row by row in chronological order to form a log semantic matrix.

[0082] For example, the log semantic matrix It can be represented as: Representing log semantic vectors Dimensions.

[0083] S226. Obtain the log occurrence matrix based on the historical log data and the time steps corresponding to the historical log data; Understandably, the historical log data also includes corresponding time steps. In this embodiment, the obtained log semantic matrix does not contain time information, and a log occurrence matrix needs to be constructed to reflect the time information of the log semantic matrix. In this embodiment, a corresponding log occurrence matrix is ​​constructed based on the time step corresponding to each piece of historical log data. If the historical log data does not appear in that time step, it is marked as not appearing; if the historical log data appears in that time step, it is marked as appearing.

[0084] For example, the log occurrence matrix It can be represented as: in, Indicates the number of time steps, when When, it indicates the first The historical log data in the first It appears in a time step, when When, it indicates the first The historical log data in the first It did not appear in the time step.

[0085] S227. Multiply the log semantic matrix and the transpose of the log occurrence matrix to obtain the initial log time sequence matrix; For example, the initial log template time series matrix It can be represented as: in, This represents the transpose of a matrix.

[0086] S228. Normalize the initial log time series matrix to obtain the log time series matrix.

[0087] In this embodiment, the data in each column of the initial log time series matrix is ​​normalized and mapped to the [0,1] interval to obtain the log time series matrix.

[0088] In this embodiment, the log time series matrix integrates log occurrence information from the log occurrence matrix and log semantic information from the log semantic matrix, and is ultimately represented by a time series matrix with a time step length equal to that of the time step.

[0089] S300. Input the first modality data and the log time series matrix into a preset language model for feature extraction to obtain the corresponding first modality feature matrix and second modality feature matrix; In this embodiment, two identical but independent language models are constructed. Preferably, the language model can be a generative pre-trained transformer (GPT-2). The language model can serve as the backbone network for feature extraction, obtain the features corresponding to the first modality data and the log time series matrix, and generate the corresponding first modality feature matrix and second modality feature matrix.

[0090] Specifically, the step of inputting the first modality data and the log time series matrix into a preset language model for feature extraction to obtain the corresponding first modality feature matrix and second modality feature matrix includes: The first modality data is input into the first neural network to obtain the first embedding matrix, and the first embedding matrix is ​​input into the preset first language model to extract features and obtain the first modality feature matrix; The log time series matrix is ​​input into the second neural network to obtain the second embedding matrix, and the second embedding matrix is ​​input into the preset second language model to extract features and obtain the second modality feature matrix; The first neural network and the second neural network each include a linear neural network layer and a ReLU activation function connected in sequence.

[0091] In this embodiment, a first neural network and a second neural network are constructed using a sequentially connected linear neural network and a ReLU activation function. The first neural network can replace the pre-trained embedding layer of the first language model, and the second neural network can replace the pre-trained embedding layer of the second language model.

[0092] Understandably, the first neural network can adjust the dimension of the first modality data to the fixed embedding dimension of the first language model. Similarly, the second neural network can adjust the dimension of the log time series matrix to the fixed embedding dimension of the second language model. Preferably, the fixed embedding dimensions of the first and second language models are the same; for example, the fixed embedding dimension can be 768 dimensions.

[0093] In this embodiment, the first modality data, after passing through the first embedding matrix, is input into the first language model for processing. Preferably, the sixth layer output of the first language model is extracted as the first modality feature matrix. Similarly, the log time series matrix, after passing through the second embedding matrix, is input into the second language model for processing. Preferably, the sixth layer output of the second language model is extracted as the second modality feature matrix. The first and second modality feature matrices have the same matrix size, with rows representing time steps and columns representing the fixed embedding dimensions.

[0094] S400. Input the first modality feature matrix and the second modality feature matrix into a preset classification network model to perform anomaly detection and obtain historical anomaly detection results; In this embodiment, the acquired first modality feature matrix and second modality feature matrix are input into a preset classification network model for anomaly detection. Preferably, the preset classification network model can be a multilayer perceptron classification network.

[0095] Specifically, such as Figure 7 As shown, the step of inputting the first modality feature matrix and the second modality feature matrix into a preset classification network model for anomaly detection to obtain historical anomaly detection results may include the following steps: S410. Concatenate the first modality feature matrix and the second modality feature matrix column by column to obtain a fusion matrix; Preferably, the first modality feature matrix and the second modality feature matrix are concatenated column by column to obtain the fusion matrix.

[0096] S420. Divide the fusion matrix based on a preset ratio to obtain the pre-data and detection data of the fusion matrix; In this embodiment, the fusion matrix is ​​divided into preliminary data and detection data according to a preset ratio. The preliminary data can provide early warning information of anomalies in the microservice system and does not participate in anomaly detection; the detection data is the data foundation for anomaly detection, thereby enabling accurate acquisition of anomalies in the microgrid system while enhancing the capture of early warning information.

[0097] Understandably, the fusion matrix uses time steps as the number of rows, and can acquire preceding data and detection data based on the ratio corresponding to the time steps. The data corresponding to earlier time steps is the preceding data, and the remaining data is the detection data. Preferably, the preset ratio can be set to 1:1. That is, when there are 20 time steps in the time window, the data corresponding to the first 10 time steps in the fusion matrix is ​​the preceding data, and the data corresponding to the last 10 time steps is the detection data. The preset ratio can be set based on actual conditions.

[0098] S430. Input the pre-processed data and the detection data into a preset classification network model for detection to obtain the probability of an anomaly occurring at each time step of the microservice system. In this embodiment, the classification network can be a fully connected layer and an activation function connected sequentially. Based on the preliminary information provided by the preceding data, the detection data is mapped layer by layer through the fully connected layer and the activation function. The last layer uses the Sigmoid activation function to output the probability of anomalies occurring at each time step.

[0099] S440. If the probability of an anomaly occurring at the time step exceeds a preset anomaly threshold, then the microservice system is determined to have an anomaly at the time step. If the probability of an anomaly occurring at the time step does not exceed the preset anomaly threshold, then the microservice system is determined to have no anomaly at the timestamp.

[0100] In this embodiment, the abnormal threshold can be set according to the actual situation.

[0101] S500: Input the first modal feature matrix and the second modal feature matrix into a preset modal reconstruction model to perform modal reconstruction, and obtain the corresponding first modal reconstruction data and second modal reconstruction data; Specifically, the step of inputting the first modal feature matrix and the second modal feature matrix into a preset modal reconstruction model for modal reconstruction to obtain corresponding first modal reconstruction data and second modal reconstruction data includes: The first modal feature matrix is ​​input into a preset first modal reconstruction model for modal reconstruction to obtain first modal reconstruction data, wherein the modal size of the first modal reconstruction data is the same as that of the first modal data. The second modality feature matrix is ​​input into a preset second modality reconstruction model to perform modality reconstruction, thereby obtaining second modality reconstruction data, wherein the modality size of the second modality reconstruction data is the same as that of the log time series data.

[0102] In this embodiment, the first modality feature matrix is ​​flattened into a first one-dimensional vector, and the first one-dimensional vector is input into the first modality reconstruction model for reconstruction. Preferably, the first modality reconstruction model can be a multilayer perceptron reconstruction model, whose output layer uses the Sigmoid activation function to generate a normalized value between 0 and 1, so that the output first modality reconstruction data is normalized data and has the same size dimension as the input first modality data.

[0103] Similarly, the second modality feature matrix is ​​flattened into a second one-dimensional vector, and the second one-dimensional vector is input into the second modality reconstruction model for reconstruction. Preferably, the second modality reconstruction model can be a multilayer perceptron reconstruction model, whose output layer uses the Sigmoid activation function to generate a normalized value between 0 and 1, so that the output second modality reconstruction data is normalized data and has the same size dimension as the input log time series data.

[0104] In this embodiment, the first modality feature matrix and the second modality feature matrix are reconstructed to obtain data with the same size and dimension as the input data. Subsequently, the processed first modality feature matrix and the second modality feature matrix can be compared with the input data separately to ensure that the first modality feature matrix and the second modality feature matrix have a high similarity with the corresponding input data, thereby reducing the overfitting of the model learning.

[0105] S600. Optimize the language model and the classification network based on the historical anomaly detection results, the historical anomaly labels, the first modality data, the log time series matrix, the first modality feature matrix, the second modality feature matrix, and a preset total loss function to obtain an optimized language model and an optimized classification network. In this embodiment, reconstruction training is performed based on the first modality data, the log time-series matrix, the first modality feature matrix, and the second modality feature matrix, with minimizing the reconstruction mean square error as the first optimization objective. Based on the historical anomaly detection results and the historical anomaly labels, the classification network error is calculated using the cross-entropy loss function as the optimization objective, with minimizing the classification network error as the second optimization objective. Furthermore, a total loss function is constructed, prioritizing the first optimization objective in the early stages of training and gradually increasing the weight of the second optimization objective in the later stages, thereby achieving a smooth transition from unsupervised feature learning to supervised classification and detection.

[0106] Preferably, the language model is adjusted using a parameter-efficient fine-tuning method. This method includes freezing the original network parameters except for the position embedding and layer normalization parameters, allowing gradient updates only for the position embedding and layer normalization parameters, and inserting them as additional modules into the Q-projection matrix and K-projection matrix of the attention mechanism in the language model, ultimately resulting in an optimized language model.

[0107] Specifically, the total loss function is constructed based on the first mode reconstruction loss function, the second mode reconstruction loss function, and the cross-entropy classification loss value; The total loss function for: Wherein, the first mode reconstruction loss function for: The second mode reconstruction loss function for: This represents the first modality data. This represents the reconstructed data of the first mode. Represents the log time series matrix. This represents the reconstructed data from the second mode; This represents the cross-entropy classification loss value, which is calculated by performing cross-entropy loss on the historical anomaly detection results and the historical anomaly labels. This indicates the number of training rounds.

[0108] S700: The real-time service indicator data, real-time log data, and real-time call chain data obtained from the real-time collection of the microservice system are preprocessed and input into the optimized language model and the optimized classification network for anomaly detection to obtain real-time anomaly detection results.

[0109] In this embodiment, the real-time service indicator data, real-time log data, and real-time call chain data collected in real time are preprocessed. The preprocessing steps are the same as those for historical data preprocessing, thereby obtaining real-time first modality data and real-time log time-series data. The real-time first modality data and real-time log time-series data are input into the optimized language model and the optimized classification network for anomaly detection, thereby obtaining real-time anomaly detection results and improving the accuracy of anomaly detection in the microservice system.

[0110] For example, this method can achieve the corresponding technical effect in practical applications. Specifically, two open-source datasets are used: the MSDS dataset (Nedelkoski, S.; Bogatinovski, J.; Mandapati, AK; Becker, S.; Cardoso, J.; and Kao, O. 2020. Multi-source distributed systemdata for ai-powered analytics. In Service-Oriented and Cloud Computing: 8thIFIP WG 2.14 European Conference, ESOCC 2020) and the Eadro dataset (Lee C, Yang T, Chen Z, et al. Eadro: An end-to-end troubleshooting framework for microservices on multi-source data[C] / / 2023 IEEE / ACM 45th International Conference on Software Engineering (ICSE).). The MSDS dataset was collected from an OpenStack-based open-source cloud platform. It includes seven service metrics collected from five physical service nodes, 130,326 log entries, and 1,514,358 call chains. The anomaly rate of the dataset after partitioning into sliding windows is 4.70%. The Eadro dataset was collected from the open-source microservice benchmark SocialNetwork. It includes seven metrics collected from twelve service nodes, 94,319 log entries, and 1,438,191 call chains. The anomaly rate of the dataset after partitioning into sliding windows is 4.37%. Both datasets are divided into training, testing, and validation sets in a 6:1:3 ratio.

[0111] Figure 8 This chart compares the performance of our proposed method on the Eadro dataset with two existing open-source microservice anomaly detection methods, namely, those based on the Eadro algorithm and those based on the MSTGAD algorithm. Precision, recall, and F1 score were selected as evaluation metrics. Figure 8As shown, this application can effectively detect faults in microservice systems with low false positive and false negative rates. Furthermore, experiments were conducted on a Linux host equipped with an NVIDIA GeForce RTX 4090D GPU, an AMD EPYC 9754128-core CPU processor, and 60GB of memory. The time taken for this application to achieve 90% and 95% F1 scores on the validation set was 93s and 112s, respectively, verifying its suitability for scenarios where microservice systems frequently update and require retraining. In addition, the average data preprocessing time and average model inference time of this application were statistically analyzed on the same host. For a time window of 20s, this application requires only 11ms of preprocessing time and 13ms of model inference time, verifying its adaptability for real-time anomaly detection.

[0112] like Figure 9 As shown in the illustration, this application also provides a microservice anomaly detection device. Optionally, the device includes: The module comprises: data acquisition module 811, preprocessing module 812, feature extraction module 813, historical prediction module 814, reconstruction module 815, optimization module 816, and anomaly detection module 817, among which: Data acquisition module 811 is used to acquire historical data of the microservice system, including historical service indicator data, historical log data, historical call chain data, and historical anomaly tags corresponding to the historical data; In this embodiment, the data acquisition 811 can be used to perform... Figure 2 For a detailed description of step S100, which is shown below, please refer to the description of step S100.

[0113] The data acquisition 811 is also used to preset a time window, wherein the time window includes several time steps with equal time intervals; acquire historical service indicator data, historical log data and historical call chain data of the time step corresponding to the historical data, and acquire historical anomaly tags corresponding to the time step in the historical data.

[0114] Preprocessing module 812 is used to preprocess the historical service indicator data and the historical call chain data to obtain first modal data, and to preprocess the historical log data to obtain a log time series matrix; In this embodiment, the preprocessing module 812 can be used to perform... Figure 2 For a detailed description of the preprocessing module 812, please refer to the description of step S200 shown.

[0115] The preprocessing module is further configured to preprocess the historical service indicator data to obtain a service indicator time series matrix; preprocess the historical call chain data to obtain a call chain time series matrix; and fuse the service indicator time series matrix and the call chain time series matrix to obtain first modal data.

[0116] In this embodiment, the preprocessing module 812 can be used to perform... Figure 3 For a detailed description of the preprocessing module 812, see steps S210-S230 shown.

[0117] The preprocessing module is further configured to obtain corresponding triplet data based on the historical service indicator data, wherein the triplet data includes service type, service indicator and time step; arrange all the triplet data in order according to the corresponding time step; construct an initial service indicator time series matrix based on the triplet data in the completed order arrangement; and normalize the initial service indicator time series matrix to obtain the service indicator time series matrix. The preprocessing module is further configured to obtain corresponding quadruple data based on the historical call chain data, wherein the quadruple data includes the caller service type, the callee service type, the call type, and a timestamp; arrange all the quadruple data in order according to their corresponding timestamps; calculate the average span time interval in units of the time step based on the timestamps corresponding to the quadruple data; construct an initial call chain timing matrix based on the quadruple data, the time step, and the average span time interval; and normalize the initial call chain timing matrix to obtain the call chain timing matrix.

[0118] In this embodiment, the preprocessing module 812 can be specifically used to perform... Figure 4 Steps A1-A4 shown Figure 5 For a detailed description of the preprocessing module 812, see steps B1-B5.

[0119] The preprocessing module is further configured to: obtain the corresponding variable fields, anomaly risk levels, and description fields in the historical log data based on a preset parsing algorithm; convert the variable fields into placeholders; construct formatted historical log data based on the placeholders, anomaly risk levels, and description fields; perform vector transformation on the formatted historical log data to obtain a log vector set, which includes at least one log vector; input each log vector in the log vector set into a preset word vector model for processing to obtain distributed word vectors; aggregate all the distributed word vectors to obtain the log semantic vectors corresponding to the historical log data; stack all the log semantic vectors corresponding to the historical log data to obtain a log semantic matrix; obtain a log occurrence matrix based on the historical log data and the time steps corresponding to the historical log data; multiply the log semantic matrix and the transpose of the log occurrence matrix to obtain an initial log time series matrix; and normalize the initial log time series matrix to obtain the final log time series matrix.

[0120] In this embodiment, the preprocessing module 812 can be specifically used to perform... Figure 6 For a detailed description of the preprocessing module 812, see steps S221-S228 shown.

[0121] The feature extraction module 813 is used to input the first modality data and the log template time series matrix into a preset language model for feature extraction, so as to obtain the corresponding first modality feature matrix and second modality feature matrix; In this embodiment, the feature extraction module 813 can be used to perform... Figure 2 For a detailed description of the feature extraction module 813, please refer to the description of step S300 shown.

[0122] The feature extraction module 813 is specifically used to input the first modality data into a first neural network to obtain a first embedding matrix, and input the first embedding matrix into a preset first language model to extract features to obtain a first modality feature matrix; input the log time series matrix into a second neural network to obtain a second embedding matrix, and input the second embedding matrix into a preset second language model to extract features to obtain a second modality feature matrix; wherein the first neural network and the second neural network respectively include a linear neural network layer and a ReLU activation function connected in sequence.

[0123] The historical prediction module 814 is used to input the first modality feature matrix and the second modality feature matrix into a preset classification network model for anomaly detection and to obtain historical anomaly detection results. In this embodiment, the historical prediction module 814 can be used to perform... Figure 2 For a detailed description of the historical prediction module 814, please refer to the description of step S400 shown.

[0124] The historical prediction module 814 is further configured to concatenate the first modality feature matrix and the second modality feature matrix column by column to obtain a fusion matrix; divide the fusion matrix based on a preset ratio to obtain the pre-data and detection data of the fusion matrix; input the pre-data and the detection data into a preset classification network model for detection to obtain the probability of an anomaly occurring at each time step of the microservice system; if the probability of an anomaly occurring at the time step exceeds a preset anomaly threshold, it is determined that the microservice system has an anomaly at the time step; if the probability of an anomaly occurring at the time step does not exceed the preset anomaly threshold, it is determined that the microservice system has no anomaly at the timestamp.

[0125] In this embodiment, the historical prediction module 814 can be specifically used to perform... Figure 7 For a detailed description of the historical prediction module 814, see steps S410-S440 shown.

[0126] The reconstruction module 815 is used to input the first modal feature matrix and the second modal feature matrix into a preset modal reconstruction model to perform modal reconstruction, thereby obtaining corresponding first modal reconstruction data and second modal reconstruction data. In this embodiment, the reconstruction module 815 can be used to perform... Figure 2 For a detailed description of the reconstruction module 815, please refer to the description of step S500 shown.

[0127] The reconstruction module is further configured to input the first modal feature matrix into a preset first modal reconstruction model for modal reconstruction to obtain first modal reconstruction data, wherein the first modal reconstruction data has the same modal size as the first modal data; and to input the second modal feature matrix into a preset second modal reconstruction model for modal reconstruction to obtain second modal reconstruction data, wherein the second modal reconstruction data has the same modal size as the second modal data.

[0128] The optimization module 816 is used to optimize the language model and the classification network based on the historical anomaly detection results, the historical anomaly labels, the first modality data, the log time series matrix, the first modality feature matrix, the second modality feature matrix, and a preset total loss function, so as to obtain an optimized language model and an optimized classification network. In this embodiment, the optimization module 816 can be used to perform... Figure 2 For a detailed description of the optimization module 816, please refer to the description of step S600 shown.

[0129] The anomaly detection module 817 is used to input the preprocessed real-time service indicator data, real-time log data and real-time call chain data obtained from the real-time collection of the microservice system into the optimized language model and the optimized classification network for anomaly detection, and obtain real-time anomaly detection results.

[0130] In this embodiment, the anomaly detection module 817 can be used to perform... Figure 2 For a detailed description of the anomaly detection module 817, please refer to the description of step S700 shown.

[0131] This application also provides an electronic device, the structure of which is as follows: Figure 10 As shown, the electronic device includes a memory 911, a processor 912, a communication module 913, and an input / output interface 914, etc. Optionally, the memory 911, the processor 912, the communication module 913, and the input / output interface 914 can be connected and communicate with each other through a bus 915.

[0132] The memory 911 is used to store one or more computer programs and to transfer the code of the computer programs to the processor 912; when the one or more computer programs are executed by the processor 912, a microservice anomaly detection method in this application embodiment is implemented.

[0133] Optionally, the electronic device can be connected to a network via the communication module 913 to communicate with other devices, such as terminals or servers, to achieve data interaction. The electronic device can be various forms of digital computers, exemplarily such as desktop computers, servers, workbenches, mainframes, or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily such as smartphones, tablets, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0134] Optionally, the electronic device can connect to required input / output devices, such as a keyboard or display device, via the input / output interface 914. The electronic device itself may have a display device, and other display devices can also be connected externally via the input / output interface 914. Optionally, a storage device, such as a hard disk, can also be connected via the input / output interface 914 to store data from the electronic device, read data from the storage device, or store data from the storage device in the memory 911. It is understood that the input / output interface 914 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 914 can be a component of the electronic device or an external device connected to the electronic device when needed.

[0135] Optionally, the memory 911 may be a volatile memory and / or a non-volatile memory. The volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0136] Optionally, the computer program stored in the memory 911 can be divided into one or more modules, which are stored in the memory 911 and executed by the processor 912 to perform the method provided in this embodiment. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0137] Optionally, the processor 912 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 912 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and can also be any suitable controller, microcontroller, processor, etc. The processor 912 executes the various methods and processes of this embodiment, exemplified by a microservice anomaly detection method according to an embodiment of this application.

[0138] Optionally, the bus 915 may include a path for transmitting information. Depending on its function, the bus 615 may be divided into an address bus, a data bus, a control bus, etc.

[0139] In an optional implementation, this application embodiment also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods described in the above method embodiments. Part or all of the computer program can be loaded and / or installed on the memory 911 of an electronic device. When the computer program is executed by the processor 912, one or more steps of a microservice anomaly detection method according to this application embodiment can be performed.

[0140] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0141] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A microservice anomaly detection method, characterized in that, The method includes: Obtain historical data from the microservice system, including historical service metric data, historical log data, historical call chain data, and historical anomaly tags corresponding to the historical data; The historical service indicator data and the historical call chain data are preprocessed to obtain the first modality data, and the historical log data are preprocessed to obtain the log time series matrix; The first modality data and the log time series matrix are respectively input into a preset language model for feature extraction to obtain the corresponding first modality feature matrix and second modality feature matrix; The first modality feature matrix and the second modality feature matrix are input into a preset classification network model for anomaly detection to obtain historical anomaly detection results; The first modal feature matrix and the second modal feature matrix are respectively input into a preset modal reconstruction model for modal reconstruction to obtain the corresponding first modal reconstruction data and second modal reconstruction data; The language model and the classification network are optimized based on the historical anomaly detection results, the historical anomaly labels, the first modality data, the log time series matrix, the first modality feature matrix, the second modality feature matrix, and the preset total loss function to obtain an optimized language model and an optimized classification network. The real-time service metric data, real-time log data, and real-time call chain data obtained from the microservice system are preprocessed and then input into the optimized language model and the optimized classification network for anomaly detection to obtain real-time anomaly detection results.

2. The method according to claim 1, characterized in that, The acquisition of historical data from the microservice system includes: A preset time window, wherein the time window includes several time steps of equal time intervals; Obtain historical service indicator data, historical log data, and historical call chain data of the time window corresponding to the time step in the historical data, and obtain historical anomaly tags corresponding to the time step in the historical data.

3. The method according to claim 2, characterized in that, The preprocessing of the historical service indicator data and the historical call chain data to obtain the first modal data includes: The historical service indicator data is preprocessed to obtain the service indicator time series matrix; The historical call chain data is preprocessed to obtain the call chain timing matrix; The service metric time series matrix and the call chain time series matrix are fused to obtain the first modality data.

4. The method according to claim 3, characterized in that, The historical service indicator data is preprocessed to obtain a service indicator time series matrix, including: Obtain the corresponding triplet data based on the historical service indicator data. The triplet data includes service type, service indicator, and time step. Arrange all the triplet data in order according to the corresponding time step. Construct an initial service indicator time series matrix based on the ordered triplet data. Normalize the initial service indicator time series matrix to obtain the service indicator time series matrix. And / or, preprocessing the historical call chain data to obtain a call chain time sequence matrix, including: Obtain corresponding quadruple data based on the historical call chain data. The quadruple data includes the caller service type, the callee service type, the call type, and a timestamp. Arrange all the quadruple data in order according to their corresponding timestamps. Calculate the average span time interval in units of the time steps based on the timestamps corresponding to the quadruple data. Construct an initial call chain timing matrix based on the quadruple data, the time steps, and the average span time interval. Normalize the initial call chain timing matrix to obtain the final call chain timing matrix.

5. The method according to claim 2, characterized in that, The process of preprocessing the historical log data to obtain the log time-series matrix includes: Based on a preset parsing algorithm, the corresponding variable fields, anomaly risk levels, and description fields in the historical log data are obtained. The variable fields are converted into placeholders, and formatted historical log data is constructed based on the placeholders, anomaly risk levels, and description fields. The formatted historical log data is vectorized to obtain a log vector set, wherein the log vector set includes at least one log vector; Each log vector in the log vector set is input into a preset word vector model for processing to obtain distributed word vectors; Aggregate all the distributed word vectors to obtain the log semantic vector corresponding to the historical log data; The log semantic vectors corresponding to all the historical log data are stacked to obtain the log semantic matrix; Obtain the log occurrence matrix based on the historical log data and the time steps corresponding to the historical log data; The initial log time series matrix is ​​obtained by multiplying the transpose of the log semantic matrix and the log occurrence matrix; The initial log time series matrix is ​​normalized to obtain the log time series matrix.

6. The method according to claim 1, characterized in that, The step of inputting the first modality data and the log time series matrix into a preset language model for feature extraction to obtain the corresponding first modality feature matrix and second modality feature matrix includes: The first modality data is input into the first neural network to obtain the first embedding matrix, and the first embedding matrix is ​​input into the preset first language model to extract features and obtain the first modality feature matrix; The log time series matrix is ​​input into the second neural network to obtain the second embedding matrix, and the second embedding matrix is ​​input into the preset second language model to extract features and obtain the second modality feature matrix; The first neural network and the second neural network each include a linear neural network layer and a ReLU activation function connected in sequence.

7. The method according to claim 1, characterized in that, The step of inputting the first modal feature matrix and the second modal feature matrix into a preset modal reconstruction model for modal reconstruction to obtain corresponding first modal reconstruction data and second modal reconstruction data includes: The first modal feature matrix is ​​input into a preset first modal reconstruction model for modal reconstruction to obtain first modal reconstruction data, wherein the modal size of the first modal reconstruction data is the same as that of the first modal data. The second modal feature matrix is ​​input into a preset second modal reconstruction model to perform modal reconstruction, thereby obtaining second modal reconstruction data, wherein the modal size of the second modal reconstruction data is the same as that of the second modal data.

8. The method according to claim 1, characterized in that, The step of inputting the first modality feature matrix and the second modality feature matrix into a preset classification network model for anomaly detection to obtain historical anomaly detection results includes: The first modality feature matrix and the second modality feature matrix are concatenated column-wise to obtain the fusion matrix; The fusion matrix is ​​divided based on a preset ratio to obtain the pre-data and detection data of the fusion matrix. The pre-processed data and the detection data are input into a preset classification network model for detection to obtain the probability of an anomaly occurring at each time step of the microservice system. If the probability of an anomaly occurring at the time step exceeds a preset anomaly threshold, the microservice system is determined to have an anomaly at that time step. If the probability of an anomaly occurring at the time step does not exceed the preset anomaly threshold, the microservice system is determined to have no anomaly at that timestamp.

9. The method according to claim 7, characterized in that, The total loss function is constructed based on the first modality reconstruction loss function, the second modality reconstruction loss function, and the cross-entropy classification loss value; The total loss function for: Wherein, the first mode reconstruction loss function for: The second mode reconstruction loss function for: This represents the first modality data. This represents the reconstructed data of the first mode. Represents the log time series matrix. This represents the reconstructed data from the second mode; This represents the cross-entropy classification loss value, which is calculated by performing cross-entropy loss on the historical anomaly detection results and the historical anomaly labels. This indicates the number of training rounds.

10. A microservice anomaly detection device, characterized in that, The device includes: The data acquisition module is used to acquire historical data of the microservice system. The historical data includes historical service indicator data, historical log data, historical call chain data, and historical anomaly tags corresponding to the historical data. The preprocessing module is used to preprocess the historical service indicator data and the historical call chain data to obtain the first modality data, and to preprocess the historical log data to obtain the log time series matrix. The feature extraction module is used to input the first modality data and the log time series matrix into a preset language model for feature extraction, so as to obtain the corresponding first modality feature matrix and second modality feature matrix; The historical prediction module is used to input the first modality feature matrix and the second modality feature matrix into a preset classification network model for anomaly detection and to obtain historical anomaly detection results. The reconstruction module is used to input the first modal feature matrix and the second modal feature matrix into a preset modal reconstruction model to perform modal reconstruction, thereby obtaining the corresponding first modal reconstruction data and second modal reconstruction data. The optimization module is used to optimize the language model and the classification network based on the historical anomaly detection results, the historical anomaly labels, the first modality data, the log time series matrix, the first modality feature matrix, the second modality feature matrix, and a preset total loss function, so as to obtain an optimized language model and an optimized classification network. The anomaly detection module is used to preprocess real-time service indicator data, real-time log data, and real-time call chain data obtained from real-time collection of the microservice system, and then input them into the optimized language model and the optimized classification network for anomaly detection to obtain real-time anomaly detection results.

Images