Middleware anomaly detection method and device based on auto-encoder and data enhancement

Abstract

The invention provides a middleware anomaly detection method and device based on an auto-encoder and data enhancement, and relates to the technical field of data processing.The method comprises the steps that log data of middleware and related components are collected, and data cleaning is conducted on the log data according to characteristics of middleware logs; performing feature extraction operation on the cleaned log data to obtain feature data; determining an abnormal log in the log data according to the feature data, and expanding the abnormal log; and performing anomaly detection on the log data by using a kernel matrix-based clustering optimization method. The de-noising auto-encoder is used for feature extraction and selection, the problem that information data extracted from original log data is low in quality or contains noise is solved, and the accuracy of middleware anomaly detection is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a middleware anomaly detection method and device based on autoencoder and data enhancement. Background Art

[0002] Middleware is a type of software used to connect different applications, systems, or data to provide seamless communication and data management. It is located between the client and the server and plays multiple roles such as data transmission, message service, authentication, encryption, etc.

[0003] Middleware anomalies may be caused by a variety of reasons, including configuration errors, resource competition, hardware failures, etc. The accuracy of middleware anomaly detection in related technologies is low, and it is difficult to detect middleware anomalies. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first objective of this application is to propose a middleware anomaly detection method based on autoencoder and data enhancement.

[0006] The second object of the present application is to provide a device.

[0007] The third objective of the present application is to provide an electronic device.

[0008] A fourth objective of the present application is to provide a computer-readable storage medium.

[0009] A fifth object of the present application is to provide a computer program product.

[0010] To achieve the above objectives, the first embodiment of the present application proposes a middleware anomaly detection method based on autoencoder and data enhancement, including:

[0011] Collect log data of middleware and related components, and clean the log data according to the characteristics of the middleware logs;

[0012] Performing a feature extraction operation on the cleaned log data to obtain feature data;

[0013] Determine an abnormal log in the log data according to the characteristic data and expand the abnormal log;

[0014] Anomaly detection is performed on the log data using a kernel matrix-based clustering optimization method.

[0015] Optionally, collecting log data of the middleware and related components includes:

[0016] Determine the target middleware type and target related components, wherein the target related components include: database system, operating system, load balancer;

[0017] Configuring so that the target middleware type and the target-related component output the log data;

[0018] A data collection script is deployed to automatically collect the log data and temporarily store it in the file system.

[0019] Optionally, the log data corresponding to the middleware includes at least one of the following: request log, error log, transaction log, system event log, performance log;

[0020] The log data corresponding to the database system includes at least one of the following: SQL query log, error log, transaction log, performance index;

[0021] The log data corresponding to the operating system and the load balancer includes at least one of the following: a system event log, an error log, a resource usage log, and a load balancer log.

[0022] Optionally, the performing data cleaning on the log data according to the characteristics of the middleware log includes:

[0023] Parsing the log data according to the log format, extracting key information from each line or each log entry, and converting the information into a structured data format;

[0024] Filter irrelevant information and exclude entries in the log data that are irrelevant to the normal operation or performance of the system;

[0025] The data type and format of the log data after filtering are unified.

[0026] Optionally, extracting key information from each line or each log entry and converting it into a structured data format includes:

[0027] Define a multi-line string variable to store log entries of the log data, each log entry contains a timestamp, log level and descriptive message;

[0028] Define and compile regular expressions for log matching patterns, which are used to identify and extract timestamps, log levels, and message content in log entries;

[0029] Initialize a structured log list, where the structured log list is used to store log entries that are parsed and converted into a dictionary format;

[0030] Traverse each line in the log data to obtain a list containing all log lines;

[0031] Use the regular expression of the log matching pattern to match each of the log lines, and return a matching object for the log lines that match the regular expression;

[0032] Creating a log entry dictionary, and storing the timestamp, log level, and message content extracted from the matching object in the log entry dictionary;

[0033] The log entry dictionary is added to the list, and the list is converted to a target format to obtain structured log data.

[0034] Optionally, the operation of performing feature extraction on the cleaned log data to obtain feature data includes:

[0035] Determine the feature set, including four categories: performance features, request processing features, error and transaction features, and system event features;

[0036] Define a unified data model that includes all possible fields, and specify standardized names and data types for each field;

[0037] Reading the structured log data, and identifying and extracting relevant feature data from the structured log data according to the feature set;

[0038] Data preprocessing is performed for numerical and non-numerical feature data respectively;

[0039] The preprocessed feature data are associated to generate a new data set.

[0040] Optionally, the method further includes:

[0041] Using different methods for each type of feature data to generate noise and adding it to the feature data to generate a training data set;

[0042] A denoising autoencoder model is constructed, and the denoising autoencoder model is trained according to the training data set.

[0043] Optionally, the data model includes:

[0044] Basic information, including: a unique identifier corresponding to a log entry or a performance indicator entry; a component type of the data source; and a specific component name;

[0045] Log-specific information, including: log level, detailed message content of the log, and error log.

[0046] Performance indicator information, the performance indicator information includes: the name of the performance indicator, the specific value of the performance indicator;

[0047] Additional information, which is used to store other related data or additional information that is useful in a specific context.

[0048] Optionally, the adopting different methods for each type of feature data to generate noise and adding the noise to the feature data to generate a training data set includes:

[0049] Adding Gaussian noise to the feature data of the numerical type to simulate noise;

[0050] The bits in the one-hot encoding vector are randomly flipped with a preset probability to add noise to the feature data of the non-numeric class.

[0051] Optionally, determining an abnormal log in the log data according to the characteristic data and expanding the abnormal log includes:

[0052] Determine the category distribution of the log data according to the denoising autoencoder model, and determine that a minority of the categories are abnormal logs;

[0053] Oversampling the abnormal log to generate an expanded sample, and forming an expanded data set with the expanded samples;

[0054] The expanded data set is merged with the original log data.

[0055] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a middleware anomaly detection device based on autoencoder and data enhancement, including:

[0056] A data cleaning module, used to collect log data of the middleware and related components, and perform data cleaning on the log data according to the characteristics of the middleware log;

[0057] A feature extraction module, used for performing a feature extraction operation on the cleaned log data to obtain feature data;

[0058] A data expansion module, used for determining an abnormal log in the log data according to the characteristic data and expanding the abnormal log;

[0059] The anomaly monitoring module is used to perform anomaly detection on the log data using a clustering optimization method based on a kernel matrix.

[0060] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0061] The memory stores computer-executable instructions;

[0062] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0063] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0064] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, which implements any method in the first aspect when executed by a processor.

[0065] The middleware anomaly detection method, device, electronic device and storage medium based on autoencoder and data enhancement provided by the present application use denoising autoencoder for feature extraction and selection to solve the problem of low quality or noise of information data extracted from original log data, thereby improving the accuracy of middleware anomaly detection. The data quality is improved by advanced data preprocessing and feature engineering technology (denoising autoencoder), combined with the full solution of abnormal logs to solve the problem of data category imbalance, and then the optimized K-Means clustering method is used for efficient and accurate anomaly detection. This scheme takes into account the complexity of middleware log data, and maps data to high-dimensional space through kernel technology to improve clustering quality.

[0066] 1. The application of denoising autoencoder (DAE) in the feature extraction stage can extract more refined and representative features from the original log data. The denoising process enables the model to learn how to express data in a noisy environment, enhancing the model's ability to identify abnormal data, especially when processing log data containing a lot of noise and irrelevant information.

[0067] 2. By simulating the noise addition process of data loss or deviation, the model trained by the denoising autoencoder can better adapt to the data quality problems that may be encountered in actual situations, and improve the robustness and generalization ability of the model in practical applications.

[0068] 3. For middleware log data, abnormal events are often minority classes. Using the SMOTE algorithm for oversampling can effectively solve the problem of data imbalance. By generating synthetic minority class samples, the model's ability to identify uncommon but critical abnormal events can be enhanced.

[0069] 4. The data augmentation process not only balances the dataset, but also improves the generalization ability of the model by increasing sample diversity.

[0070] 5. The K-Means optimization method based on the kernel matrix enables the K-Means clustering algorithm to effectively handle complex, nonlinearly separable data distributions, especially for high-dimensional and extremely complex data sets such as middleware log data.

[0071] 6. Using the kernel matrix optimized K-Means clustering method, the real clustering structure in the data can be more accurately identified. Compared with the traditional K-Means algorithm, more compact and separated clustering results can be obtained when processing data sets with complex intrinsic structures.

[0072] 7. This proposal builds a comprehensive data processing and feature engineering process by combining data cleaning, feature extraction, data enhancement, and advanced clustering techniques. This not only improves data quality, but also ensures that the features extracted from the data can effectively support subsequent anomaly detection tasks.

[0073] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0075] Figure 1 A flowchart of a middleware anomaly detection method based on autoencoder and data enhancement provided in an embodiment of the present application;

[0076] Figure 2 A bar chart corresponding to a data set;

[0077] Figure 3 A schematic diagram of the structure of a middleware anomaly detection device based on autoencoder and data enhancement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0079] Middleware is a type of software used to connect different applications, systems, or data to provide seamless communication and data management. It is located between the client and the server and plays multiple roles such as data transmission, message service, authentication, encryption, etc.

[0080] The key reason for the importance of middleware is that it acts as a bridge between different systems and applications, ensuring smooth information flow and efficient data processing, which is essential for building scalable, highly available and secure IT systems.

[0081] Middleware exceptions may be caused by a variety of reasons, including configuration errors, resource competition, hardware failures, etc. Common types of middleware exceptions and their impacts include: performance bottlenecks, service interruptions, and data loss or inconsistency.

[0082] The monitoring and management of middleware is an important part of maintaining system stability and high performance. Through real-time monitoring of the running status of middleware, performance bottlenecks, resource leakage, abnormal behavior and other problems can be discovered and solved in a timely manner. Effective middleware management includes log analysis, performance optimization, security policy implementation and other aspects, aiming to improve the system's response speed, ensure data security, and ensure the continuous availability of services.

[0083] The relevant technical solution adopts threshold-based monitoring, based on predefined indicator thresholds (such as CPU usage, memory consumption, response time, etc.). Once the monitored indicators exceed these thresholds, the system will issue an alarm.

[0084] Identify potential anomalies and errors by analyzing the log files generated by the middleware.

[0085] Continuously track the performance indicators of the middleware and its related components to facilitate real-time understanding of system status and performance. This includes but is not limited to resource usage (such as CPU, memory usage), service response time, and throughput.

[0086] However, they rely on simple log analysis and threshold monitoring, which limits their ability to handle complex, high-dimensional and noisy data. This leads to insufficient data preprocessing and the inability to fully utilize key information in the logs of middleware and related IT components.

[0087] Middleware abnormal events are usually minority classes in log data. Existing methods are not effective in dealing with this class imbalance problem, resulting in the model being unable to effectively learn the characteristics of abnormal events, thus affecting detection accuracy.

[0088] Traditional anomaly detection methods, such as threshold-based monitoring and simple log analysis, face challenges in accuracy and efficiency, especially when dealing with large-scale, high-dimensional data, where they often have difficulty identifying anomalies quickly and accurately.

[0089] It is not very effective in processing high-dimensional log data generated by middleware and related IT components, and often cannot effectively handle the complexity and inherent relationships of data dimensions.

[0090] To address this problem, the present invention provides a middleware anomaly detection method based on autoencoder and data enhancement. Figure 1 The following is a flow chart of a middleware anomaly detection method based on autoencoder and data enhancement provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0091] Step 101, collecting log data of the middleware and related components, and performing data cleaning on the log data according to the characteristics of the middleware log;

[0092] In this embodiment, log data of the middleware, as well as logs and performance indicators of other IT components such as related databases and operating systems are collected.

[0093] Step 102, performing a feature extraction operation on the cleaned log data to obtain feature data;

[0094] In this embodiment, data cleaning is performed based on the characteristics of the middleware log, the log format is parsed, irrelevant information is filtered, and missing values ​​are processed. A denoising autoencoder (DAE) is used to extract key features from the cleaned log.

[0095] Step 103, determining an abnormal log in the log data according to the characteristic data and expanding the abnormal log;

[0096] In this embodiment, the SMOTE algorithm is applied to oversample the abnormal logs (minority class) of the middleware, especially for uncommon but important abnormal events, to solve the problem of data category imbalance.

[0097] Step 104: Perform anomaly detection on the log data using a kernel matrix-based clustering optimization method.

[0098] In this embodiment, a K-Means optimization method based on a kernel matrix is ​​used to perform anomaly detection on the data set after preprocessing, feature engineering and data enhancement in the above steps.

[0099] Optionally, the step 101 collects log data of the middleware and related components, including:

[0100] Determine the target middleware type and target related components, wherein the target related components include: database system, operating system, load balancer;

[0101] Configuring so that the target middleware type and the target-related component output the log data;

[0102] A data collection script is deployed to automatically collect the log data and temporarily store it in the file system.

[0103] Optionally, the log data corresponding to the middleware includes at least one of the following: request log, error log, transaction log, system event log, performance log;

[0104] The log data corresponding to the database system includes at least one of the following: SQL query log, error log, transaction log, performance index;

[0105] The log data corresponding to the operating system and the load balancer includes at least one of the following: a system event log, an error log, a resource usage log, and a load balancer log.

[0106] In this embodiment, for the middleware, the collected log data includes:

[0107] Request log: records all request details entering the middleware, including timestamp, request type, request parameters, response time, and status code. Error log: records error events in detail, such as exception stack, error code, error message, etc. Transaction log: for transactional middleware, records the time when the transaction starts, executes, and ends, as well as the transaction status (success, failure). System event log: records middleware system-level events, such as start, stop, and configuration changes. Performance log: records middleware performance indicators, such as throughput, latency, and resource utilization (CPU, memory).

[0108] For database systems, the collected log data includes: SQL query log: records the executed SQL statements and their execution time and execution result information. Error log: database operation errors, connection failures and other error information. Transaction log: database transaction start and end time and transaction status. Performance indicators: database response time, query processing time, number of connections, cache hit rate.

[0109] For operating systems and load balancing devices, the collected log data includes: System event log: records operating system level events such as logins, permission changes, and system calls. Error log: system errors, hardware failures, and driver error information. Resource usage log: CPU usage, memory usage, disk I / O, and network I / O indicators. Load balancer log: request allocation, session duration, and status code.

[0110] Optionally, step 101 performs data cleaning on the log data according to the characteristics of the middleware log, including:

[0111] Parsing the log data according to the log format, extracting key information from each line or each log entry, and converting the information into a structured data format;

[0112] Filter irrelevant information and exclude entries in the log data that are irrelevant to the normal operation or performance of the system;

[0113] The data type and format of the log data after filtering are unified.

[0114] In this embodiment, data cleaning is performed based on the characteristics of the middleware log, the log format is parsed, irrelevant information is filtered, and missing values ​​are processed. Then, irrelevant information is filtered, and entries in the log that are irrelevant to the normal operation or performance of the system are excluded by writing scripts, such as software version information, configuration loading messages, etc.

[0115] Standardize data types and formats to ensure that the data type and format of each field are unified. Use ISO 8601 (YYYY-MM-DDTHH:MM:SSZ) as the target format for all timestamp data. Convert log levels to uppercase or lowercase to ensure consistent representation of log levels. Error codes: Ensure that error codes follow a unified naming convention. If multiple systems generate similar but different format error codes, unify the error code representation through a mapping mechanism. Performance indicator values: Determine a unified numeric format and use formatted strings or decimal modules to achieve precision control. Additional information is uniformly stored in text form.

[0116] The steps to handle missing values ​​are:

[0117] 1) Identify missing values: Use built-in functions provided by programming libraries for data analysis and machine learning (such as Python’s Pandas) to detect missing values, for example: Pandas’ isnull() or isna() functions.

[0118] 2) Handling missing values: Select the corresponding processing method based on the characteristics of the data.

[0119] a) If there are relatively few records with missing values ​​and removing them will not cause serious skew in the data, then delete these records. This can be done using the dropna() function of Pandas. If there are a lot of missing values ​​in a field, then delete the entire field.

[0120] b) If the missing value represents “not present” or “not applicable”, the constant “0” is used to fill the missing value, otherwise “1” is used to fill it.

[0121] Optionally, extracting key information from each line or each log entry and converting it into a structured data format includes:

[0122] Define a multi-line string variable to store log entries of the log data, each log entry contains a timestamp, log level and descriptive message;

[0123] Define and compile regular expressions for log matching patterns, which are used to identify and extract timestamps, log levels, and message content in log entries;

[0124] Initialize a structured log list, where the structured log list is used to store log entries that are parsed and converted into a dictionary format;

[0125] Traverse each line in the log data to obtain a list containing all log lines;

[0126] Use the regular expression of the log matching pattern to match each of the log lines, and return a matching object for the log lines that match the regular expression;

[0127] Creating a log entry dictionary, and storing the timestamp, log level, and message content extracted from the matching object in the log entry dictionary;

[0128] The log entry dictionary is added to the list, and the list is converted to a target format to obtain structured log data.

[0129] In this embodiment, the target format is JSON format, and the log format is parsed and saved in JSON format. Develop a parsing script, traverse the log file in the script, extract the key information of each line or each log entry, and convert it into a structured data format. For JSON and XML format log files, use Python's json module or xml.etree.ElementTree) for parsing. For the most common plaintext format log files for middleware and related IT components, the parsing process is as follows:

[0130] Introduce Python's regular expression library and JSON library.

[0131] Define a multi-line string variable to store log raw data entries. Each log entry contains a timestamp, a log level (such as INFO or ERROR), and a descriptive message.

[0132] Define and compile a regular expression "log matching pattern" to identify and extract timestamps, log levels, and message content from log entries.

[0133] Initialize an empty list "Structured Log List" to store log entries parsed and converted into dictionary format.

[0134] Iterate over each line in the "log raw data", first use the strip() method to remove the whitespace characters at both ends of the string, then use the split("\n") method to split the string by newline characters to get a list containing all log lines.

[0135] Try to use the "log matching pattern" to match the current log line. If the current line conforms to the defined log format, return a match object; otherwise, return None.

[0136] Checks whether there is a matching object. If the current line matches the "log matching pattern", executes the subsequent process.

[0137] Create a dictionary "logentry-dictionary" that will store the timestamp, log level, and message content extracted from the matching objects.

[0138] Add dictionary 'log_entry_dictionary' to the list.

[0139] Use the JSON library to convert the parsed and structured log data in the above steps into JSON format.

[0140] Optionally, the operation of performing feature extraction on the cleaned log data to obtain feature data in step 102 includes:

[0141] Determine the feature set, including four categories: performance features, request processing features, error and transaction features, and system event features;

[0142] Define a unified data model that includes all possible fields, and specify standardized names and data types for each field;

[0143] Reading the structured log data, and identifying and extracting relevant feature data from the structured log data according to the feature set;

[0144] Data preprocessing is performed for numerical and non-numerical feature data respectively;

[0145] The preprocessed feature data are associated to generate a new data set.

[0146] In this embodiment, the performance-related features include:

[0147] Throughput: The number of requests processed per second, reflecting the processing capacity of the middleware.

[0148] Response Time: The average time from receiving a request to completing a response, including network delays and processing delays.

[0149] CPU Utilization: The percentage of CPU usage.

[0150] Memory Utilization: The ratio of the amount of memory used by the middleware to the total available memory.

[0151] Disk I / O: The frequency and amount of read and write disk operations, used to evaluate disk access performance.

[0152] Network I / O: Network data transmission rate, including upload and download speeds, reflects network performance.

[0153] Request processing characteristics include:

[0154] Request Type: The type of HTTP request.

[0155] Request Parameters: Parameters and data contained in the request.

[0156] Status Code: The HTTP status code of the response.

[0157] Session Duration: The start and end time of a user session.

[0158] 3) Error and transaction characteristics

[0159] Exception Stack: Exception stack information when an error occurs.

[0160] Error Code: A code that identifies the type of error.

[0161] Error Message: Detailed error description.

[0162] Transaction Status: The execution result of a transaction, such as "success", "failure", or "rollback".

[0163] System event characteristics include:

[0164] System Start / Stop: The time when the middleware and operating system start and stop.

[0165] Configuration Changes: System configuration modification events, including time and modified content.

[0166] System Calls: Call events at the operating system level.

[0167] In the feature identification and extraction process, the library (e.g., json module) provided by the programming language (e.g., Python) is used to read the JSON file, and the relevant information is identified and extracted from the JSON object of each log entry based on the predefined feature set.

[0168] A unified relational database table is used to store all the extracted log information. In a possible embodiment, the relational database table is shown in Table 1.

[0169]

[0170] Table 1

[0171] The minimum-maximum normalization method is used to process numerical feature data, adjust the scale of the features, reduce the scale differences between features, make different features comparable, and keep the relative relationship in the data unchanged.

[0172] The minimum-maximum normalization formula is as follows:

[0173]

[0174] in:

[0175] X is the raw data value, which corresponds to the specific value of "CPU utilization" or "Memory utilization" in each row.

[0176] X_min is the minimum value of the feature column, and X_max is the maximum value of the feature column, which correspond to the minimum and maximum values ​​of the numerical features in the entire data set.

[0177] X_norm is the normalized value obtained after applying the above formula.

[0178] Note: LogID is used as an identifier and does not participate in data conversion, model training and other operations in subsequent links.

[0179] Assume that two performance indicators, "CPU utilization" and "memory utilization", are selected from the extracted data set for normalization. The input data set example is as follows:

[0180]

[0181]

[0182] In the actual implementation, the Pandas library is used to process the data set, and the MinMaxScaler of the Sklearn library is used to normalize the feature columns that need to be processed. After applying the minimum-maximum normalization, the original data set will be converted to:

[0183]

[0184] Convert non-numeric feature data to One-Hot Encoding. One-Hot Encoding maps a category value to a binary vector:

[0185] OHE(c)=[b_1,b_2,...,b_n]

[0186] Where (c) is the original category value.

[0187] (n) is the total number of unique category values.

[0188] For each (b_i), if the category corresponding to (c) is the (i)th category, then (b_i=1), otherwise (b_i=0).

[0189] Assume that the log data set contains two non-numeric categorical features: request type (RequestType) and error code (ErrorCode). For each category, create a new binary column, where 1 indicates the existence of the category and 0 indicates the absence. The input data set example is as follows:

[0190]

[0191] The implementation steps are as follows:

[0192] Use the Pandas library to create a DataFrame.

[0193] Use Pandas's `get_dummies` function to perform one-hot encoding on the columns that need to be encoded.

[0194] The transformed DataFrame will include new columns created for each unique category value.

[0195] The following is an example of a transformed dataset, which contains the original LogID column and a one-hot encoded column generated for each category value:

[0196]

[0197]

[0198] The data after data preprocessing are associated with LogID. Each LogID row includes all the data columns after minimum-maximum normalization and the data columns after one-hot encoding conversion, thereby generating a new data set. An example is shown in Table 2:

[0199] LogID CPU Utilization Memory Utilization Request Type Request Type Error Code Error Code … 1 0.5 0.6 1 0 1 0 … 2 0.7 0.8 0 1 0 1 … 3 … … …

[0200] Table 2

[0201] Optionally, the method further includes:

[0202] Using different methods for each type of feature data to generate noise and adding it to the feature data to generate a training data set;

[0203] A denoising autoencoder model is constructed, and the denoising autoencoder model is trained according to the training data set.

[0204] Optionally, the data model includes:

[0205] Basic information, including: a unique identifier corresponding to a log entry or a performance indicator entry; a component type of the data source; and a specific component name;

[0206] Log-specific information, including: log level, detailed message content of the log, and error log.

[0207] Performance indicator information, the performance indicator information includes: the name of the performance indicator, the specific value of the performance indicator;

[0208] Additional information, which is used to store other related data or additional information that is useful in a specific context.

[0209] In this embodiment, when configuring the data model, the basic information is:

[0210] Provide a unique identifier for each log entry or performance indicator entry and set it as an auto-increment primary key. Data type: BIGINT.

[0211] The component type of the data source, such as "middleware", "database", "operating system", etc. Data type: VARCHAR, length: 255.

[0212] Specific component name, such as "Apache" for middleware, "MySQL" for database, etc. Data type: VARCHAR, length: 255.

[0213] Log-specific information:

[0214] Log level, such as "INFO", "ERROR", "DEBUG", etc. Data type: VARCHAR, length: 50.

[0215] The detailed message content of the log, that is, the specific information of the event. Data type: TEXT.

[0216] Error log, indicating the specific error code. Data type: VARCHAR, length: 100.

[0217] Performance indicator information:

[0218] The name of the performance indicator, such as "CPU utilization", "Memory usage", etc. Data type: VARCHAR, length: 255.

[0219] The specific value of the performance indicator. Depending on the performance indicator, this value can represent utilization, response time, etc. Data type: DECIMAL(10,2).

[0220] Additional information:

[0221] Stores any other related data or extra information that is useful in a specific context. Data type: TEXT.

[0222] Optionally, the adopting different methods for each type of feature data to generate noise and adding the noise to the feature data to generate a training data set includes:

[0223] Adding Gaussian noise to the feature data of the numerical type to simulate noise;

[0224] The bits in the one-hot encoding vector are randomly flipped with a preset probability to add noise to the feature data of the non-numeric class.

[0225] In this embodiment, for numerical feature data, noise is usually simulated by adding Gaussian (normal) noise, and the specific steps are as follows:

[0226] Determine the noise parameters: Choose appropriate mean 0 and standard deviation 0.1 to define a Gaussian distribution.

[0227] Generate noise: For each numerical feature, generate a corresponding number of noise values ​​according to the Gaussian distribution parameters mentioned above. Use Python's numpy.random.normal function to achieve this.

[0228] Add noise: Add the generated noise value to the original numerical feature to obtain the noisy feature value.

[0229] For non-numerical feature data, we randomly flip the bits in the one-hot encoded vector with a probability of 1% to add noise to the one-hot encoded categorical features, simulating the uncertainty of the classification information or the situation of wrong labels. Specifically, we use Python's numpy.random.choice function combined with the flipping probability to select the bits that need to be flipped, and then negate the values ​​of these bits.

[0230] Taking Python's NumPy library as an example, the implementation process is as follows:

[0231] Define a numerical noise function to add noise to a numerical feature array. The function receives the feature array, the mean value and the standard deviation of the noise as parameters.

[0232] Use the np.random.normal function to generate a noise array noise that is the same as the feature array. This noise follows a normal distribution with a mean of 0 and a standard deviation of 0.1.

[0233] Add the noise array to the original feature array to get the noisy feature array.

[0234] Define another feature noise function to add noise to the one-hot encoded categorical feature array. Receive the one-hot encoded feature array and the flip probability as parameters.

[0235] Iterate over each feature of each sample and randomly flip bits with a 1% probability, that is, change 1 to 0 or 0 to 1.

[0236] Finally, the numerical and categorical feature arrays after adding noise are output.

[0237] In a possible embodiment, the numerical feature is: [[0.5, 0.6], [0.7, 0.8]], and the categorical feature is: [1, 0, 0, 1, 0], [0, 1, 0, 0, 1]]. After adding noise, the numerical feature value becomes: [[0.56097536, 0.66005927], [0.71487396, 0.79054407]], and the categorical feature becomes: [[1, 0, 0, 1, 0], [0, 1, 0, 0, 1]].

[0238] Optionally, constructing the denoising autoencoder model includes: determining the dimension of the input data based on the feature data in the training data set; determining the number of layers and the number of neurons of the encoder and decoder in the denoising autoencoder model; determining the activation functions of the hidden layer and the output layer in the denoising autoencoder model; determining the loss function and optimizer in the denoising autoencoder model; and determining the initial weights and bias values ​​of each layer in the denoising autoencoder model.

[0239] In this embodiment, when determining the dimension of the input data, the input dimension of the model is determined based on the processed data set. Assuming that the merged data set contains 10 features (numerical features and categorical features after one-hot encoding), the number of neurons in the input layer will match these 10 features. The number of neurons in the input layer is equal to the number of features in the data set, which is set to n.

[0240] Choose the number of layers and neurons for the encoder and decoder

[0241] Encoder: Select an encoder with two hidden layers, with 6 and 3 neurons in each layer. The first layer reduces the dimensionality of the input data from 10 to 6, and the second layer further reduces the dimensionality to 3, resulting in a more compact data representation. Number of neurons in the hidden layer: n1 for the first layer of the encoder and n2 for the second layer; n1 for the first layer of the decoder and n for the output layer.

[0242] Decoder: The decoder chooses a structure symmetrical to the encoder, so it has two hidden layers with 6 and 10 neurons respectively.

[0243] When selecting an activation function, for the hidden layer, the ReLU (Rectified Linear Unit) function is selected as the activation function, which can speed up training while reducing the gradient vanishing problem. For the ReLU function, σ(x) = max(0, x).

[0244] For the output layer, since the dataset contains normalized numerical features and one-hot encoded categorical features, the Sigmoid activation function is selected to ensure that the output is between 0 and 1, matching the range of the input data.

[0245] Activation functions are responsible for introducing nonlinearity in neural networks, helping them learn complex data patterns. x represents the weighted sum of inputs to a neuron in a neural network layer, which consists of the dot product of the output of the previous layer and the weight of the current layer plus a bias term.

[0246] When defining the loss function and optimizer, the loss function (Loss Function): Select the mean squared error (MSE) that can measure the difference between the reconstructed data and the original data as the loss function. Loss function: L(x, g(f(x))) = ||xg(f(x))|| 2 , where x is the input data and g(f(x)) is the decoder output.

[0247] Optimizer: The Adam optimizer was chosen because it has shown good performance on many different deep learning tasks and is relatively insensitive to the choice of initial learning rate.

[0248] Finally, according to the dimension and characteristics of the input data, a denoising autoencoder model consisting of an encoder and a decoder was designed, and the initial weights and bias values ​​were set for each layer in the model.

[0249] Optionally, training the denoising autoencoder model according to the training data set includes: dividing the training data set into a training set and a test set according to a preset ratio; initializing the weights and biases of the denoising autoencoder model; loading the training set into the denoising autoencoder model; configuring the network structure, learning rate, training process, and verification process of the denoising autoencoder model; iteratively training the denoising autoencoder model and completing the training when the loss function reaches a preset value.

[0250] In this embodiment, the model is divided into a training set and a test set in a ratio of 70% training set and 30% test set, and the generalization ability of the model is evaluated during the training process.

[0251] Training preparation: Initialize the weights and biases of the denoising autoencoder (DAE) model to accelerate the convergence of model training, prevent the activation function from saturating at the beginning of training, and improve the efficiency and stability of model learning.

[0252] The initialization method selection method is as follows:

[0253] For the ReLU activation function: select He for initialization, and randomly select weights with a mean of 0 and a standard deviation of The normal distribution is completed, where n l-1 is the number of nodes in the previous layer.

[0254] For the Sigmoid activation function: select Xavier initialization, and randomly select weights with a mean of 0 and a standard deviation of The normal distribution is completed, where l represents the level number of the current layer, n l-1 is the number of neurons in the previous layer (i.e., layer l-1).

[0255] b) Implement initialization: Use the selected initialization method to assign initial values ​​to the weights and biases of each layer of the model. In deep learning frameworks (such as PyTorch), the weight initialization strategy is directly specified when defining the model. Taking PyTorch as an example, the implementation process is as follows:

[0256] Import the neural network module nn in PyTorch to provide the necessary tools for defining custom models and initialization.

[0257] Initialize the encoder to contain two linear layers, each followed by a ReLU activation function. The first linear layer reduces the dimension of the input data from the initial size to 6, and the second linear layer further reduces the dimension to 3.

[0258] Initialize the decoder, which is also a sequential model. The first linear layer increases the dimension from 3 to 6, and the second linear layer expands it back to the initial input size. The last linear layer is followed by a Sigmoid activation function to ensure that the output data value is between 0 and 1, matching the normalized input data.

[0259] Finally, the model traverses all modules (including linear layers in encoder and decoder) and applies He initialization method to these linear layers. For each linear layer, the weights are initialized using Kaiming normal distribution method and the bias is set to a constant value of 0.01.

[0260] By implementing the initialization step, the weights and biases of each layer of the model are given initial values ​​and saved in the model's parameters.

[0261] Data loading: Use PyTorch's DataLoader function to load the training data into the model. The training set has been split and contains data with noise added. The following is the data loading process:

[0262] a) Create a dataset object, create a TensorDataset object, which accepts two parameters: a tensor containing all training samples, and a corresponding label tensor. b) Initialize the data loader, initialize a DataLoader object to load the data provided by TensorDataset, where the dataset object to be loaded corresponds to the previously created training set. For example: define the size of each data batch as 64, and each time the training is iterated, 64 samples will be randomly selected from the dataset for training. Set the shuffle switch to True to randomly shuffle the order of the data in the dataset at the beginning of each training round to improve the generalization performance during the model learning process.

[0263] Model configuration: a) Determine the network structure: Choose a basic denoising autoencoder network structure, specifically: the input layer corresponds to the dimension of the feature, two hidden layers (for example, the first hidden layer is 50% of the feature dimension, the second hidden layer is 25%), and finally the output layer, whose dimension is the same as the input layer. Use ReLU as the activation function. b) Hyperparameter setting: Learning rate: Use 0.001 as the starting learning rate. Batch size: Choose a medium-sized batch, such as 32 or 64, to balance training speed and memory usage. Number of iterations: Set the maximum number of iterations to 100 rounds, but combine the early stopping strategy to prevent overfitting. c) Training process: Use Adam optimizer for parameter update. Use mean squared error (MSE) as the loss function. Implement early stopping strategy: Stop training if the loss on the validation set does not decrease significantly in 10 consecutive iterations. d) Model validation: Split 20% of the data as a validation set and the rest as a training set to evaluate the generalization ability of the model.

[0264] Model training cycle: Taking the PyTorch framework as an example, the process of model training is as follows:

[0265] a) Import the deep learning framework library PyTorch.

[0266] b) Set the parameter update algorithm: Define an optimizer (using Adam optimizer) to update the weights of the model. Get all trainable parameters in the model and set a small step value as the learning rate (set to 0.001 in this case).

[0267] c) Define loss function: Define the loss function to evaluate the performance by comparing the difference between the predicted output of the model and the actual target value. Here, mean squared error loss is used.

[0268] d) Set the number of iterations: Specify that the total number of times the data will be traversed during model training is 20.

[0269] e) Training loop starts: Open the outer loop and iterate according to the set rounds.

[0270] f) Traverse data batches: Open an internal loop, traverse the training data loader, and process the data one by one in batches. Each batch contains a set of data and corresponding labels for training the model.

[0271] g) Clear previous gradient information: Before each iteration, clear the accumulated gradient information to avoid interfering with the calculation of the current step.

[0272] h) Forward propagation: The model forward propagates the input data and calculates the model output, that is, the model processes each batch of data and generates prediction results.

[0273] i) Calculate loss: According to the model outputs and the true label labels, the loss value is calculated using the loss function criterion.

[0274] j) Back propagation: Back propagation is performed based on the loss value, the calculated error value is passed back to the model, and the gradient of each parameter is calculated.

[0275] k) Update parameters: Use optimizer.step() to update the model parameters according to the calculated gradients, and adjust the internal parameters of the model through the parameter update algorithm.

[0276] l) Output training information: At the end of each round of training, the number of the current round and the error value of the last batch of the round are output to monitor the progress of training.

[0277] The following output shows the loss value (Loss) calculated by the model at the end of each epoch in multiple training rounds (Epochs). The loss value gradually decreases as the training progresses, which shows that the model is gradually learning and the prediction results are getting closer and closer to the true label.

[0278] Epoch 1, Loss: 0.0234

[0279] Epoch 2, Loss: 0.0187

[0280] Epoch 3, Loss: 0.0156

[0281] Epoch 4, Loss: 0.0132

[0282] Epoch 5, Loss: 0.0115

[0283] Epoch 6, Loss: 0.0101

[0284] After each training round, the model performance is evaluated on the validation set to monitor whether the model is overfitting or underfitting. The specific process is as follows: a) Isolate a part of the original dataset as a validation set. Make sure that this part of the data is not used during the model training process to fairly evaluate the generalization ability of the model. b) Switch the model to evaluation mode. Taking PyTorch as an example, the model can be set to evaluation mode by calling the .eval() method of the model. Turn off operations unique to the training phase (such as Dropout) to ensure the accuracy of the evaluation process. c) Disable gradient calculation during the evaluation process to save computing resources and speed up the operation. This is achieved by using torch.no_grad().

[0285] d) Traverse the validation set data, for each batch: use the model to perform forward propagation and obtain the model prediction results. Use the predefined loss function to calculate the loss value. The specific process is as follows:

[0286] i. Set the cumulative loss, cumulative accuracy index and number of samples processed to zero.

[0287] ii. Traverse the loader of the validation dataset, which contains the batched feature sets and corresponding label sets.

[0288] iii. For each batch of data, prediction is performed through the model (forward propagation) to obtain a set of prediction results.

[0289] iv. Based on the prediction result set and the corresponding label set, use the loss function calculation formula to calculate the cumulative loss of the current batch.

[0290] v. Multiply the cumulative loss of the current batch by the number of samples in the batch, add it to the total cumulative loss, and finally calculate the average cumulative loss on the entire validation set.

[0291] vi. If the task is a classification problem, determine the predicted category of each sample based on the prediction result set, and compare it with the actual label to calculate the number of correct predictions and add it to the total accuracy.

[0292] vii. Update the total number of samples processed, plus the number of samples in the current batch.

[0293] viii. After completing all batches, the average cumulative loss is calculated by dividing the total cumulative loss by the total number of samples, and the overall accuracy is calculated by dividing the total accuracy by the total number of samples.

[0294] ix. Output the calculated average cumulative loss and overall accuracy.

[0295] e) Based on the performance on the validation set, analyze whether the model has overfitting or underfitting problems, and consider necessary adjustment measures. In this embodiment, the loss rate during training is less than the initial loss value * 10%, which is considered to be low. In the classification task, an accuracy rate of more than 80% can be considered to meet the requirements.

[0296] Optionally, step 103 determines abnormal logs in the log data according to the feature data and expands the abnormal logs, including: determining the category distribution of the log data according to the denoising autoencoder model, and determining that a minority of categories therein are abnormal logs; oversampling the abnormal logs to generate expanded samples, and forming an expanded data set with the expanded samples; and merging the expanded data set with the original log data.

[0297] Optionally, step 104 uses a clustering optimization method based on a kernel matrix to perform anomaly detection on the log data, including: dividing the log data into a feature set and a label set, wherein the feature set includes features for anomaly detection and the label set includes corresponding category information; selecting a kernel function and constructing a kernel matrix; initializing clustering centers and clustering the data in the feature set; and evaluating the clustering results.

[0298] In this embodiment, data enhancement is performed by applying the SMOTE algorithm to oversample the abnormal logs (minority class) of the middleware, especially for uncommon but important abnormal events, to solve the problem of data category imbalance.

[0299] The aforementioned preprocessing and feature engineering improve the model's ability to identify abnormal logs, but some abnormal event logs may be relatively small, which will lead to data category imbalance, which may affect the performance of the model. The SMOTE algorithm is applied to enhance the data set, especially for minority class abnormal events, to improve the model's ability to identify these uncommon but important events.

[0300] The log input after processing in step 103 is as follows:

[0301]

[0302] Analyze the category distribution of the dataset: 1) Import necessary libraries, use Python's data processing library such as Pandas, and the visualization library Matplotlib to analyze the category distribution. 2) Load the dataset: Load the processed dataset into Pandas' DataFrame. Assume that the category label is in the column category. 3) Count the number of samples in each category: Use Pandas' value_counts() method to calculate the number of samples in each category. 4) Visualize the category distribution: Use a bar chart to visualize the number of samples in each category. The specific method is as follows:

[0303] a) Create a new graphics window and specify the window size height and width. b) Use the barplot function of the Seaborn library to draw a bar chart. x specifies the horizontal axis of the bar chart, which is the index of the category label; y specifies the vertical axis of the bar chart, which is the number of samples in each category. c) Set the label of the x-axis to "Category" and the label of the y-axis to "Number of Samples". Set the rotation angle of the x-axis scale label to 45 degrees to prevent the label text from overlapping each other. d) Display the graph. Figure 2 It is a histogram corresponding to a data set. Taking the sample data set as an example, the final output histogram includes the number of samples corresponding to each category.

[0304] 5) Identify minority categories: Based on the visualization results and sample size statistics, identify those categories with relatively small sample sizes. These minority categories will be the targets for subsequent oversampling using the SMOTE algorithm.

[0305] Applying the SMOTE algorithm: The main parameters of the SMOTE algorithm include the number of neighbors (k), the oversampling ratio, and the distance metric used to generate new samples. The steps to select these parameters using the SMOTE class in Python's imbalanced-learn library are as follows:

[0306] Use the SMOTE class in Python's imbalanced-learn library to implement oversampling. The specific process is as follows:

[0307] 1) Import the SMOTE class from the imbalanced-learn library. 2) Load the dataset. 3) Create a SMOTE instance. The sampling strategy is selected as automatic, which means that new samples are automatically generated according to the number of minority classes to achieve class balance. The number of neighbors k is selected as 5, so that the 5 nearest neighbors of the minority class samples are used to generate new samples. 4) Use the fit_resample method to apply the SMOTE algorithm to the dataset. The first parameter of this method represents the feature matrix, which contains all encoded features such as Feature_1 to Feature_5, and the second parameter is a vector corresponding to the category label, such as Error, Warning, Info, etc.

[0308] The category distribution of the data set processed in step 103 is unbalanced, especially the number of samples in the Critical and Debug categories is relatively small. After the above processing, the number of samples in the minority category is increased through SMOTE oversampling, making the number of samples in each category more balanced. The following is an output example after oversampling processing as shown in Table 3:

[0309]

[0310]

[0311] Table 3

[0312] "X" represents the index of the last row in the original dataset plus one, which is used to mark the starting position of the augmented dataset.

[0313] The augmented data set enhanced by the SMOTE algorithm is merged with the original data set processed in step 103, both of which contain feature codes and category labels.

[0314] The merging method uses the merging function of the data processing library, such as concat, to merge the original data set and the augmented data set along the sample axis (data rows), that is, setting the axis parameter to 0.

[0315] The K-Means optimization method based on the kernel matrix is ​​used to perform anomaly detection on the data set after preprocessing, feature engineering and data enhancement in the above steps.

[0316] Preparation. Feature and label separation: Divide the dataset into two parts, feature set (x) and label set (y). The feature set contains all the features used for anomaly detection, while the label set contains the corresponding category information. Taking the dataset output in step 103 as an example, the feature set is a matrix in which each row represents a data point and each column represents a feature. The label set (y) includes the classification label corresponding to each data point in the feature set, i.e., the Category column.

[0317] Prepare the dataset and use the array() method to convert the feature set X and label set y into NumPy arrays (or other formats suitable for processing, Python implementation is used here as an example).

[0318] Construct the kernel matrix. The RBF kernel can complete most middleware anomaly detection tasks, can handle nonlinear features, and can effectively map data to high-dimensional space, so it is selected as the kernel function. The process is as follows:

[0319] 1) Define a function to create a kernel matrix based on the similarity between data points, thereby revealing the distribution of data in the new feature space. The function accepts two parameters: a set of data points and a gamma parameter (γ).

[0320] Among them, the current gamma parameter (γ) adopts the default value (gamma = 1 / n_features, where n_features is the number of features)

[0321] 2) Get the sample size of the data point set, that is, the total number of data points in the data point set.

[0322] 3) Initialize a matrix of all zeros to store the values ​​of the kernel matrix, with dimensions of the number of samples times the number of samples, so that each data point is compared with all other data points.

[0323] 4) Start a loop, iterating from the first data point to the last data point. This loop is used to select a benchmark data point.

[0324] 5) Inside the first loop, start another loop, again iterating from the first data point until the last data point. This loop is used to compare the benchmark data point with all other data points.

[0325] In the inner loop, the similarity between the reference data point and the current data point being compared is calculated and stored in the corresponding position of the kernel matrix. The similarity is calculated by the negative exponential of the difference between the gamma parameter and the square of the Euclidean distance between the two data points.

[0326] Specifically, each kernel matrix element K ij The calculation uses the Gaussian radial basis function (RBF) kernel formula (exp(-γ·|x i -x j | 2 )), where (|x i -x j | 2 ) represents the data point x i and x j The square of the Euclidean distance between them, and exp refers to the natural exponential function. Through this formula, the similarity of data points in the high-dimensional feature space can be quantified, and then the kernel matrix K can be constructed.

[0327] Combined with the data set, the feature set X contains multi-dimensional feature information, such as `FEATURE_1`, `FEATURE_2` to `FEATURE_5`, etc., which represent the characteristics of log data in different dimensions. Correspondingly, each data point x i and x j represents two sample points in this feature set, that is, two rows of data. i -x j | 2 ) is calculated as x i and x j The square of the Euclidean distance between the two sample points is calculated, that is, the sum of the squares of the differences between the two sample points in all feature dimensions. For example, if x i is the first sample in the dataset, and x j is the second sample, the square of the Euclidean distance between them is calculated by calculating the difference between the values ​​of the two on each feature (such as `FEATURE_1` to `FEATURE_5`), and then adding the squares of these differences.

[0328] When all data points have been compared, the filled kernel matrix is ​​returned.

[0329] Taking the above sample data set as an example, the kernel matrix K is as follows:

[0330]

[0331] The kernel matrix K is of size N×N, where N is the number of data points.

[0332] Initialize the cluster center. The specific steps are as follows:

[0333] Determine the number of clusters: Determine the number of clusters to be formed, K. In the environment of the middleware and related IT systems in this case, this number is 3, which means there are three different anomalies: performance anomalies, functional failures, and security-related issues.

[0334] Select initial cluster centers: In the data set, the feature set contains multiple dimensions (such as Feature_1, Feature_2,..., Feature_5, etc.). Use the random selection method to randomly select K data points from the feature set as the initial cluster centers.

[0335] a) Number of clusters (K): Use the value confirmed in the previous step: 3.

[0336] b) Feature data: that is, the feature set (X), stored in the form of a two-dimensional array or Pandas DataFrame, where the data should include all feature columns used for clustering, such as Feature_1 to Feature_5.

[0337] c) Select initial cluster centers: Use Numpy’s random.choice method to randomly select the indices of K data points.

[0338] d) Based on the index obtained in the previous step, extract these data points from the feature set (X). These points will serve as the centers of the initial clusters.

[0339] The example is as follows: In this case, the feature set (X) contains the following data points, each row represents a data point, and each column represents a feature:

[0340]

[0341] Randomly select 3 of these data points as the initial cluster centers. Assuming the randomly selected index is [0,2,3], the 1st, 3rd and 4th rows are selected from the feature set as the initial cluster centers. Based on these indexes, extract the corresponding data points from the feature set (X) to obtain the initial cluster centers. The example is as follows:

[0342]

[0343] Cluster assignment and center update: assign each data point to the nearest cluster center to form a cluster. This process can be implemented in a programming language as follows (using the numpy library as an example):

[0344] 1) Set a matrix representing a feature information set (feature set (X)), a matrix representing a cluster center point set, and a kernel matrix representing data similarity that has been calculated in advance.

[0345] 2) Get the total number of data points by getting the number of rows in the feature set matrix.

[0346] 3) Determine the number of cluster center points by obtaining the number of rows of the center point set matrix.

[0347] 4) Initialize an allocation array for each data point, the size of which is equal to the total number of data points. This array is used to record the index of the cluster center to which each data point is assigned.

[0348] 5) Define the function of calculating the core of the distance from the data point to the cluster center in the kernel space. The first step is to calculate the similarity between a specific data point and itself in the kernel space. This is obtained by looking at the same position of the corresponding row and column of the data point in the kernel matrix, which is expressed as the "self-similarity term". This term is essentially the inner product of the data point with itself in the high-dimensional space. The second step is to calculate the average similarity in the kernel space between the data point and all the data points contained in the cluster center (for a given cluster, the average similarity can be obtained by calculating the average of the corresponding values ​​of all data points in the cluster in the kernel matrix. Specifically, if the kernel matrix is ​​K and the set of data point indices contained in the cluster is S, then the average similarity of the cluster can be calculated by the following formula: This is done by finding the row corresponding to the data point in the kernel matrix and the column corresponding to all the data points representing the cluster center, and calculating the average of their similarities. This process uses a portion of the kernel matrix, expressed as "twice the similarity with the cluster center", because this value will be multiplied by 2 in the final distance calculation. The third step is to calculate the average similarity of the data points inside the cluster center in the kernel space. For all the data points in the cluster center, find their similarities with each other in the kernel matrix and calculate the average of these similarities.

[0349] 6) For each data point in the feature set, perform the following operations:

[0350] a) Initialize a distance array to store the distance from the data point to each cluster center.

[0351] b) For each cluster center, calculate the distance from the current data point to the center. The specific method is to call the "core function for calculating the distance from the data point to the cluster center in the kernel space" defined in the previous link, and calculate the distance based on the kernel matrix and the information of the current data point and the cluster center point.

[0352] c) After calculating the distances of all cluster centers, find the index of the minimum distance and record this index as the cluster assignment result of the current data point.

[0353] The output of this step is the cluster label to which each data point belongs, represented as an array where each element corresponds to the cluster assignment result for each data point in the feature set.

[0354] Update the cluster center to make it the center (mean) of all points in the cluster. Repeat the cluster assignment and center update steps until the stopping condition is met.

[0355] 1) Recalculate the center of each cluster: Based on the cluster assignment results of each data point, recalculate the center of each cluster by averaging all data points belonging to the same cluster in the kernel space. Taking Python programming language and NumPy library as an example, the implementation method is as follows:

[0356] 1) Indirect update of cluster centers: Since the kernel space is used, the new location of the cluster center cannot be directly calculated, but it is achieved by finding the data point that best represents the current cluster members. Specifically, it includes:

[0357] a) Define a function that accepts three parameters: a list of cluster member indices, a kernel matrix, and the number of clusters.

[0358] b) Inside the function, create an empty list to store the calculated new cluster centers.

[0359] c) Loop through the indices of each cluster, ranging from 0 to the number of index lists - 1.

[0360] d) For each cluster, get the index of all members of the current cluster through the index list + index value, and store it in the first variable. If this variable is not empty, execute: get the kernel matrix value between all members in the current cluster, store it in the second variable, calculate the average value of this variable along each row, which represents the average similarity of each member with other members in the cluster, and store the result in the third variable and add it to a new list (assuming it is new_centers).

[0361] e) If the current cluster has no members, add None to the new_centers list as a placeholder.

[0362] f) The function finally returns the new_centers list, which contains the new centers of all clusters.

[0363] 2) Indirect update of cluster centers: Since the kernel space is used, the new location of the cluster center cannot be calculated directly, but it is achieved by finding the data point that best represents the current cluster members. Specifically, it includes:

[0364] a) Calculate the similarity within the cluster: traverse each cluster, select the rows and columns corresponding to the members of the cluster in the kernel matrix, and form a sub-kernel matrix. This sub-matrix contains the similarities between all points in the cluster. For each member in the cluster, calculate the average of all its elements in the sub-kernel matrix, which represents the average similarity between the member and other members of the cluster. Record the average similarity of each member.

[0365] b) Select new cluster centers: For each cluster, find the member with the highest average similarity, that is, in the sub-core matrix, select the row with the largest average value. This member has the highest average similarity within the cluster and can therefore become the new cluster center.

[0366] c) Update the index of the cluster center: record the index of the new cluster center selected in the previous step. These indexes represent the corresponding rows in the original feature set.

[0367] 3) Selection of cluster centers: For each cluster, traverse all its members, calculate the average distance from each point to other members of the cluster in the kernel space, and select the point with the smallest average distance as the new cluster center. Specifically, it includes: randomly selecting several data points in the feature set as the initial cluster centers. Select the initial cluster centers randomly from the feature set or use more complex methods to reduce the randomness of the clustering results. In each iteration, update the center of each cluster according to the current cluster assignment results. For each cluster, calculate the average similarity of all members in the cluster in the kernel matrix. After updating the cluster centers, determine the indexes of these new centers in the original feature set. By comparing the average similarities of members in the cluster, select the index of the data point that best represents the cluster as the new cluster center.

[0368] Through the above steps, the cluster center will be updated according to the characteristics of the current cluster in each iteration, and the clustering result can better reflect the real distribution of the data. This process is repeated many times until the change of the cluster center is very small or the predetermined number of iterations is reached, and the clustering process ends.

[0369] Clustering effect evaluation: After completing the K-Means clustering method using the kernel matrix, use the silhouette coefficient and Davies-Bouldin index as evaluation indicators to conduct a comprehensive clustering effect evaluation to ensure that the model training and optimization process can be reproduced and verified. Among them, the silhouette coefficient (Silhouette Coefficient): measures the difference between the similarity of each sample point with other points of the same category and the similarity of points of different categories. The value range of the silhouette coefficient is from -1 to 1, and the larger the value, the better the clustering result. Davies-Bouldin Index (Davies-Bouldin Index): Evaluate the clustering quality based on the similarity within the cluster and the dissimilarity between clusters. The smaller the value of this index, the higher the clustering quality. The evaluation process is as follows: Calculate the above indicators for the clustering results. Explain the calculation method of each indicator and its significance in clustering evaluation. Based on the results of the evaluation indicators, judge the effectiveness of clustering and provide suggestions for improving clustering quality (if necessary).

[0370] The key points and protection points of the present invention include: constructing a similarity measure based on the RBF kernel matrix, mapping the data to a high-dimensional space, initializing the cluster center, and optimizing the position of the cluster center through an iterative update strategy to improve the accuracy and reliability of clustering. The silhouette coefficient and the Davies-Bouldin index are used to evaluate the clustering effect to ensure the effectiveness and reliability of the clustering method. The SMOTE algorithm is used to oversample the abnormal events (minority classes) in the middleware log to solve the problem of data category imbalance and enhance the model's recognition ability for rare abnormal events. The denoising autoencoder (DAE) is used to extract key features, including performance indicators, request processing, error and transaction features, system events, etc. Numerical and non-numerical features are normalized and uniquely encoded to optimize model input. Specific data cleaning techniques are used, including log format parsing, irrelevant information filtering, and missing value processing to ensure data quality. The complete data processing process, including every step from data collection, cleaning, feature extraction, data enhancement to anomaly detection, ensures efficient conversion from raw log data to effective anomaly detection.

[0371] In order to implement the above embodiments, the present application also proposes a middleware anomaly detection device based on autoencoder and data enhancement. Figure 3 The schematic diagram of the structure of a middleware anomaly detection device based on autoencoder and data enhancement provided in an embodiment of the present application. Figure 3 As shown, the device comprises:

[0372] A data cleaning module 310 is used to collect log data of the middleware and related components, and clean the log data according to the characteristics of the middleware log;

[0373] A feature extraction module 320, configured to perform a feature extraction operation on the cleaned log data to obtain feature data;

[0374] A data expansion module 330, configured to determine an abnormal log in the log data according to the characteristic data and expand the abnormal log;

[0375] The anomaly monitoring module 340 is used to perform anomaly detection on the log data using a kernel matrix-based clustering optimization method.

[0376] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0377] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0378] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0379] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0380] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0381] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0382] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0383] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0384] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0385] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0386] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0387] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0388] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0389] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A middleware anomaly detection method based on autoencoder and data enhancement, characterized in that: The following steps are involved: Collect log data of middleware and related components, and clean the log data according to the characteristics of the middleware logs; Performing a feature extraction operation on the cleaned log data to obtain feature data; Determine an abnormal log in the log data according to the characteristic data and expand the abnormal log; Anomaly detection is performed on the log data using a kernel matrix-based clustering optimization method.

2. The method according to claim 1, characterized in that The log data of the middleware and related components are collected, including: Determine the target middleware type and target related components, wherein the target related components include: database system, operating system, load balancer; Configuring so that the target middleware type and the target-related component output the log data; A data collection script is deployed to automatically collect the log data and temporarily store it in the file system.

3. The method according to claim 2, characterized in that The log data corresponding to the middleware includes at least one of the following: request log, error log, transaction log, system event log, performance log; The log data corresponding to the database system includes at least one of the following: SQL query log, error log, transaction log, performance index; The log data corresponding to the operating system and the load balancer includes at least one of the following: a system event log, an error log, a resource usage log, and a load balancer log.

4. The method according to claim 3, characterized in that The step of cleaning the log data according to the characteristics of the middleware log includes: Parsing the log data according to the log format, extracting key information from each line or each log entry, and converting the information into a structured data format; Filter irrelevant information and exclude entries in the log data that are irrelevant to the normal operation or performance of the system; The data type and format of the log data after filtering are unified.

5. The method according to claim 4, characterized in that The key information of each line or each log entry is extracted and converted into a structured data format, including: Define a multi-line string variable to store log entries of the log data, each log entry contains a timestamp, log level and descriptive message; Define and compile regular expressions for log matching patterns, which are used to identify and extract timestamps, log levels, and message content in log entries; Initialize a structured log list, where the structured log list is used to store log entries that are parsed and converted into a dictionary format; Traverse each line in the log data to obtain a list containing all log lines; Use the regular expression of the log matching pattern to match each of the log lines, and return a matching object for the log lines that match the regular expression; Creating a log entry dictionary, and storing the timestamp, log level, and message content extracted from the matching object in the log entry dictionary; The log entry dictionary is added to the list, and the list is converted to a target format to obtain structured log data.

6. The method according to claim 5, characterized in that The operation of performing feature extraction on the cleaned log data to obtain feature data includes: Determine the feature set, including four categories: performance features, request processing features, error and transaction features, and system event features; Define a unified data model that includes all possible fields, and specify standardized names and data types for each field; Reading the structured log data, and identifying and extracting relevant feature data from the structured log data according to the feature set; Data preprocessing is performed for numerical and non-numerical feature data respectively; The preprocessed feature data are associated to generate a new data set.

7. The method according to claim 6, characterized in that The method further comprises: Using different methods for each type of feature data to generate noise and adding it to the feature data to generate a training data set; A denoising autoencoder model is constructed, and the denoising autoencoder model is trained according to the training data set.

8. The method according to claim 6, characterized in that The data model includes: Basic information, including: a unique identifier corresponding to a log entry or a performance indicator entry; a component type of the data source; and a specific component name; Log-specific information, including: log level, detailed message content of the log, and error log. Performance indicator information, the performance indicator information includes: the name of the performance indicator, the specific value of the performance indicator; Additional information, which is used to store other related data or additional information that is useful in a specific context.

9. The method according to claim 7, characterized in that: The method of using different methods to generate noise for each type of feature data and adding the noise to the feature data to generate a training data set includes: Adding Gaussian noise to the feature data of the numerical type to simulate noise; The bits in the one-hot encoding vector are randomly flipped with a preset probability to add noise to the feature data of the non-numeric class.

10. The method according to claim 7, characterized in that The construction of the denoising autoencoder model comprises: Determine the dimension of input data according to the feature data in the training data set; Determining the number of layers and neurons of the encoder and decoder in the denoising autoencoder model; Determining activation functions of hidden layers and output layers in the denoising autoencoder model; Determining a loss function and an optimizer in the denoising autoencoder model; Determine initial weights and bias values ​​for each layer in the denoising autoencoder model.

Images