AIOps-based data center full-stack intelligent operation and maintenance and root cause analysis system

Abstract

This invention discloses a full-stack intelligent operation and maintenance and root cause analysis system for data centers based on AIOps, belonging to the field of intelligent operation and maintenance technology. It includes a data acquisition module, a processing module, a storage module, a detection module, an analysis module, and an execution module. The acquisition module captures heterogeneous data and removes duplicates using EBPF probes; the processing module transforms, cleans, and populates the data; the storage module flexibly manages the data; the detection module identifies anomalies through time-series modeling; the analysis module calibrates and locates root causes based on causal inference and historical data; and the execution module automatically repairs data, improving data processing efficiency and root cause location accuracy, thereby enhancing operation and maintenance efficiency and precision.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology, and in particular to a full-stack intelligent operation and maintenance and root cause analysis system for data centers based on AIOps. Background Technology

[0002] As critical infrastructure, data centers are continuously expanding in scale, significantly increasing the complexity of the services they support, and experiencing explosive growth in heterogeneous datasets such as network traffic, hardware status, and server logs. Traditional operation and maintenance models that rely on manual inspections and rule engines are struggling to cope with the demands for real-time processing of massive amounts of data and the accurate identification and rapid location of faults and anomalies, resulting in problems such as low operation and maintenance efficiency and long downtime due to faults.

[0003] Existing operation and maintenance technologies lack unified metadata management and efficient redundancy identification methods, resulting in large amounts of data redundancy and inconsistent structures, which wastes resources and affects the accuracy of root cause analysis. Furthermore, root cause analysis is difficult to deeply explore the causal relationships between nodes and lacks historical data calibration, leading to low accuracy in root cause localization. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a full-stack intelligent operation and maintenance and root cause analysis system for data centers based on AIOps. It improves data processing efficiency and accuracy by efficiently collecting and filtering heterogeneous data, and significantly improves the efficiency and accuracy of data center operation and maintenance through precise root cause analysis and automatic repair mechanisms.

[0005] The technical solution to achieve the objective of this invention is as follows:

[0006] AIOps-based full-stack intelligent operation and maintenance and root cause analysis system for data centers, including acquisition module, processing module, storage module, detection module, analysis module and execution module;

[0007] The acquisition module captures heterogeneous datasets at equal intervals based on the EBPF probe and generates dataset snapshots. It collects field structures through a dynamic metadata registry, retrieves redundant field libraries based on a hash index matching algorithm, filters redundant fields in heterogeneous datasets, and generates deduplicated datasets.

[0008] The processing module converts the deduplicated dataset into a numerical dataset and performs anomaly cleaning and missing value imputation based on the fusion engine to generate a standard dataset. The numerical dataset includes the original numerical data in the deduplicated dataset and the equivalent numerical data generated by converting the original structured data. Anomaly cleaning is based on the random forest algorithm and entropy statistics, and missing value imputation is based on the nearest neighbor algorithm within the first sliding window.

[0009] The storage module efficiently stores dataset snapshots, standard datasets, and operation and maintenance logs through global data heat management and performs elastic management.

[0010] The detection module receives the standard dataset at the current moment and calls the historical standard dataset within the first time period before the current moment. It combines and constructs a detection sequence and performs temporal modeling through a bidirectional long short-term memory network. It extracts temporal feature vectors and uses a deep classification network to map the output anomaly probability distribution. It performs probability correction based on confidence assessment, retains anomaly classes whose anomaly correction probability is greater than or equal to the probability threshold, and generates an anomaly class set.

[0011] The analysis module receives the anomaly class set and constructs a node dependency graph based on the dataset snapshot. It identifies the upstream node set of each anomaly class in the anomaly class set, analyzes the causal relationship of nodes through a causal inference model, calculates the root cause probability of each upstream node of each anomaly class using a dynamic Bayesian network, calls the relevant historical operation and maintenance logs of each anomaly class for similarity matching to linearly calibrate the root cause probability of each upstream node of each anomaly class, determines the root cause node of each anomaly class, and constructs a type-node group set.

[0012] The execution module retrieves the repair strategy library based on the type-node group retrieval rule engine, generates automatic execution instructions for each type-node group, and sends them to the corresponding root cause node to perform repair operations. It packages the type-node group, standard dataset, execution time, automatic execution instructions, and execution results to generate operation and maintenance logs.

[0013] Furthermore, the acquisition module includes a capture unit and a preprocessing unit;

[0014] The crawling unit uses an EBPF probe compiled based on CO-RE to map each field of the heterogeneous dataset to an intermediate representation and record the mapping relationship. The crawling program of the EBPF probe is compiled based on the intermediate representation. When the EBPF probe crawls, CO-RE compiles and extracts BTF information from the device or application. The data is crawled through symbol resolution and dynamic binding mechanism to build a heterogeneous dataset. The dataset snapshot is copied and delivered to the storage module.

[0015] The preprocessing unit collects the field structure of each type of data in the heterogeneous dataset through a dynamic metadata registry. It compares the fields of the heterogeneous dataset with the historical heterogeneous dataset in the dynamic metadata registry using a structure comparison algorithm, identifies and synchronizes the field structure, and compares the dataset snapshot with the structure comparison algorithm to verify the synchronization accuracy. Based on the hash index matching algorithm, it performs fast hash verification on the field structure of the heterogeneous dataset, generates a hash value to be verified for each field through the hash algorithm, matches it with the hash index in the redundant field library, filters out the successfully matched hash values ​​to be verified, generates a deduplicated dataset, and delivers it to the processing module.

[0016] Furthermore, the processing module distinguishes between numerical and structured data in the deduplicated dataset. It converts text, date, and discrete fields in the structured data into word vectors, timestamps, and integers based on word embedding, field parsing, and label encoding, respectively, to generate equivalent numerical data. The fusion engine is then activated, and the numerical dataset is anomaly located using the random forest algorithm and verified and removed based on entropy statistics to achieve anomaly cleaning. Missing data in the numerical dataset is counted, and missing data is filled using the nearest neighbor algorithm within the first sliding window to generate a standard dataset.

[0017] Furthermore, the abnormal cleaning process performed by the cleaning filling unit includes the following steps:

[0018] Input the numerical dataset into the random forest. The root node of the tree;

[0019] Confirm the Given the selected field and split value of the current node in a tree, extract the field value of the selected field from the numerical dataset at the current node. , The total number of trees;

[0020] Based on the relationship between the splitting value of the current node and the field value of the selected field, the numerical dataset is assigned to either the left or right child node of the current node, making the numerical dataset at the [missing information - likely a specific node or region]. Path length of trees Add 1;

[0021] Recursively perform the partitioning until the numerical dataset is partitioned to the first position. Stop at the leaf node of the tree, and obtain the numerical dataset at the [number]th [node]. Path length of trees Based on numerical datasets Average path length of trees Calculate anomaly scores ;

[0022] abnormal scores Numerical datasets that are greater than or equal to the anomaly score threshold are marked as potential anomaly datasets. All historical standard datasets for the first time period before the current moment are called as historical sequences. The historical field entropy value of each field in the historical sequence is calculated. Potential anomaly datasets are assigned to the historical sequence and the field entropy value of each field is recalculated.

[0023] Based on the field entropy value and the rate of change of the corresponding historical field entropy value, abnormal fields are marked and the data belonging to the abnormal fields are cleaned to generate a standard data subset.

[0024] Furthermore, the construction of random forests requires using historical standard datasets as training sequences, based on... The secondary bootstrap sampling randomly extracts historical standard datasets with replacement from the training sequence to obtain... The nth different training subsequence, the nth The training subsequence is used as the th training subsequence The root node of the tree, , , , will the The training subsequence is used as the th training subsequence The root node of the tree is randomly selected from a historical standard dataset. A splitting value is then randomly chosen within the range of values. Based on the selected splitting value, the root node is split into child nodes. For each child node, a field and splitting value are randomly selected again to continue splitting, until the root node is... The historical standard dataset is the same in all unsplit child nodes of the tree, the th Once the tree training is complete, define the unsplit child nodes as leaf nodes and continue training. Use trees to build a random forest.

[0025] Specifically, missing data filling is based on the first sliding window selection to call historical standard datasets, confirm the missing data in the standard data subset at the current moment, and for each type of missing data, calculate the average similarity between each data in the standard data subset at the current moment and the corresponding data in each historical standard dataset within the first sliding window. The historical standard dataset with the highest average similarity is taken as the nearest neighbor of the standard data subset at the current moment, and the data corresponding to the missing data is selected from the nearest neighbors to fill in the standard data subset at the current moment, thus generating the standard dataset at the current moment.

[0026] Furthermore, the storage module stores the standard dataset in a relational database and employs global data heat management. Based on a first time interval and a second time interval, data prior to the current moment is divided into hot data, warm data, and cold data, where the first time interval is shorter than the second time interval. The latest standard dataset is moved as hot data to the resident cache database, the earliest historical standard dataset in the resident cache database is moved as warm data to the memory area of ​​the time series database, and the earliest historical standard dataset in the memory area of ​​the time series database is moved as cold data to the disk area of ​​the time series database. A two-dimensional partitioning strategy based on timestamps and Internet protocols is used to allocate a dedicated root directory for the dataset snapshot at the current moment in the distributed object storage and write it. The operation and maintenance logs are stored in the relational database. When the operation and maintenance logs are determined to be cold data over time, a hash function is used to encrypt and calculate the operation hash before moving it to the distributed object storage and synchronizing the operation hash to the blockchain evidence storage interface.

[0027] Furthermore, the detection module includes a temporal modeling unit, a deep classification unit, and a confidence correction unit;

[0028] The temporal modeling unit receives the standard dataset for the current moment and calls the historical standard dataset within the first time period before the current moment. It normalizes and constructs the detection sequence and inputs it into the bidirectional long short-term memory network. The forward long short-term memory network and the reverse long short-term memory network perform forward modeling and reverse modeling from the earliest moment and the current moment in the detection sequence, respectively. The reverse modeling features are flipped and concatenated with the forward modeling features and input into global average pooling to generate a temporal feature vector.

[0029] The deep classification unit uses three layers of one-dimensional convolutions of different sizes to process the temporal feature vectors, extracting adjacent features, short-period features and long-period features respectively. The first feature vector, the second feature vector and the third feature vector are generated by the fully connected layer and the ReLU function respectively, and then residual superimposed with the temporal feature vectors. Finally, the anomaly probability distribution is generated by mapping through the fully connected layer and the Softmax function.

[0030] The confidence correction unit calculates the cosine similarity between each pair of the first, second, and third feature vectors and uses the average as a consistency score. It then moves the second sliding window sequentially through the time-series feature vectors and calculates the window feature variance within the second sliding window. Finally, it averages the window feature variances calculated in each sliding step to obtain the time-series average variance. And it is mapped to a stability score through Gaussian function normalization. Based on the allocation coefficient, the anomaly probability distribution is weighted and corrected by combining the consistency score and stability score, generating an anomaly correction probability distribution. Anomaly classes with anomaly correction probabilities greater than or equal to the probability threshold are retained, generating an anomaly class set and delivering it to the analysis module.

[0031] Furthermore, the analysis module includes a map generation unit and a root cause determination unit;

[0032] The graph generation unit calls the dataset snapshot and extracts the traffic logs from it. It summarizes the data interaction paths and data flow paths in the traffic logs through semantic recognition. The devices and applications that deploy EBPF probes are regarded as nodes. The nodes are connected according to the interaction and flow relationships to generate a node dependency graph. The interaction and flow relationships correspond to bidirectional and unidirectional edges between nodes, respectively.

[0033] The root cause determination unit is based on the first anomaly class set. The main body of each anomaly class determines the corresponding anomaly node, and the first anomaly node is obtained based on the node dependency graph. Types of exceptions The upstream node was selected as the PC algorithm for causal inference model analysis. Types of exceptions The causal relationship of the upstream node is calculated in real time using a dynamic Bayesian network. Types of exceptions The root cause probability of the upstream node is called. Extract the first relevant historical operation and maintenance log from the standard dataset corresponding to the anomaly class set. Types of exceptions Standard data for each upstream node is extracted from all relevant historical operation and maintenance logs. Based on the historical standard data of the upstream node, calculate the... The first of the exception classes The standard data of the first upstream node and the first The average cosine similarity of all historical standard data of the upstream nodes, based on the calibration weighting ratio and the... The first of the exception classes We take the weighted sum of the root cause probabilities of the nth upstream node to obtain the nth... The first of the exception classes The calibration root cause probability of the upstream node is selected. Types of exceptions Among the upstream nodes, the upstream node with the highest probability of calibrating the root cause is selected as the first upstream node. The root cause nodes of each anomaly class are combined into the first... Types of exception classes - node groups, integrated to construct type - node group sets, where, , This represents the total number of existing exception classes. , For the first The total number of upstream nodes for each anomaly type.

[0034] Furthermore, the PC algorithm includes the following steps:

[0035] Preserving the node dependency graph For each anomaly class, connect all upstream nodes and related edges, and set all related edges to undirected edges to construct the first... A preliminary undirected graph of various anomaly classes. ;

[0036] Ignore the first In the anomaly class, the upstream nodes with a degree of 0 and 1 are considered, and the degree of the upstream node is equal to the number of related edges of the upstream node.

[0037] Starting from a node with a degree of 1, increment by 1, and iterate through the nodes to select the 1st degree. Among the exception types, upstream nodes that satisfy the degree of the current node are considered as the target upstream node;

[0038] Extract all adjacent upstream nodes that have an undirected edge with the target upstream node and pair them up to form a set of adjacent nodes of the target upstream node;

[0039] Ignore adjacent node groups of the upstream node of the target node that satisfy the condition that there are no edges between nodes within the adjacent node group. Then, sequentially select adjacent node groups where nodes within the group have edges, and then... The upstream node set of the anomaly class is used to remove the target upstream node and the selected adjacent node group and generate all possible condition node sets;

[0040] For each condition node set in the selected adjacent node group, check the conditional independence. If and only if there exists a condition node set that satisfies the conditional independence of the selected adjacent node group, remove the edges between nodes within the selected adjacent node group.

[0041] Until the traversal of the first Stop when the upstream node set of the anomaly class is reached, and generate the first one. For each anomaly class, a one-stage undirected graph is generated and each undirected edge is selected and redirected sequentially. The corresponding edge in the node dependency graph is queried. If the corresponding edge is a unidirectional edge, the selected undirected edge is kept consistent with the corresponding edge.

[0042] If the corresponding edge is a bidirectional edge, determine the first... Does the upstream node set of the anomaly class contain a third-party node that has edges with both the left and right ends of the selected undirected edge?

[0043] If no such node exists, the selected undirected edge is kept consistent with the corresponding edge. If a third-party node exists, the left and right nodes are adaptively decided to keep consistent with the corresponding edge based on the conditional independence of the given third-party node, or a unidirectional edge is established between the left, third-party, and right nodes. The direction of the unidirectional edge is determined based on the conditional independence.

[0044] Until the redirection The process stops when all undirected edges in the one-stage undirected graph of the anomaly class are found, generating the first... Cause-effect graphs for various anomaly classes.

[0045] Compared with existing technologies, this invention deploys EBPF probes on various devices and applications to capture heterogeneous datasets. It records the field structures of these datasets through a dynamic metadata registry, filters redundant fields using a hash index matching algorithm, and resolves structural inconsistencies in heterogeneous data through unified metadata management. Efficient redundancy identification reduces data redundancy, lowers storage and processing resource consumption, and improves data processing efficiency and accuracy. In the root cause analysis stage, it constructs a node dependency graph based on dataset snapshots to filter upstream node sets, mines causal relationships between nodes using a causal inference model, calculates root cause probabilities using a dynamic Bayesian network, and performs similarity matching based on historical operation and maintenance logs to calibrate root cause probabilities. This approach overcomes the limitations of simple association rules or manual experience, clarifies causal relationships between nodes, reduces misjudgments in root cause localization, and improves the accuracy of root cause localization. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of a full-stack intelligent operation and maintenance and root cause analysis system for data centers based on AIOps;

[0047] Figure 2 This is a flowchart of the abnormal cleaning process;

[0048] Figure 3 Flowchart for processing time-series modeling units;

[0049] Figure 4 Flowchart for deep classification unit processing. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0051] like Figure 1 As shown, a specific embodiment of the present invention discloses a data center full-stack intelligent operation and maintenance and root cause analysis system based on AIOps, including a data acquisition module, a processing module, a storage module, a detection module, an analysis module and an execution module;

[0052] The acquisition module is based on an EBPF probe (Extended Berkeley Packet Filter) at a sampling frequency. Heterogeneous datasets are captured at different times and dataset snapshots are generated. The field structure of the heterogeneous datasets is recorded through a dynamic metadata registry. The redundant field library is retrieved based on a hash index matching algorithm. Redundant fields of each type of data in the heterogeneous dataset are filtered to generate a deduplicated dataset. The deduplicated dataset and dataset snapshots are delivered to the processing module and storage module, respectively. The EBPF probe is deployed on each device and application.

[0053] The processing module identifies the structured data in the deduplicated dataset and transforms it into equivalent numerical data. Based on the fusion engine, it performs anomaly cleaning and missing data imputation on the numerical dataset, generates a standard dataset, and delivers it to the storage module and detection module simultaneously. The numerical dataset includes the original numerical data in the deduplicated dataset and the transformed equivalent numerical data. Anomaly cleaning is performed based on the dual anomaly localization based on the random forest algorithm and entropy statistics. Missing data imputation is performed based on the nearest neighbor algorithm within the first sliding window.

[0054] The storage module is based on relational database (PostgreSQL), distributed object storage (MinIO), resident cache database (Redis), and time series database (InfluxDB). It efficiently stores dataset snapshots, standard datasets, and operation and maintenance logs through global data heat management and performs elastic management.

[0055] The detection module receives the standard dataset for the current moment and retrieves the historical standard dataset from the storage module within the first time period before the current moment. It combines and constructs a detection sequence, performs temporal modeling on the detection sequence through a bidirectional long short-term memory network, extracts temporal feature vectors and inputs them into a deep classification network, performs feature fusion through convolution and residual connections, maps and outputs anomaly probability distributions through fully connected layers, performs probability correction based on confidence assessment, retains anomaly classes whose anomaly correction probabilities are greater than or equal to the probability threshold, generates anomaly class sets and delivers them to the analysis module.

[0056] The analysis module receives the anomaly class set and calls the dataset snapshot in the storage module to construct a node dependency graph. It filters the corresponding upstream node set for each anomaly class in the anomaly class set, analyzes the causal relationship between nodes through a causal inference model, calculates the root cause probability of each upstream node in the upstream node set of each anomaly class using a dynamic Bayesian network, calls the relevant historical operation and maintenance logs of each anomaly class in the storage module for similarity matching to linearly calibrate the root cause probability of each upstream node of each anomaly class, determines the root cause node of each anomaly class, and constructs a type-node group set.

[0057] The execution module uses each type-node group in the type-node group set as a two-dimensional index, searches the preset rule engine repair strategy library, retrieves the repair strategy corresponding to each type-node group, generates automatic execution instructions that support API calls, and sends them to the device or application corresponding to the root cause node in each type-node group through the API interface. The module performs the repair operation and packages the type-node group, standard dataset, execution time, automatic execution instructions, and execution results to generate operation and maintenance logs and transmits them to the storage module.

[0058] Furthermore, the acquisition module includes a capture unit and a preprocessing unit;

[0059] The crawling unit uses an EBPF probe compiled based on CO-RE. Since each data point in the heterogeneous dataset has a different kernel version and therefore different field structures, CO-RE compilation abstracts and maps each field of the heterogeneous dataset to an intermediate representation independent of the kernel version and records the mapping relationship. The crawling program for the EBPF probe is then compiled based on this intermediate representation, and the sampling frequency is set. When the EBPF probe captures data from different devices or applications, CO-RE compiles and extracts BTF information from different applications. BTF information is metadata that records the field structure of data in the kernel version. Through symbol resolution and dynamic binding mechanism, data is directly captured from different kernel versions to build a heterogeneous dataset without intruding on the device or application. The dataset snapshot is synchronously copied and delivered to the storage module to ensure the comprehensiveness of data capture and avoid interfering with the normal operation of the device or application. The heterogeneous dataset includes, but is not limited to, traffic logs, sensor data and server logs. Sensor data includes, but is not limited to, memory usage, disk read / write latency, CPU temperature, packet loss rate and network latency.

[0060] The preprocessing unit collects the field structure of each type of data in the heterogeneous dataset through a dynamic metadata registry. It then uses a structure comparison algorithm to compare the fields of the heterogeneous dataset captured by the EBPF probe with the historical heterogeneous dataset stored in the dynamic metadata registry. Field change events are identified and pushed to the dynamic metadata registry to achieve field structure synchronization. The dataset snapshot is compared again using the structure comparison algorithm to verify the synchronization accuracy. Field change events include field additions, field deletions, and field attribute changes. A hash index matching algorithm is used to quickly verify the field structure of the heterogeneous dataset. Each field in the heterogeneous dataset generates a hash value to be verified using a hash algorithm, which is then matched with the hash index in the redundant field library. When the hash value to be verified successfully matches the hash index in the redundant field library, the field corresponding to the hash value is considered a redundant field and filtered out, generating a deduplicated dataset, which is then delivered to the processing module. The deduplicated dataset includes deduplicated traffic logs, deduplicated sensor data, and deduplicated server logs.

[0061] Furthermore, the processing module receives the deduplicated dataset and performs data format recognition, distinguishing between deduplicated sensor data belonging to numerical data and deduplicated traffic logs and deduplicated server logs belonging to structured data. It converts text fields in the structured data into word vectors through word embedding, parses date fields in the structured data into timestamps, and maps discrete fields in the structured data to integers through label encoding, thus transforming the structured data into equivalent numerical data. The fusion engine is then activated, using a random forest algorithm to locate anomalies in the numerical dataset and performing verification and removal based on entropy statistics to achieve anomaly cleaning. Missing data in the numerical dataset is counted, and missing data is filled using a nearest neighbor algorithm within a first sliding window, generating a noise-free and anomaly-free standard dataset. The standard dataset includes standard traffic equivalent data, standard sensor data, and standard server equivalent data. Specifically, the standard traffic equivalent data and standard server equivalent data are generated from deduplicated traffic logs and deduplicated server logs after transformation, anomaly cleaning, and missing data filling, respectively.

[0062] like Figure 2As shown, further, the abnormal cleanup performed by the processing module includes the following steps:

[0063] Input the numerical dataset into the random forest. The root node of the tree;

[0064] Confirm the Given the selected field and split value of the current node in a tree, extract the field value of the selected field from the numerical dataset at the current node, and compare the split value of the current node with the field value of the selected field. , The total number of trees; the field is a set of fields that constitute each type of data in the numeric data, that is, all fields included in the dynamic metadata registry.

[0065] If the split value of the current node is greater than or equal to the value of the selected field, the numeric dataset is assigned to the left child node of the current node. If the split value of the current node is less than the value of the selected field, the numeric dataset is assigned to the right child node of the current node. Each time this assignment is performed, the numeric dataset is assigned to the right child node of the current node. Path length of trees Add 1;

[0066] Recursively perform the partitioning of the numerical dataset until the numerical dataset is partitioned to the nth... Stop at the leaf node of the tree, and obtain the numerical dataset at the [number]th [node]. Path length of trees ;

[0067] Based on numerical datasets Average path length of trees Calculate anomaly scores The specific formula is as follows:

[0068] ,

[0069] in, This represents an exponential function with base 2. To quantify the correction coefficient and avoid affecting the total number of trees The difference in settings causes the average path length Statistical bias, ensuring outlier scores. The computational stability;

[0070] abnormal scores Numerical datasets that are greater than or equal to the anomaly score threshold are marked as potential anomaly datasets. Historical standard datasets from different times within the first time period before the current time are retrieved from the storage module as historical sequences. The historical field entropy value of each field in the historical sequence is calculated. Potential anomaly datasets are then assigned to the historical sequence and the field entropy value of each field is recalculated.

[0071] Fields whose field entropy value changes at a rate greater than or equal to the entropy value change threshold compared to their corresponding historical field entropy values ​​are marked as anomalous fields. The data to which the anomalous fields belong are identified and removed from the potential anomalous dataset, generating a standard data subset. In other words, based on the anomalous fields, it is determined which data in the numerical dataset is identified as a potential anomalous dataset and removed. The remaining data in the potential anomalous dataset is considered standard data, and the set of the remaining data must be included in the standard dataset generated by the missing data imputation process, i.e., the standard data subset.

[0072] In all other cases, no processing is performed on the numerical dataset. The numerical dataset is directly used as a subset of the standard dataset. In this case, the standard dataset has no missing data and is equal to the standard dataset after missing data filling.

[0073] Furthermore, building a random forest requires calling a number of methods greater than the total number of trees. Using more than twice the number of historical standard datasets from different times as training sequences, a greater number of samples with replacement were randomly drawn from the training sequences based on bootstrap sampling than the total number of trees. Historical standard datasets with a number less than the total number of sequences, repeated bootstrap sampling. This time received The nth different training subsequence, utilizing the nth... The training subsequence is used to construct the training subsequence. A tree, , , , To determine the total number of training subsequences, the th... The training subsequence is used as the th training subsequence The root node of the tree is randomly selected from a field in the historical standard dataset. A splitting value is randomly chosen from the range of values ​​for this field. The root node is then split based on this splitting value. For the resulting child nodes, fields and splitting values ​​are randomly selected again to continue splitting, until the [number]th [tree] is reached. The historical standard datasets in all unsplit child nodes of the tree are exactly the same, the first... After training the tree, define the unsplit child node as the first... The leaf nodes of a tree, when When the training of the trees is complete, the random forest is constructed. The random forest determines whether the standard dataset at each time step is a potentially anomalous dataset based on the field values ​​of each field in the standard dataset at each time step.

[0074] Specifically, missing data filling sequentially calls the historical standard datasets that are closest in time sequence to the current moment from the storage module until the total number of moments, including the current moment, equals the size of the first window of the first sliding window. It then identifies the missing data in the standard data subset of the current moment. For each type of missing data, it calculates the average similarity between each data in the standard data subset of the current moment and the corresponding data in each historical standard dataset within the first sliding window. The historical standard dataset with the highest average similarity is taken as the nearest neighbor of the standard data subset of the current moment. Data corresponding to the missing data is selected from the nearest neighbors and added to the standard data subset of the current moment to generate the standard dataset of the current moment.

[0075] Furthermore, the storage module stores the standard dataset in a relational database and adopts global data heat management. Data within the first time period before the current moment is considered hot data, data outside the first time period but within the second time period before the current moment is considered warm data, and data outside the second time period before the current moment is considered cold data. The first time period is shorter than the second time period. As time progresses, the data is dynamically moved. The standard dataset of the latest moment is moved from the relational database to the resident cache database as hot data so that other modules can read and call it at high speed. The historical standard dataset of the earliest moment in the resident cache database is moved into the memory area of ​​the time series database as warm data. The historical standard dataset of the earliest moment in the memory area of ​​the time series database is moved into the disk area of ​​the time series database as cold data.

[0076] A two-dimensional partitioning strategy based on timestamps and Internet protocols is adopted to allocate a dedicated root directory for the dataset snapshot at the current moment in the distributed object storage and write it. The collected data snapshot is used as a backup and is cold data by default.

[0077] Operation and maintenance logs are stored in a relational database. As time goes by, when the operation and maintenance logs are determined to be cold data by global data heat management, a hash function is used to encrypt and calculate the operation hash, and then the logs are moved to a designated directory in a distributed object storage. The hash operation is also synchronized to the blockchain evidence storage interface, and the immutability of the blockchain is used to ensure the traceability of the operation and maintenance logs.

[0078] Furthermore, the detection module includes a temporal modeling unit, a deep classification unit, and a confidence correction unit;

[0079] like Figure 3As shown, the temporal modeling unit receives the standard dataset at the current moment and retrieves the historical standard dataset within the first time period before the current moment from the storage module. It eliminates the dimensional differences of different data in the standard dataset through min-max normalization and combines them to construct the detection sequence. It performs temporal modeling on the detection sequence through a bidirectional long short-term memory network to capture trends and periodicity. The forward long short-term memory network and the reverse long short-term memory network each have two layers. The number of neurons in the second layer is set to half the number of neurons in the first layer. The forward long short-term memory network performs forward modeling from the earliest moment in the detection sequence, and the reverse long short-term memory network performs reverse modeling from the current moment in the detection sequence. The temporal flipping result of the reverse modeling features is concatenated with the forward modeling features in the channel dimension, and the temporal feature vector is generated by global average pooling compression.

[0080] like Figure 4 As shown, the deep classification unit uses three stacked one-dimensional convolutional layers to process the temporal feature vector. The first layer uses 16 convolutional kernels of size 1 to extract adjacent features from the temporal feature vector. The second layer uses 16 convolutional kernels of size 4 to extract short-period features from the temporal feature vector. The third layer uses 16 convolutional kernels of size 7 to extract long-period features from the temporal feature vector. The adjacent features, short-period features, and long-period features are all adjusted to the same dimension as the temporal feature vector through a fully connected layer and the nonlinear representation is enhanced based on the ReLU function to generate the first feature vector, the second feature vector, and the third feature vector, respectively. The temporal feature vector, the first feature vector, the second feature vector, and the third feature vector are superimposed based on the residual structure and the dimension is adjusted again through a fully connected layer. After being mapped by the Softmax function, an anomaly probability distribution with a dimension equal to the total number of anomaly categories is generated. The anomaly probability distribution records the anomaly probability of each anomaly class. The anomaly classes include the subject and the anomaly problem. The subject includes each device and application that deploys the EBPF probe.

[0081] The confidence correction unit calculates the cosine similarity between each pair of the first, second, and third feature vectors and takes the mean as the consistency score. A second sliding window slides from left to right across the time-series feature vectors. The size of the second window is set to 5, and it slides one dimension at a time. The variance of the window features within the second sliding window is calculated sequentially until the second sliding window selects the last 5 dimensions of the time-series feature vectors. The mean of the variance of the window features calculated in each slide is then calculated to obtain the time-series average variance. A stability score is generated by normalizing the mapping using a Gaussian function. Based on preset allocation coefficients, the anomaly probability of each anomaly class in the anomaly probability distribution is weighted and corrected with the consistency score and stability score to generate an anomaly correction probability distribution. The allocation coefficients for the anomaly probability distribution, consistency score, and stability score are 0.6, 0.2, and 0.2, respectively. That is, the anomaly correction probability distribution equals 0.6 multiplied by the anomaly probability distribution plus 0.2 multiplied by the consistency score plus 0.2 multiplied by the stability score. Anomaly classes with anomaly correction probabilities greater than or equal to a probability threshold are retained, and an anomaly class set is generated and delivered to the analysis module. The specific mapping formula for the Gaussian function is as follows:

[0082] ,

[0083] in, This is the decay coefficient, used to adjust the decay rate of the Gaussian function. The standard deviation threshold can be set according to different scenarios. It is based on the mapping of a Gaussian function and the time series mean variance. As the stability score approaches 0, it approaches the highest stability score of 1. This represents an exponential function.

[0084] Furthermore, the analysis module includes a map generation unit and a root cause determination unit;

[0085] The graph generation unit calls the dataset snapshot in the storage module, extracts traffic logs, and identifies data exchange records, process call information, and resource access records between devices, between devices and applications, and between applications through the existing BERT model. It summarizes the data interaction path and data flow path, treats each device and application that deploys the EBPF probe as a node, and connects the nodes according to the interaction and flow relationship to generate a node dependency graph corresponding to the dataset snapshot. If there is data interaction between node A and node B, a bidirectional edge is established between node A and node B. If the outflow data of node A is the inflow data of node B, a unidirectional edge is established from node A to node B.

[0086] The root cause determination unit is based on the first anomaly class set. The subject of the anomaly class is determined. For each type of exception node, the first exception node is obtained based on the node dependency graph. Types of exceptions The upstream nodes constitute the first For each anomaly class's upstream node set, the PC algorithm is selected as the causal inference model to deeply mine the first... Types of exceptions The causal relationships between upstream nodes are calculated in real time using a dynamic Bayesian network. Types of exceptions The root cause probability of the upstream node, based on the . The first exception class retrieves all relevant historical operation and maintenance logs from the storage module, and extracts the first exception from the standard dataset corresponding to the exception class set. Types of exceptions Standard data for each upstream node is extracted from all relevant historical operation and maintenance logs. Based on the historical standard data of the upstream node, calculate the... The first of the exception classes The standard data of the first upstream node and the first The average cosine similarity of all historical standard data of the upstream nodes, and based on the preset calibration weight ratio, with the first upstream node. The first of the exception classes We take the weighted sum of the root cause probabilities of the nth upstream node to obtain the nth... The first of the exception classes The calibration root cause probability of the upstream node is selected. Types of exceptions Among the upstream nodes, the upstream node with the highest probability of calibrating the root cause is selected as the first upstream node. The root cause node of the anomaly class, combined with the first The construction of the first type of anomaly and root cause node Type-node group corresponding to each exception class, retrieve the exception class set. For each exception class, a type-node group is constructed, and a set of type-node groups corresponding to the exception class set is built. , This represents the total number of existing exception classes. , For the first The total number of upstream nodes for each anomaly class is given. Dynamic Bayesian networks are existing technology and will not be discussed further.

[0087] Furthermore, the PC algorithm is used to prune the node dependency graph to clarify the causal relationships between upstream nodes for each anomaly class, including the following steps:

[0088] Confirm the The upstream node set of the anomaly class is used to remove nodes from the node dependency graph that do not belong to the first class. The upstream node set and related edges of the anomaly class are selected, retaining only the first one. Generate the first type of anomaly by connecting all upstream nodes and related edges. The unprocessed subgraph of anomaly class, where the related edges are related to the first... One-way and two-way edges connecting upstream nodes of anomaly class. ;

[0089] The first In the case of an anomaly, all edges in the subgraph to be processed are converted to undirected edges, and the first type of anomaly is constructed. The initial undirected graph of the anomaly class is constructed and the node degree is initialized to 0, where the node degree of the upstream node is equal to the number of edges associated with the upstream node;

[0090] For the For upstream nodes in the anomaly class with a degree of 0 or 1, direct determination is made that no pruning is required;

[0091] Starting with a node degree equal to 1, increment the node degree by 1 with each increment, and iterate through the nodes to select the node with the first degree. For each exception type, the upstream node that satisfies the current node degree is considered the target upstream node;

[0092] Extract all adjacent upstream nodes that have an undirected edge with the target upstream node and pair them up to form the adjacent node set of the target upstream node. For example, if the target upstream node C has adjacent upstream nodes D, E, and F, then the adjacent node set of the target upstream node C is: ,in, , and All are adjacent node groups of the upstream node C of the target;

[0093] Select each adjacent node group of the target upstream node in turn, and determine whether there is an undirected edge between two adjacent upstream nodes in the selected adjacent node group. If there is no undirected edge, skip it directly; if there is, proceed to the next step. The target upstream node and the selected adjacent node group are removed from the upstream node set of the anomaly class, and all possible subsets are generated as condition node sets for the selected adjacent node group respectively.

[0094] The Fisher exact test or chi-square test is used to test the conditional independence of each conditional node set in the selected adjacent node group. If any conditional node set in all conditional node sets of the selected adjacent node group satisfies the conditional independence, the undirected edge between two adjacent upstream nodes in the selected adjacent node group is removed. Otherwise, the undirected edge between two adjacent upstream nodes in the selected adjacent node group is retained. The Fisher exact test and chi-square test are existing methods and will not be elaborated on in detail.

[0095] Until the traversal of the first Stop when the upstream node set of the anomaly class is reached, and complete the first step. Pruning of the initial undirected graph of the anomaly class to generate the first... A one-stage undirected graph of anomaly classes;

[0096] Select the first one in sequence For each undirected edge in a one-stage undirected graph of an anomaly class, redirect it, query the corresponding edge in the node dependency graph, and if the corresponding edge is a unidirectional edge, keep the selected undirected edge consistent with the corresponding edge.

[0097] If the corresponding edge is a bidirectional edge, determine the first... Does the upstream node set of the anomaly class contain a third-party node that has edges with both the left and right ends of the selected undirected edge?

[0098] If it does not exist, keep the selected undirected edge consistent with the corresponding edge. If there is a third-party node, determine whether the left node is independent of the right node when a third-party node is given, and whether the right node is not independent of the left node when a third-party node is given.

[0099] If satisfied, the bidirectional edge between the left and right nodes is redirected to a unidirectional edge from the right node to the third-party node and a unidirectional edge from the third-party node to the left node.

[0100] If not satisfied, then it is determined again whether the right node is independent of the left node when given a third node and whether the left node is not independent of the right node when given a third node, in order to further determine whether to redirect the bidirectional edge between the left node and the right node to a unidirectional edge from the left node to the third node and a unidirectional edge from the third node to the right node.

[0101] If none of these conditions are met, the selected undirected edge will be kept consistent with its corresponding edge until the redirection is performed. The process stops when all undirected edges in the one-stage undirected graph of the anomaly class are found, generating a result reflecting the first... Types of exceptions A causal graph representing the causal relationships between upstream nodes clearly defines the causal direction and provides a computational foundation for dynamic Bayesian networks.

[0102] This invention discloses a full-stack intelligent operation and maintenance and root cause analysis system for data centers based on AIOps, including a data acquisition module, a processing module, a storage module, a detection module, an analysis module, and an execution module. The acquisition module uses EBPF probes to periodically capture heterogeneous datasets and generate dataset snapshots. It records field structures through a dynamic metadata registry, uses a hash index matching algorithm to retrieve redundant field libraries, filters redundant fields in heterogeneous datasets, and obtains deduplicated datasets, effectively reducing data volume and avoiding redundant information interfering with subsequent processing. The processing module converts the deduplicated datasets into numerical datasets, uses a fusion engine, employs a random forest algorithm and entropy statistics for anomaly cleaning, and uses a nearest neighbor algorithm within a first sliding window for missing data filling, generating a standard dataset. This achieves data format unification, eliminates the impact of outliers and missing values, and provides reliable data support for subsequent detection and analysis stages. The storage module uses global data heat management to efficiently store dataset snapshots, standard datasets, and operation and maintenance logs and implements elastic management. It rationally allocates and schedules data based on heat, improving storage resource utilization and ensuring fast data access and effective management. The detection module receives the standard dataset at the current moment, calls historical standard datasets from the first time interval prior to this moment, and combines them to construct a detection sequence. Temporal modeling is performed using a bidirectional long short-term memory network to extract temporal feature vectors. A deep classification network is then used to map the output anomaly probability distribution. Probability correction is performed based on confidence assessment, retaining anomaly classes whose corrected probabilities are greater than or equal to a probability threshold, forming an anomaly set. This accurately captures data anomalies. Through temporal modeling and probability correction, the accuracy and reliability of anomaly detection are improved, providing strong support for timely discovery of potential problems. The analysis module receives the anomaly set, constructs a node dependency graph based on dataset snapshots, determines the upstream node set for each anomaly class, mines causal relationships between nodes using a causal inference model, calculates the root cause probability of each upstream node for each anomaly class using a dynamic Bayesian network, and performs similarity matching by calling relevant historical operation and maintenance logs for each anomaly class. The system performs linear calibration on the root cause probability to identify the root cause node for each anomaly type, constructs type-node sets, and delves into the root causes of anomalies. Through multi-step analysis and calibration, it improves the accuracy of root cause localization and provides clear targets for subsequent remediation operations. The execution module retrieves the rule engine's remediation strategy library based on the type-node sets, generates automatic execution instructions corresponding to each type-node set, sends them to the corresponding root cause nodes to perform remediation operations, and packages them to generate operation and maintenance logs. This enables rapid anomaly response and provides a complete record for subsequent operation and maintenance system optimization and problem tracing. All modules work together to achieve full-stack intelligent operation and maintenance and root cause analysis of the data center, significantly improving data center operation and maintenance efficiency and problem-solving capabilities, and ensuring the stable and efficient operation of the data center.

[0103] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A data center full-stack intelligent operation and maintenance and root cause analysis system based on AIOps, characterized in that, It includes a data acquisition module, a processing module, a detection module, and an analysis module; The acquisition module captures heterogeneous datasets at equal intervals based on the EBPF probe and generates dataset snapshots. It collects field structures through a dynamic metadata registry, retrieves redundant field libraries based on a hash index matching algorithm, filters redundant fields in heterogeneous datasets, and generates deduplicated datasets. The processing module converts the deduplicated dataset into a numerical dataset and performs anomaly cleaning and missing data imputation to generate a standard dataset. The detection module receives the standard dataset for the current moment, calls the historical standard dataset within the first time period before the current moment to construct the detection sequence, and generates an anomaly class set through time series modeling and classification processing. The analysis module receives anomaly sets, constructs a node dependency graph based on dataset snapshots, identifies the upstream node set for each anomaly class, analyzes the causal relationships between nodes through a causal inference model, calculates the root cause probability of each upstream node using a dynamic Bayesian network, calls relevant historical operation and maintenance logs for similarity matching to linearly calibrate the root cause probability, and determines the root cause node for each anomaly class. The acquisition module includes a crawling unit and a preprocessing unit. The crawling unit uses an EBPF probe compiled based on CO-RE to map each field of the heterogeneous dataset to an intermediate representation and record the mapping relationship. It compiles a crawling program based on the intermediate representation and extracts BTF information through CO-RE compilation during crawling. It constructs the heterogeneous dataset and generates a dataset snapshot by using symbol resolution and dynamic binding mechanisms. The preprocessing unit records the field structure in the heterogeneous dataset through a dynamic metadata registry, performs field comparison and synchronization, and filters redundant fields based on a hash index matching algorithm to generate a deduplicated dataset. The processing module converts the structured data in the deduplicated dataset into equivalent numerical data, and together with the original numerical data, constructs a numerical dataset. Anomaly cleaning is based on the random forest algorithm and entropy statistics, and missing data filling is based on the nearest neighbor algorithm within the first sliding window. The anomaly cleaning includes: inputting the numerical dataset into the tree of the random forest, obtaining the path length through node splitting, calculating anomaly scores to mark potential anomaly datasets, calling historical standard datasets to calculate the field entropy change rate, marking anomaly fields and cleaning them to generate a standard data subset.

2. The data center full-stack intelligent operation and maintenance and root cause analysis system based on AIOps as described in claim 1, characterized in that, The causal inference model of the analysis module adopts the PC algorithm, which includes: constructing a preliminary undirected graph based on the node dependency graph, pruning through node degree traversal and conditional independence test, and redirecting undirected edges to generate a causal graph.

3. The AIOps-based full-stack intelligent operation and maintenance and root cause analysis system for data centers as described in claim 1, characterized in that, It also includes a storage module; The storage module manages the storage dataset snapshots, standard datasets, and operation and maintenance logs through global data heat management, dividing the data into hot data, warm data, and cold data and managing them elastically.

4. The AIOps-based full-stack intelligent operation and maintenance and root cause analysis system for data centers as described in claim 3, characterized in that, The storage module uses a two-dimensional partitioning strategy to store dataset snapshots; operation logs identified as cold data are stored in distributed object storage after calculating operation hashes using a hash function, and the operation hashes are synchronized to the blockchain evidence storage interface.

5. The data center full-stack intelligent operation and maintenance and root cause analysis system based on AIOps as described in claim 1, characterized in that, The temporal modeling and classification processing of the detection module includes: performing temporal modeling on the detection sequence through a bidirectional long short-term memory network to extract temporal feature vectors; processing the temporal feature vectors using a deep classification network, mapping and outputting anomaly probability distributions, performing probability correction based on confidence assessment, and retaining anomaly classes whose anomaly correction probabilities are greater than or equal to the probability threshold to generate anomaly class sets.

6. The AIOps-based full-stack intelligent operation and maintenance and root cause analysis system for data centers as described in claim 5, characterized in that, The deep classification network uses three layers of one-dimensional convolutions of different sizes to extract adjacent features, short-period features and long-period features respectively. It generates corresponding feature vectors through fully connected layers and ReLU function, and maps them with the temporal feature vectors after residual superposition to generate anomaly probability distribution.

7. The data center full-stack intelligent operation and maintenance and root cause analysis system based on AIOps as described in claim 1, characterized in that, It also includes an execution module; The execution module generates automatic execution instructions based on the root cause node retrieval rule engine repair strategy library determined by the analysis module, sends them to the corresponding root cause nodes, and packages them to generate operation and maintenance logs.

Images