A self-healing operation and maintenance method, device, equipment and storage medium for big data components

Abstract

The present application relates to a self-healing operation and maintenance method, apparatus, equipment and storage medium for big data components. By preprocessing and fusing features of multimodal data of big data components, a high-dimensional component health state vector is obtained and input into a large fault diagnosis model, and the target fault type and root cause confidence are output; when the root cause confidence meets a preset confidence threshold, a historical repair strategy is determined based on the target fault type, and the cluster environment similarity is calculated based on the cluster environment characteristic parameters of the current cluster environment and the historical repair strategy. If the cluster environment similarity meets the preset cluster environment similarity threshold, the historical repair strategy is used as a candidate repair strategy; if not, a candidate repair strategy is generated based on a reinforcement learning strategy generation engine, and the candidate repair strategy is executed based on the strategy level matched by the candidate repair strategy; compared with the existing technology, the technical solution of the present application can effectively solve the problems of missed detection, inefficiency, and solidification in traditional operation and maintenance.

Description

Technical Field

[0001] The present application relates to the technical field of big data component operation and maintenance, and in particular to a self-healing operation and maintenance method, apparatus, device and storage medium for big data components. Background Art

[0002] As the scale of distributed big data clusters expands, the complexity of operation and maintenance increases significantly.

[0003] In existing technologies, mainstream operation and maintenance solutions are mostly based on the "monitoring alarm + preset script repair" model, supplemented by simple machine learning models to assist in diagnosis. For example, abnormal monitoring is achieved by manually setting thresholds for indicators such as CPU usage and memory usage. When the indicators exceed the standard, an alarm is triggered. The repair link calls fixed scripts, such as restarting services, expanding nodes, and handling known faults.

[0004] However, mainstream operation and maintenance solutions also have many flaws. First, monitoring alarms rely on manually set fixed thresholds and can only detect obvious anomalies. They lack the ability to perceive hidden faults such as slow memory leaks and configuration parameter drift. Second, repair scripts cannot cope with complex fault scenarios. For example, HDFS block loss may be caused by various reasons such as disk failure and network partitioning. Traditional scripts cannot dynamically match root cause repair strategies, and the average repair time exceeds 30 minutes. Third, multimodal data such as time series, logs, and dependency graphs are not integrated, and repair logs are not effectively utilized, resulting in a lack of continuous optimization capabilities for fault diagnosis models. Summary of the Invention

[0005] This application provides a self-healing operation and maintenance method, device, equipment and storage medium for big data components. It improves fault perception capabilities through multimodal data fusion, ensures accuracy through large-scale model diagnosis, and takes into account both repair efficiency and adaptability to complex scenarios through a layered strategy mechanism, forming a closed-loop intelligent operation and maintenance, effectively solving the problems of missed detection, inefficiency, and solidification in traditional operation and maintenance.

[0006] In a first aspect, the present application provides a self-healing operation and maintenance method for a big data component, comprising: collecting multimodal data of the big data component, preprocessing the multimodal data to obtain preprocessed multimodal data, and performing feature fusion processing on the preprocessed multimodal data to obtain a high-dimensional component health state vector; inputting the high-dimensional component health state vector into a fault diagnosis big model so that the fault diagnosis big model outputs a target fault type and a root cause confidence; when the root cause confidence meets a preset confidence threshold, obtaining cluster environment characteristic parameters of the current cluster environment, and determining a historical repair strategy based on the target fault type, calculating the cluster environment similarity based on the cluster environment characteristic parameters and the historical repair strategy; if the cluster environment similarity meets the preset cluster environment similarity threshold, using the historical repair strategy as a candidate repair strategy; if the cluster environment similarity does not meet the preset cluster environment similarity threshold, generating a candidate repair strategy based on a preset reinforcement learning strategy generation engine; matching the candidate repair strategy with a corresponding strategy level, and executing the candidate repair strategy based on the strategy level.

[0007] In one possible implementation, the multimodal data is preprocessed to obtain preprocessed multimodal data, specifically including: the multimodal data includes time series performance indicator data, log data and component dependency graph; the time series performance indicator data is standardized to obtain standard time series performance indicator data, and statistical features are extracted from the time series performance indicator data to generate a time series performance indicator feature vector of a preset dimension; structured information is extracted from the log data to obtain structured log data, and semantic features are encoded on the structured log data to generate a log feature vector of a preset dimension; graph embedding is performed on the nodes in the component dependency graph to generate a graph feature vector of a preset dimension; and the time series performance indicator feature vector, the log feature vector and the graph feature vector are used as preprocessed multimodal data.

[0008] In one possible implementation, the preprocessed multimodal data is subjected to feature fusion processing to obtain a high-dimensional component health status vector, specifically including: mapping the time series performance indicator feature vector, the log feature vector, and the graph feature vector to the same dimension based on an independent fully connected neural network layer to obtain a first time series performance indicator feature vector, a first log feature vector, and a first graph feature vector of the same latitude; adding corresponding position encoding vectors and modality type identifiers to the first time series performance indicator feature vector, the first log feature vector, and the first graph feature vector to obtain a second time series performance indicator feature vector, a second log feature vector, and a second graph feature vector; concatenating the second time series performance indicator feature vector, the second log feature vector, and the second graph feature vector to obtain a multimodal feature vector sequence; inputting the multimodal feature vector sequence into a multi-layer multi-head attention mechanism encoder so that the multi-layer multi-head attention mechanism encoder calculates the cross-modal association weights in the multimodal feature vector sequence to generate a fusion feature vector; extracting a target fusion feature vector from the fusion feature vector, and mapping the target fusion feature vector to the target dimension based on a fully connected layer to obtain a high-dimensional component health status vector.

[0009] In one possible implementation, the high-dimensional component health state vector is input into a fault diagnosis big model so that the fault diagnosis big model outputs the target fault type and the root cause confidence, specifically including: the fault diagnosis big model includes a Transformer encoder, a fault classification head and a confidence calculation head; the high-dimensional component health state vector is input into the fault diagnosis big model so that the fault diagnosis big model normalizes the high-dimensional component health state vector to obtain a normalized high-dimensional component health state vector; hierarchical feature extraction is performed on the normalized high-dimensional component health state vector based on the Transformer encoder, Obtain deep features; input the deep features into the fault classification head so that the fault classification head outputs a fault type raw score vector corresponding to each fault type; perform Softmax normalization on the fault type raw score vector to obtain a fault type probability value corresponding to each fault type, and select the fault type corresponding to the maximum fault type probability as the target fault type; input the deep features into the confidence calculation head so that the confidence calculation head outputs a basic confidence, determine a risk weighting factor based on the abnormal dimension value in the high-dimensional component health status vector, and calculate the root cause confidence based on the risk weighting factor and the basic confidence.

[0010] In a possible implementation, the cluster environment similarity is calculated based on the cluster environment characteristic parameters and the historical repair strategy, specifically including: the cluster environment characteristic parameters include software version characteristics, core configuration parameter characteristics and cluster scale characteristics; the dimensional similarity of the software version characteristics, the core configuration parameter characteristics, the cluster scale characteristics and the historical repair strategy are calculated respectively to obtain software version characteristic similarity, core configuration parameter characteristic similarity and cluster scale characteristic similarity; based on a preset weight value, the software version characteristic similarity, the core configuration parameter characteristic similarity and the cluster scale characteristic similarity are weightedly fused to obtain cluster environment similarity.

[0011] In one possible implementation, a candidate repair strategy is generated based on a preset reinforcement learning strategy generation engine, specifically including: constructing a state space, wherein the state space includes a multi-dimensional real-time state vector; defining an action space, wherein the action space includes basic operations and operation combination rules; designing a reward function, wherein the reward function is calculated based on the strategy execution time, resource consumption and repair result; optimizing the initial strategy using a proximal strategy optimization algorithm to obtain an optimized strategy, wherein the optimization includes a strategy network update and an acceleration mechanism; verifying the optimized strategy in a simulation environment, and if the optimized strategy meets the preset success rate threshold and timeliness requirements, outputting the optimized strategy as a candidate repair strategy.

[0012] In one possible implementation, the candidate repair strategy is matched to the corresponding policy level, and the candidate repair strategy is executed based on the policy level, specifically including: based on the fault type, mapping the candidate repair strategy to the corresponding policy level, wherein the policy level includes at least a first-level policy, a second-level policy, and a third-level policy; executing the candidate repair strategy based on the policy level, and monitoring the policy execution time of the candidate repair strategy; if the policy execution time exceeds the preset timeout threshold corresponding to the policy level, triggering the corresponding timeout processing mechanism, wherein the timeout processing mechanism includes executing a backup strategy of the same level or upgrading the policy level.

[0013] In the second aspect, the present application provides a self-healing operation and maintenance device for a big data component, comprising: a multimodal data processing module, a fault diagnosis module, a candidate repair strategy generation module and a candidate repair strategy execution module; wherein the multimodal data processing module is used to collect multimodal data of the big data component, pre-process the multimodal data to obtain pre-processed multimodal data, and perform feature fusion processing on the pre-processed multimodal data to obtain a high-dimensional component health state vector; the fault diagnosis module is used to input the high-dimensional component health state vector into a fault diagnosis large model so that the fault diagnosis large model outputs the target fault type and root cause confidence; the candidate repair strategy generation module is used when When the root cause confidence meets the preset confidence threshold, the cluster environment characteristic parameters of the current cluster environment are obtained, and based on the target fault type, the historical repair strategy is determined. Based on the cluster environment characteristic parameters and the historical repair strategy, the cluster environment similarity is calculated. If the cluster environment similarity meets the preset cluster environment similarity threshold, the historical repair strategy is used as a candidate repair strategy. If the cluster environment similarity does not meet the preset cluster environment similarity threshold, a candidate repair strategy is generated based on a preset reinforcement learning strategy generation engine; the candidate repair strategy execution module is used to match the candidate repair strategy with a corresponding policy level, and execute the candidate repair strategy based on the policy level.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0016] The present invention provides a method, apparatus, device, and storage medium for self-healing operation and maintenance of a big data component, which has the following advantages over the prior art:

[0017] By collecting multimodal data of big data components; preprocessing the multimodal data to obtain preprocessed multimodal data, and performing feature fusion processing on the preprocessed multimodal data to obtain a high-dimensional component health state vector; inputting the high-dimensional component health state vector into the fault diagnosis big model, so that the fault diagnosis big model outputs the target fault type and root cause confidence; when the root cause confidence meets the preset confidence threshold, obtaining the cluster environment characteristic parameters of the current cluster environment, and based on the target fault type, determining the historical repair strategy, calculating the cluster environment similarity based on the cluster environment characteristic parameters and the historical repair strategy, if the cluster environment similarity meets the preset cluster environment similarity threshold, then using the historical repair strategy as a candidate repair strategy; if the cluster environment similarity does not meet the preset cluster environment similarity threshold, then generating a candidate based on the preset reinforcement learning strategy generation engine. Repair strategy, match the corresponding strategy level to the candidate repair strategy, and execute the candidate repair strategy based on the strategy level; compared with the existing technology, the technical solution of the present application breaks through the limitations of traditional single data through multimodal data fusion, and improves the perception ability of hidden faults; and based on the root cause confidence output by the fault diagnosis large model, provides an accurate basis for the selection of repair strategy to avoid blind operation; through the hierarchical mechanism generated by historical strategy reuse and reinforcement learning, mature strategies are reused to improve efficiency when the cluster environment is similar, and new strategies are dynamically generated to adapt to complex scenarios when the environment does not match, which not only ensures the rapid repair of known faults, but also enhances the ability to respond to new and complex faults. At the same time, through the in-depth use of multimodal data and dynamic optimization of strategies, it solves the problems of high missed detection rate, low repair efficiency, model solidification and poor cross-scenario adaptability in traditional operation and maintenance, and forms a closed-loop intelligent operation and maintenance system of perception-diagnosis-repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1This is a flowchart of an embodiment of a self-healing operation and maintenance method for a big data component provided by the present application;

[0022] Figure 2 This is a structural diagram of an embodiment of a self-healing operation and maintenance device for a big data component provided by the present application;

[0023] Figure 3 This is a structural diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0026] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] Example 1, see Figure 1 , Figure 1 This is a flow chart of an embodiment of a self-healing operation and maintenance method for a big data component provided by this application, such as Figure 1 As shown, the method includes steps 101 to 104, which are specifically as follows:

[0031] Step 101: Collect multimodal data of a big data component, preprocess the multimodal data to obtain preprocessed multimodal data, and perform feature fusion processing on the preprocessed multimodal data to obtain a high-dimensional component health status vector.

[0032] In one embodiment, the multimodal data includes time series performance indicator data, log data, and component dependency graphs.

[0033] Specifically, the time series performance data includes at least one of CPU usage, memory usage, disk IO latency, and component throughput; real-time acquisition is achieved through distributed collection agents and standardized protocols; for example, a lightweight agent is deployed on each node of a big data cluster as the core carrier for indicator collection, and runtime indicators of Java class components are captured through the JMX protocol, such as JVM heap memory usage, GC recycling frequency, and disk IO latency of Hadoop DataNode; and component-specific indicators exposed by the Exporter are pulled through the Prometheus protocol, such as Kafka Broker's message throughput, Spark Executor's memory usage, and YARN's container resource allocation. Indicators are collected according to a preset period, and the data is compressed and written to the time series database in real time, and stored in layers according to time granularity to obtain time series performance indicator data.

[0034] Specifically, for log data, tools such as Flink and Beats are deployed on each component node to collect core log files, including HDFS NameNode logs, Kafka Broker logs, and YARN ResourceManager logs.

[0035] Specifically, for the component dependency graph, the node roles and service endpoints are identified by parsing the configuration files of the big data components; the connection status between components is detected by simulating RPC calls or network connection tests, and a directed dependency graph containing node dependencies is generated, and the directed dependency graph is used as the component dependency graph.

[0036] Preferably, basic information about node roles and service endpoints is obtained through static configuration file parsing, such as the Agent deployed on each node automatically locates and parses the configuration files of the core components of the Hadoop ecosystem; by extracting key attributes from the configuration files, the core roles of each node are accurately identified, and the service endpoint information is parsed, thereby building a static node role mapping view to clarify the service identity of each node and the basic address for providing services to the outside world; secondly, the connection status and dependency direction between components are dynamically detected by simulating RPC calls; based on the service endpoints obtained through static parsing, the Agent or central coordination service periodically initiates lightweight simulation calls: for services based on specific protocols such as HDFS and YARN, harmless operations are performed using the client SDK; for services that provide HTTP interfaces, GET requests are sent to the health check endpoint. The connection status is determined by the response result. A successful response is "normal," while a timeout or error response is "abnormal." A directed dependency is established based on the call relationship. If component A needs to call component B's service and the connection is normal, a directed edge is established from A to B, with the edge weight initialized to the connection success rate or response delay. Finally, static roles and dynamic connection information are integrated to generate a directed dependency graph G = (V, E), where the vertex set V represents the cluster nodes and the edge set E represents the service dependencies between nodes. By using the nodes and corresponding roles as graph vertices and the dynamically detected valid dependencies as directed edges, a component dependency graph covering the entire cluster is formed. This graph clearly displays the upstream dependencies between services, with an edge count of over 100,000. Periodic updates ensure real-time reflection of the cluster's current topology and health status, providing structured dependency data support for fault propagation path analysis.

[0037] In one embodiment, the time series performance indicator data is standardized to obtain standard time series performance indicator data, and statistical features are extracted from the time series performance indicator data to generate a time series performance indicator feature vector of a preset dimension.

[0038] Specifically, the mean value and standard deviation of the timing performance index data corresponding to the timing performance index data are calculated, and the mean value and standard deviation of the timing performance index data are substituted into the Z-Score formula for standardization conversion to obtain the standard timing performance index data, wherein the Z-Score formula is ;in, is the original value, It is the average value of the time series performance indicator data based on the historical data of the last 24 hours. is the standard deviation of the time series performance indicator data; through standardization processing, all indicator data in the time series performance indicator data are converted into standardized data with a mean of 0 and a standard deviation of 1, which can eliminate the dimensional differences of different indicators.

[0039] Specifically, based on a sliding window with a fixed window length, the standard time series performance indicator data is slid according to the sliding step size, and statistical features of the standard time series performance indicator data in the sliding window are extracted during the sliding process to obtain basic statistical quantity features, change trend features, distribution features and abnormal indication features.

[0040] Specifically, the basic statistical features are the mean, standard deviation, minimum value, maximum value, and median of the standard time series performance indicator data within the sliding window.

[0041] Specifically, the change trend feature is the slope of the standard time series performance indicator data in the sliding window calculated by linear regression, which is used to reflect the rising or falling trend of the indicator within 5 minutes.

[0042] Specifically, the distribution features are the 25th percentile, 75th percentile, skewness, and kurtosis of the standard time series performance indicator data within the sliding window, which are used to characterize the shape of data distribution.

[0043] Specifically, the abnormal indication feature is that the standard time series performance indicator data in the sliding window exceeds The percentage of data points within a range, used to flag potential abnormal fluctuations.

[0044] Specifically, the above sliding window statistical calculation is performed independently for each indicator in the standard time series performance indicator data. If the system monitors N core numerical indicators, and each indicator generates 10 statistical features in each window, the total number of features in a single window is N×10. These features are arranged in a fixed order of "indicator type → feature category", such as first classifying by indicators such as CPU and memory, and then sorting by features such as mean and standard deviation to form a time series performance indicator feature vector of preset dimensions, which is updated every 1 minute to represent the statistical characteristics of the node's operating status in the last 5 minutes in real time. The number of dimensions of the preset dimensions is determined by the product of the N core numerical indicators monitored by the system and the multiple statistical features generated by each indicator in each window.

[0045] In one embodiment, structured information is extracted from the log data to obtain structured log data, and semantic feature encoding is performed on the structured log data to generate a log feature vector of a preset dimension.

[0046] Specifically, the structured information extraction includes: parsing the log data through a rule engine or regular expression, extracting the timestamp, component name, log level, error code, thread ID, and key message identifier; wherein the log level includes ERROR, WARN, and INFO.

[0047] Preferably, a predefined business sensitive word dictionary or high-frequency abnormal words extracted based on TF-IDF is used for log data with log levels of ERROR and WARN to perform matching marking in the log message body, wherein the abnormal keywords include sensitive words related to faults, such as fail, timeout, lost, etc.

[0048] Specifically, the semantic feature encoding includes: splicing the extracted structured log data and the log data into a natural language sequence, inputting the natural language sequence into a pre-trained language model for encoding, and obtaining the output vector corresponding to the standard special tag position output by the pre-trained language model or the pooled vector of the hidden state by mean pooling as the log feature vector.

[0049] Preferably, the log feature vector has 768 dimensions, wherein the vector dimension is determined by the architecture of the pre-trained language model, and contains the deep semantic information of the log, context association, and the weight of the identified abnormal keywords, which is called a 768-dimensional log feature vector; a feature vector is generated in real time for each newly parsed ERROR / WARN log.

[0050] In one embodiment, graph embedding processing is performed on the nodes in the component dependency graph to generate a graph feature vector of a preset dimension.

[0051] Specifically, the graph embedding process adopts a graph neural network algorithm to sample neighbor nodes within a preset number of hops for each node in the dependency graph using a random sampling strategy of fixed depth and fixed number; it iteratively aggregates the neighbor node features through an aggregator; and maps the aggregation results to a preset dimension through a fully connected layer to generate the graph feature vector.

[0052] Specifically, the fixed depth is 2 hops, and the fixed number is 5 neighbors sampled per hop.

[0053] Specifically, when aggregating neighbor node features iteratively through an aggregator, an initial feature vector is first assigned to each node in the initialization phase, which may include node type, basic resource attributes, or set as a trainable initial vector as the starting point of embedding learning; in K iterations, in the aggregation phase: an LSTM aggregator or a mean pooling aggregator is used to aggregate the node Neighbor nodes The previous round embedding vector of Aggregate into neighbor messages , for example, aggregate the neighbor information of NameNode into a comprehensive message; in the update phase: The previous round of embedding Message neighbors Splicing, updated to the current round embedding through the fully connected layer and nonlinear activation function , the formula is simplified to CONCAT ,in, To learn weights, we can achieve the fusion of our own features and neighbor features, and after K iterations, we can get the aggregation result of node v. .

[0054] Specifically, due to the aggregation result of node v obtained in the last layer The dimension of may be higher or lower than the target dimension, so an additional fully connected layer is used to aggregate the results. Mapping to a fixed-dimensional space, obtaining the final fixed-dimensional graph feature vector, ensures that the output vector dimension is fixed and adapts to the needs of subsequent multimodal data fusion.

[0055] Specifically, the graph feature vector is used to characterize the structural importance of the node in the component dependency graph, and effectively captures the connection density and hub role of the node based on the graph neural network algorithm; and through model training, the importance differences of different node types and connection patterns are learned, so that the embedding vector of the key node occupies a specific position in the vector space or has a larger norm.

[0056] In one embodiment, the time series performance indicator feature vector, the log feature vector, and the graph feature vector are used as preprocessed multimodal data.

[0057] In one embodiment, the time series performance indicator feature vector, the log feature vector and the graph feature vector are respectively mapped to the same dimension based on an independent fully connected neural network layer to obtain a first time series performance indicator feature vector, a first log feature vector and a first graph feature vector of the same latitude.

[0058] Preferably, the same dimension includes but is not limited to 128 dimensions.

[0059] Specifically, the time series performance indicator feature vector, the log feature vector, and the graph feature vector are mapped to the same dimension through a projection layer, and the projection layer adopts LayerNormalization and ReLU activation functions to ensure that the feature distribution is stable and retains nonlinear expression capabilities.

[0060] In one embodiment, corresponding position coding vectors and modality type identifiers are added to the first time series performance indicator feature vector, the first log feature vector, and the first graph feature vector, respectively, to obtain a second time series performance indicator feature vector, a second log feature vector, and a second graph feature vector; the second time series performance indicator feature vector, the second log feature vector, and the second graph feature vector are spliced ​​to obtain a multimodal feature vector sequence.

[0061] Specifically, the position encoding vector is used to mark the order of each feature vector in the multimodal feature vector sequence.

[0062] Specifically, the modality type identifier is used to explicitly distinguish the source of each feature vector, and the modality type identifier includes but is not limited to [TIME], [LOG], and [GRAPH].

[0063] Specifically, the second time series performance indicator feature vector, the second log feature vector, and the second graph feature vector are concatenated into a multimodal feature vector sequence: , forming a 3×128-dimensional input sequence.

[0064] In one embodiment, the multimodal feature vector sequence is input into a multi-layer multi-head attention mechanism encoder, so that the multi-layer multi-head attention mechanism encoder respectively calculates the cross-modal association weights in the multimodal feature vector sequence to generate a fused feature vector.

[0065] Specifically, the multi-layer multi-head attention mechanism encoder is a 3-layer encoder, and each layer contains an 8-head attention mechanism.

[0066] Specifically, the cross-modal association weight includes a first association weight between the second log feature vector and the second time series performance indicator feature vector, a second association weight between the second log feature vector and the second graph feature vector, and a third association weight between the second time series performance indicator feature vector and the second graph feature vector.

[0067] Specifically, when the multi-layer multi-head attention mechanism encoder calculates the cross-modal association weights in the multimodal feature vector sequence respectively, each layer of the multi-head attention mechanism encoder performs a linear transformation on the input multimodal feature vector sequence X to obtain a query matrix Query(Q), a key matrix Key(K) and a value matrix Value(V), where: in, To learn weights; through the attention formula , calculate the association weight, where the product of the transpose of the Q matrix and the K matrix in the numerator measures the similarity of the eigenvectors of different modalities and is divided by the scaling factor Avoid values ​​that are too large; the Softmax function converts the similarity into a normalized weight, which is finally multiplied by the V matrix to obtain the single-head attention output, where the weight matrix directly quantifies the cross-modal association strength.

[0068] Specifically, through the 8-head attention mechanism H, the query matrix Q, key matrix K and value matrix V are divided into H parts, where H is 8; attention is calculated independently, and each head focuses on a different correlation dimension; after splicing the outputs of the 8 heads, they are integrated into the features of a single head dimension through linear transformation, which not only retains the multi-dimensional cross-modal associations but also ensures the uniformity of the output structure.

[0069] Specifically, after iterative processing by three layers of encoders, the output of each layer serves as the input to the next layer, gradually strengthening the deep association between modalities. For example, the first layer may capture the surface association between "log anomaly keywords and network delay timing characteristics", while the second layer, combined with the node dependency in the graph features, deepens it into a chain association of "log anomaly → network delay → unreachable dependent node". The third layer further integrates global structural information to form a more comprehensive association weight.

[0070] Specifically, based on multi-layer multi-head attention calculation, feature expression is optimized through residual connection and feedforward neural network. For example, the output of each layer of attention is added to the input multimodal feature vector sequence and then layer normalization is performed: , to avoid the gradient disappearance of the deep network and retain the core information of the original modal features; at the same time, through two layers of linear transformation Enhance the nonlinear expression ability of features and finally output the fused feature vector.

[0071] Preferably, the fusion process is re-executed every 30 seconds for the fused feature vector; if a modal feature vector is not updated within a period, such as no new log, the value of the previous period is used; if the component dependency graph changes, the graph feature re-embedding and fusion calculation are forcibly triggered.

[0072] In one embodiment, a target fused feature vector is extracted from the fused feature vector, and the target fused feature vector is mapped to a target dimension based on a fully connected layer to obtain a high-dimensional component health status vector.

[0073] Specifically, the [CLS] tag position vector output by the last layer encoder is taken from the fused feature vector, or the outputs of the three modal feature vectors are average pooled to obtain the target fused feature vector.

[0074] Specifically, the target fusion feature vector is mapped to the target dimension through a 512-dimensional fully connected layer, and the Tanh activation function is applied to constrain the output range to [-1, 1]. The output result is a 512-dimensional high-dimensional component health status vector, where each dimension comprehensively represents a certain health attribute of the component.

[0075] Step 102: Input the high-dimensional component health state vector into a fault diagnosis model, so that the fault diagnosis model outputs a target fault type and a root cause confidence.

[0076] In one embodiment, the fault diagnosis model includes a Transformer encoder, a fault classification head, and a confidence calculation head.

[0077] In one embodiment, the high-dimensional component health state vector is input into a large fault diagnosis model so that the large fault diagnosis model normalizes the high-dimensional component health state vector to obtain a normalized high-dimensional component health state vector.

[0078] Specifically, after the high-dimensional component health status vector is input into the large fault diagnosis model, it is first normalized through BatchNormalization processing in the input layer. The BatchNormalization layer calculates the mean and variance of each dimension of the batch input vector, performs a standardized transformation on each dimension, eliminates dimensional interference of different dimensions, and outputs the normalized high-dimensional component health status vector. This step not only accelerates the convergence speed during model training, but also ensures the fairness of the weights of features in each dimension in the subsequent feature extraction process, providing a stable input basis for hierarchical feature extraction.

[0079] In one embodiment, hierarchical feature extraction is performed on the normalized high-dimensional component health state vector based on the Transformer encoder to obtain deep features.

[0080] Specifically, the Transformer encoder includes 4 layers of encoders, and each encoder layer realizes layer-by-layer abstraction of features through a multi-head self-attention mechanism and a feedforward neural network.

[0081] Specifically, the multi-head self-attention mechanism (nhead=8) is used to split the normalized high-dimensional component health status vector into eight parallel subspaces. Each subspace independently calculates the association weights between features, and then splices the results of the eight subspaces to capture the global dependencies between features.

[0082] Specifically, the four-layer encoder processes features in a step-by-step process from shallow to deep. The first layer focuses on local features, such as outliers in a single dimension. The second layer integrates features from related dimensions, such as the coordinated changes in the CPU and memory. The third layer combines multimodal features, such as timing anomalies and log errors. The fourth layer extracts global health status features, such as the overall abnormal pattern of a component.

[0083] Specifically, after being processed by a four-layer Transformer encoder, the final output dimension is a 512-dimensional deep feature. This deep feature is no longer the original scattered feature, but an abstract feature that contains multimodal associations and global structural information, which can accurately represent the health status of the component.

[0084] In one embodiment, the deep features are input into the fault classification head so that the fault classification head outputs a fault type raw score vector corresponding to each fault type; Softmax normalization is performed on the fault type raw score vector to obtain a fault type probability value corresponding to each fault type, and the fault type corresponding to the maximum fault type probability is selected as the target fault type.

[0085] Specifically, the structure of the fault classification head consists of two layers of linear transformations. The first linear transformation layer maps 512-dimensional deep features to 256-dimensional deep features, introduces nonlinearity through the ReLU activation function, and prevents overfitting through Dropout (0.2). The second linear transformation layer maps the 256-dimensional deep features to a dimension consistent with the total number of fault types and outputs the original score vector of the fault type.

[0086] Specifically, the Softmax function is applied to the fault type raw score vector to convert the fault type raw score vector corresponding to each fault type into a probability value between 0 and 1. For example, when the fault type raw score vector corresponding to the fault type is [3.2, 1.5, 0.8], after Softmax, the value may be [0.92, 0.07, 0.01], which correspond to the probabilities of types such as "disk failure" and "network jitter", respectively. The fault type corresponding to the maximum fault type probability is selected as the target fault type. For example, in the above example, the fault type corresponding to 0.92 is "disk failure", and "disk failure" is selected as the target fault type to locate the fault type of the component health status.

[0087] Specifically, the Softmax function is: Where, is the predicted probability value of the i-th fault type, is the raw score vector of the fault type for the i-th fault Do exponential operations, is the total number of fault types; The sum of the exponential scores of all k fault types is used as the normalized denominator.

[0088] In one embodiment, the deep features are input into the confidence calculation head so that the confidence calculation head outputs a basic confidence, a risk weighting factor is determined based on the abnormal dimension value in the high-dimensional component health status vector, and the root cause confidence is calculated based on the risk weighting factor and the basic confidence.

[0089] Specifically, the deep features are simultaneously input into the confidence calculation head, which also contains two layers of linear transformation. The first linear transformation layer maps the 512-dimensional deep features to 128 dimensions and activates them through ReLU. The second linear transformation layer maps the 128 dimensions to 1 dimension and limits the output to the 0-1 range, such as 0.95, through the Sigmoid function. It is then converted into a percentage form, such as 95%, to obtain the basic confidence, which is used to reflect the initial credibility of the model's judgment of the fault type.

[0090] Specifically, based on the abnormal dimension value (MaxAnomalyScore) in the high-dimensional component health status vector (for example, the disk error dimension's Z-Score = 3.5), combined with the learnable parameter α (default 0.1), the risk weighting factor is calculated using the formula: Risk Weighting Factor = 1 + α × MaxAnomalyScore. For example, when α = 0.1 and MaxAnomalyScore = 3.5, the risk weighting factor = 1 + 0.1 × 3.5 = 1.35.

[0091] Specifically, the basic confidence is multiplied by the risk weighting factor to obtain the final root cause confidence. This root cause confidence integrates the credibility of the model judgment and the degree of characteristic abnormality, providing a more comprehensive basis for fault handling decisions.

[0092] Step 103: When the root cause confidence meets the preset confidence threshold, the cluster environment characteristic parameters of the current cluster environment are obtained, and based on the target fault type, the historical repair strategy is determined. Based on the cluster environment characteristic parameters and the historical repair strategy, the cluster environment similarity is calculated. If the cluster environment similarity meets the preset cluster environment similarity threshold, the historical repair strategy is used as a candidate repair strategy. If the cluster environment similarity does not meet the preset cluster environment similarity threshold, a candidate repair strategy is generated based on a preset reinforcement learning strategy generation engine.

[0093] In one embodiment, the preset confidence threshold is not less than 90%; when the root cause confidence meets the preset confidence threshold, it is considered that the feature matches the fault mode very well, and the self-healing operation can be directly triggered.

[0094] Preferably, when the root cause confidence level is between 70% and 90%, it is considered that a minor feature conflict exists, such as network fluctuation, and manual confirmation is required.

[0095] Preferably, when the root cause confidence is less than 70%, it is considered that there is currently a lack of key evidence to support the diagnosis conclusion. At this time, it is marked as a "potential risk" and continuously monitored.

[0096] In one embodiment, when a self-healing operation is triggered, cluster environment characteristic parameters of the current cluster environment are obtained, wherein the cluster environment characteristic parameters include software version characteristics, core configuration parameter characteristics, and cluster scale characteristics.

[0097] Specifically, the software version features include but are not limited to Hadoop version, JDK major version, and Kafka_SCALA version.

[0098] Specifically, the core configuration parameter features include but are not limited to HDFS block size and YARN container memory.

[0099] Specifically, the cluster scale characteristics include but are not limited to the total number of nodes, total storage capacity, and average daily task volume.

[0100] In one embodiment, based on the target fault type, the system queries a historical policy library and obtains a historical repair policy corresponding to the target fault type from the historical policy library.

[0101] In one embodiment, the dimensional similarities of the software version characteristics, the core configuration parameter characteristics, the cluster scale characteristics and the historical repair strategy are calculated respectively to obtain the software version characteristic similarity, the core configuration parameter characteristic similarity and the cluster scale characteristic similarity; based on the preset weight value, the software version characteristic similarity, the core configuration parameter characteristic similarity and the cluster scale characteristic similarity are weightedly fused to obtain the cluster environment similarity.

[0102] Specifically, when calculating the dimensional similarity between the software version feature and the historical repair strategy, the historical software version feature corresponding to the historical repair strategy is obtained, and the software version feature and the historical software version feature are input into the software version feature similarity calculation formula to obtain the software version feature similarity; wherein, the software version feature similarity calculation formula is as follows:

[0103] ;

[0104] Where, is the software version feature similarity, is the similarity of the i-th software version feature in the software version feature, is the i-th software version feature in the software version feature, is the i-th software version feature in the historical software version features.

[0105] Specifically, when calculating the dimensional similarity between the core configuration parameter feature and the historical repair strategy, the core configuration parameter feature is divided into discrete core configuration parameter features and continuous core configuration parameter features, and the historical discrete core configuration parameter features and the historical continuous core configuration parameter features in the historical repair strategy are obtained; the continuous core configuration parameter features and the historical continuous core configuration parameter features are input into a preset continuous core configuration parameter feature similarity calculation formula to obtain the continuous core configuration parameter feature similarity; and the number of parameters with the same configuration values ​​in the discrete core configuration parameter feature and the historical discrete core configuration parameter feature is obtained, the number of parameters and the total parameter data of the discrete core configuration parameter feature are substituted into the preset discrete core configuration parameter feature similarity calculation formula to obtain the discrete core configuration parameter feature similarity, and the continuous core configuration parameter feature similarity and the discrete core configuration parameter feature similarity are used as the core configuration parameter feature similarity, as shown below:

[0106] ;

[0107] Where, is the core configuration parameter feature similarity, is the weight of the j-th core configuration parameter, m is the number of continuous core configuration parameter features or the number of discrete core configuration parameter features, is the jth continuous core configuration parameter feature in the core configuration parameter feature, is the jth continuous historical core configuration parameter feature in the historical core configuration parameter feature; The number of parameters in the discrete core configuration parameter feature that have the same configuration values ​​as those in the historical discrete core configuration parameter feature.

[0108] Specifically, when calculating the dimensional similarity between the cluster scale feature and the historical repair strategy, the historical cluster scale feature corresponding to the historical repair strategy is obtained, and the cluster scale feature and the historical cluster scale feature are input into the cluster scale feature similarity calculation formula to obtain the cluster scale feature similarity; wherein, the cluster scale feature similarity calculation formula is as follows:

[0109] ;

[0110] Where, is the cluster size feature similarity, is the cluster size characteristic, is the historical cluster size characteristic, is the maximum value of the historical cluster size feature.

[0111] Specifically, the software version feature similarity, the core configuration parameter feature similarity and the cluster size feature similarity are substituted into the cluster environment similarity calculation formula to obtain the cluster environment similarity. , wherein the cluster environment similarity calculation formula is as follows:

[0112] .

[0113] Preferably, the preset cluster environment similarity threshold is 80%; if the cluster environment similarity When the probability is not less than 80%, the historical repair strategy is used as a candidate repair strategy.

[0114] In one embodiment, if the cluster environment similarity does not meet a preset cluster environment similarity threshold, it means that there is no matching repair strategy in the historical strategy library, and a reinforcement learning strategy generation engine is started to generate candidate repair strategies corresponding to the fault type.

[0115] In one embodiment, based on a preset reinforcement learning strategy generation engine, when generating candidate repair strategies, a state space is constructed, wherein the state space includes a multi-dimensional real-time state vector; an action space is defined, wherein the action space includes basic operations and operation combination rules; a reward function is designed, wherein the reward function is calculated based on the strategy execution time, resource consumption and repair result; the initial strategy is optimized using a proximal strategy optimization algorithm to obtain an optimized strategy, wherein the optimization includes a strategy network update and an acceleration mechanism; the optimized strategy is verified in a simulation environment, and if the optimized strategy meets the preset success rate threshold and timeliness requirements, the optimized strategy is output as a candidate repair strategy.

[0116] Specifically, when constructing the state space, a multidimensional real-time state vector is extracted, where the multidimensional real-time state vector includes but is not limited to cluster load, resource level, and fault context; the cluster load is CPU usage (5-minute average) and network I / O bandwidth occupancy; the resource level is the YARN available memory ratio and HDFS storage remaining rate; the fault context is the fault component graph centrality and the activation intensity of abnormal log words.

[0117] Specifically, when defining the action space, the basic operations include restarting services, adjusting parameters, retrying tasks, etc., and the operation combination rules support single actions or ordered action sequences, such as [isolate node → migrate data → restart].

[0118] Specifically, a reward function is designed, wherein the reward function is as follows:

[0119] ;

[0120] Where, is the weight coefficient (default 0.6, 0.4, 1.0), The execution time of the strategy is The CPU / memory resources consumed, It is the total CPU / memory resources.

[0121] Specifically, the policy network is updated as follows:

[0122]

[0123]

[0124] Where, is the advantage function, are the policy network parameters to be optimized, i.e. the weights and biases of the neural network, Take the expectation for the sample at time step t, is the importance sampling ratio; For the general Clip to interval [ ]Inside, For the new strategy in state Select Action The probability of For the old policy (policy before the last update) in the state Select Action probability.

[0125] Specifically, the acceleration mechanism includes parallel environment sampling and strategy cache reuse, wherein the parallel environment sampling is to execute actions in 8 cluster simulation environments at the same time, and the strategy cache reuse is to skip repeated calculations for similar states.

[0126] Specifically, the policy is executed in the sandbox environment, and the verification success rate is ≥95% and then output.

[0127] Specifically, the timeliness requirement is distributed training + GPU acceleration, with an average generation time of less than 9 seconds, meeting the requirement of less than 10 seconds.

[0128] Step 104: Match the candidate repair strategy with a corresponding strategy level, and execute the candidate repair strategy based on the strategy level.

[0129] In one embodiment, based on the fault type, the candidate repair strategies are mapped to corresponding strategy levels, wherein the strategy levels include at least a first-level strategy, a second-level strategy, and a third-level strategy.

[0130] Specifically, when the fault type is network transient jitter or query syntax error, the candidate repair strategy is mapped to the corresponding first-level strategy.

[0131] Specifically, when the fault type is service freeze or configuration hot reload failure, the candidate repair strategy is mapped to the corresponding secondary strategy.

[0132] Specifically, when the fault type is a hardware fault or physical data damage, the candidate repair strategy is mapped to a corresponding three-level strategy.

[0133] In one embodiment, the candidate repair policy is executed based on the policy level, and the policy execution time of the candidate repair policy is monitored.

[0134] Specifically, the policy operations corresponding to the first-level policy include but are not limited to adjusting parameters and retrying requests; the estimated time corresponding to the first-level policy is less than 2 minutes.

[0135] Specifically, the policy operations corresponding to the secondary policy include but are not limited to rolling restart of nodes and dynamic loading of resource pool configuration; the estimated time required for the secondary policy is 2-10 minutes.

[0136] Specifically, the policy operations corresponding to the three-level strategy include but are not limited to replacing failed nodes, rebalancing HDFS data, and cluster expansion; the estimated time required for the three-level strategy is greater than 10 minutes.

[0137] In one embodiment, if the execution time of the policy exceeds a preset timeout threshold corresponding to the policy level, a corresponding timeout processing mechanism is triggered, wherein the timeout processing mechanism includes executing a backup policy of the same level or upgrading the policy level.

[0138] Specifically, the timeout handling mechanism corresponding to the first-level strategy includes but is not limited to interrupting the current operation, triggering the backup strategy at the same level, such as adjusting other parameters, and upgrading to the second-level strategy if failure continues.

[0139] Specifically, the timeout handling mechanism corresponding to the secondary strategy includes but is not limited to rolling back to a safe state, such as restoring the old configuration, starting an alternative solution of the same level, such as switching to a backup node, and upgrading to a third-level strategy if failure continues.

[0140] Specifically, the timeout handling mechanism corresponding to the three-level strategy includes but is not limited to suspending all operations, sending emergency alerts, such as text messages / phone calls, and waiting for manual intervention if failures persist.

[0141] In one embodiment, typical fault cases are manually labeled and combined with a large model to generate simulated abnormal data to construct a sample training set containing input features, fault types, optimal strategies, and effect labels. After each self-healing, repair logs are automatically extracted to generate a new sample training set. The sample training set and the new sample training set are integrated to obtain a training set.

[0142] Specifically, 20,000 typical faults were manually labeled, and the large model generated 180,000 simulated data based on rules, such as injecting different types of abnormal scenarios, to form a sample training set containing input features-fault type-optimal strategy-effect labels.

[0143] Specifically, after each self-healing is completed, the system automatically extracts the repair log to generate a new sample training set. Operations and maintenance experts review cases with an accuracy rate of less than 90% every week to ensure the quality of the labeled data.

[0144] In one embodiment, based on the training set, each cluster locally trains a sub-model, iteratively calculates and encrypts the gradient; the central server dynamically weights and aggregates the gradients according to the cluster size to generate a global model; after the global model is reverse-inferred to each cluster, it is fine-tuned and adapted using local new data.

[0145] Specifically, during local training, each cluster uses PyTorch to train the sub-model, calculates the gradient and performs homomorphic encryption every 1000 iterations, and the encryption time is less than 50ms.

[0146] Specifically, when dynamically weighting and aggregating gradients, the central server collects the gradients of each cluster and aggregates them using the FedAvg algorithm. At this time, the weights are dynamically adjusted according to the cluster size to generate a global model. The aggregation period is configurable and defaults to once per hour.

[0147] Specifically, when the global model is reversed, it is reversed to each cluster and fine-tuned using local new data from the past 24 hours. The fine-tuning time is less than 30 minutes, which improves the adaptability to the specific cluster environment.

[0148] In one embodiment, the model is evaluated by fault classification F1-score, strategy generation accuracy, and repair success rate; for continuous low success rate faults, exclusive cases are extracted and model parameters are optimized through transfer learning.

[0149] Specifically, when the success rate of repairing a certain type of fault is less than 80% for three consecutive weeks, such as the failure of Kafka controller election, special training is triggered. More than 10,000 cases of this type of fault are extracted from the knowledge base, and transfer learning is used to optimize model parameters in a targeted manner. The training cycle is 4 hours.

[0150] In one embodiment, through a closed-loop federated learning model, industry-level fault case sharing is achieved while protecting data privacy, enabling the model to quickly adapt to new scenarios and avoiding the inefficiency of "reinventing the wheel in each cluster." By fully recording the repair process data and feeding it back into model training, a positive cycle of "operation and maintenance experience → data → model → better operation and maintenance" is formed, achieving continuous evolution of system capabilities.

[0151] Example 2, see Figure 2 , Figure 2This is a structural diagram of an embodiment of a self-healing operation and maintenance device for a big data component provided by the present application. Corresponding to the above self-healing operation and maintenance method for a big data component, the present application also provides a self-healing operation and maintenance device for a big data component. The self-healing operation and maintenance device for the big data component includes a module for executing the above self-healing operation and maintenance method for the big data component. The self-healing operation and maintenance device for the big data component can be configured in a desktop computer, tablet computer, laptop computer, and other terminals; specifically, the self-healing operation and maintenance device for the big data component includes a multimodal data processing module 201, a fault diagnosis module 202, a candidate repair strategy generation module 203, and a candidate repair strategy execution module 204.

[0152] The multimodal data processing module 201 is used to collect multimodal data of the big data component, preprocess the multimodal data to obtain preprocessed multimodal data, and perform feature fusion processing on the preprocessed multimodal data to obtain a high-dimensional component health status vector.

[0153] The fault diagnosis module 202 is used to input the high-dimensional component health state vector into the fault diagnosis model, so that the fault diagnosis model outputs the target fault type and root cause confidence.

[0154] The candidate repair strategy generation module 203 is used to obtain the cluster environment characteristic parameters of the current cluster environment when the root cause confidence meets the preset confidence threshold, and determine the historical repair strategy based on the target fault type, and calculate the cluster environment similarity based on the cluster environment characteristic parameters and the historical repair strategy. If the cluster environment similarity meets the preset cluster environment similarity threshold, the historical repair strategy is used as a candidate repair strategy. If the cluster environment similarity does not meet the preset cluster environment similarity threshold, a candidate repair strategy is generated based on a preset reinforcement learning strategy generation engine.

[0155] The candidate repair strategy execution module 204 is configured to match the candidate repair strategy with a corresponding strategy level and execute the candidate repair strategy based on the strategy level.

[0156] In one embodiment, the multimodal data processing module 201 is used to preprocess the multimodal data to obtain preprocessed multimodal data, specifically including: the multimodal data includes time series performance indicator data, log data and component dependency graph; standardizing the time series performance indicator data to obtain standard time series performance indicator data, and performing statistical feature extraction on the time series performance indicator data to generate a time series performance indicator feature vector of a preset dimension, extracting structured information from the log data to obtain structured log data, and performing semantic feature encoding on the structured log data to generate a log feature vector of a preset dimension; performing graph embedding processing on the nodes in the component dependency graph to generate a graph feature vector of a preset dimension; and using the time series performance indicator feature vector, the log feature vector and the graph feature vector as preprocessed multimodal data.

[0157] In one embodiment, the multimodal data processing module 201 is used to perform feature fusion processing on the pre-processed multimodal data to obtain a high-dimensional component health state vector, specifically including: mapping the time series performance indicator feature vector, the log feature vector and the graph feature vector to the same dimension based on an independent fully connected neural network layer to obtain a first time series performance indicator feature vector, a first log feature vector and a first graph feature vector of the same latitude; adding corresponding position encoding vectors and modality type identifiers to the first time series performance indicator feature vector, the first log feature vector and the first graph feature vector to obtain a second time series performance indicator feature vector. An indicator feature vector, a second log feature vector and a second graph feature vector; the second time series performance indicator feature vector, the second log feature vector and the second graph feature vector are concatenated to obtain a multimodal feature vector sequence; the multimodal feature vector sequence is input into a multi-layer multi-head attention mechanism encoder, so that the multi-layer multi-head attention mechanism encoder respectively calculates the cross-modal association weights in the multimodal feature vector sequence to generate a fusion feature vector; a target fusion feature vector is extracted from the fusion feature vector, and the target fusion feature vector is mapped to the target dimension based on a fully connected layer to obtain a high-dimensional component health status vector.

[0158] In one embodiment, the fault diagnosis module 202 is used to input the high-dimensional component health state vector into the fault diagnosis big model so that the fault diagnosis big model outputs the target fault type and root cause confidence, specifically including: the fault diagnosis big model includes a Transformer encoder, a fault classification head and a confidence calculation head; the high-dimensional component health state vector is input into the fault diagnosis big model so that the fault diagnosis big model normalizes the high-dimensional component health state vector to obtain a normalized high-dimensional component health state vector; based on the Transformer encoder, the normalized high-dimensional component health state vector is subjected to hierarchical feature extraction. Extract features to obtain deep features; input the deep features into the fault classification head so that the fault classification head outputs a fault type raw score vector corresponding to each fault type; perform Softmax normalization on the fault type raw score vector to obtain a fault type probability value corresponding to each fault type, and select the fault type corresponding to the maximum fault type probability as the target fault type; input the deep features into the confidence calculation head so that the confidence calculation head outputs a basic confidence, determines a risk weighting factor based on the abnormal dimension value in the high-dimensional component health status vector, and calculates the root cause confidence based on the risk weighting factor and the basic confidence.

[0159] In one embodiment, the candidate repair strategy generation module 203 is used to calculate the cluster environment similarity based on the cluster environment characteristic parameters and the historical repair strategy, specifically including: the cluster environment characteristic parameters include software version characteristics, core configuration parameter characteristics and cluster scale characteristics; respectively calculating the dimensional similarity of the software version characteristics, the core configuration parameter characteristics, the cluster scale characteristics and the historical repair strategy to obtain software version characteristic similarity, core configuration parameter characteristic similarity and cluster scale characteristic similarity; based on a preset weight value, performing weighted fusion processing on the software version characteristic similarity, the core configuration parameter characteristic similarity and the cluster scale characteristic similarity to obtain cluster environment similarity.

[0160] In one embodiment, the candidate repair strategy generation module 203 is used to generate candidate repair strategies based on a preset reinforcement learning strategy generation engine, specifically including: constructing a state space, wherein the state space includes a multi-dimensional real-time state vector; defining an action space, wherein the action space includes basic operations and operation combination rules; designing a reward function, wherein the reward function is calculated based on the strategy execution time, resource consumption and repair result; optimizing the initial strategy using a proximal strategy optimization algorithm to obtain an optimized strategy, wherein the optimization includes a strategy network update and an acceleration mechanism; verifying the optimized strategy in a simulation environment, and if the optimized strategy meets the preset success rate threshold and timeliness requirements, outputting the optimized strategy as a candidate repair strategy.

[0161] In one embodiment, the candidate repair strategy execution module 204 is used to match the candidate repair strategy with the corresponding strategy level and execute the candidate repair strategy based on the strategy level, specifically including: mapping the candidate repair strategy to the corresponding strategy level based on the fault type, wherein the strategy level includes at least a first-level strategy, a second-level strategy and a third-level strategy; executing the candidate repair strategy based on the strategy level, and monitoring the strategy execution time of the candidate repair strategy; if the strategy execution time exceeds the preset timeout threshold corresponding to the strategy level, triggering the corresponding timeout processing mechanism, wherein the timeout processing mechanism includes executing a backup strategy of the same level or upgrading the strategy level.

[0162] The self-healing operation and maintenance device of the big data component can implement the self-healing operation and maintenance method of the big data component of the above method embodiment. The options in the above method embodiment are also applicable to this embodiment and will not be described in detail here.

[0163] like Figure 3 As shown, Figure 3 This is a structural diagram of a computer device provided by the present application; it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs.

[0164] In one embodiment of the present application, the processor 111 is configured to implement the self-healing operation and maintenance method of the big data component provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113 .

[0165] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0166] Therefore, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the self-healing operation and maintenance method of a big data component provided in any of the aforementioned method embodiments are implemented.

[0167] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, which can store program code. The computer-readable storage medium can be either non-volatile or volatile.

[0168] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0170] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0171] If this integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application.

[0172] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, as long as these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0174] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A self-healing operation and maintenance method for a big data component, characterized in that: include: Collecting multimodal data of a big data component, preprocessing the multimodal data to obtain preprocessed multimodal data, and performing feature fusion processing on the preprocessed multimodal data to obtain a high-dimensional component health state vector, wherein the multimodal data includes time series performance indicator data, log data, and a component dependency graph; Inputting the high-dimensional component health state vector into a fault diagnosis model so that the fault diagnosis model outputs a target fault type and a root cause confidence, wherein the fault diagnosis model includes a Transformer encoder, a fault classification head, and a confidence calculation head; When the root cause confidence meets a preset confidence threshold, cluster environment characteristic parameters of the current cluster environment are obtained, and based on the target fault type, a historical repair strategy is determined. Cluster environment similarity is calculated based on the cluster environment characteristic parameters and the historical repair strategy. If the cluster environment similarity meets a preset cluster environment similarity threshold, the historical repair strategy is used as a candidate repair strategy. If the cluster environment similarity does not meet the preset cluster environment similarity threshold, a candidate repair strategy is generated based on a preset reinforcement learning strategy generation engine. Matching the candidate repair strategy with a corresponding strategy level, and executing the candidate repair strategy based on the strategy level; The multimodal data is preprocessed to obtain preprocessed multimodal data, specifically comprising: Performing graph embedding processing on the nodes in the component dependency graph to generate a graph feature vector of a preset dimension; The time series performance index feature vector, the log feature vector and the graph feature vector are used as preprocessed multimodal data; Calculating cluster environment similarity based on the cluster environment characteristic parameters and the historical repair strategy specifically includes: The cluster environment characteristic parameters include software version characteristics, core configuration parameter characteristics and cluster scale characteristics; Calculating the dimensional similarity of the software version feature, the core configuration parameter feature, the cluster size feature, and the historical repair strategy respectively, to obtain software version feature similarity, core configuration parameter feature similarity, and cluster size feature similarity; Based on a preset weight value, a weighted fusion process is performed on the software version feature similarity, the core configuration parameter feature similarity, and the cluster size feature similarity to obtain the cluster environment similarity.

2. The method according to claim 1, wherein: Preprocessing the multimodal data to obtain preprocessed multimodal data specifically includes: Standardize the time series performance indicator data to obtain standard time series performance indicator data, and extract statistical features from the time series performance indicator data to generate a time series performance indicator feature vector of a preset dimension. Structural information is extracted from the log data to obtain structured log data, and semantic feature encoding is performed on the structured log data to generate a log feature vector of a preset dimension.

3. The method according to claim 2, wherein: Performing feature fusion processing on the pre-processed multimodal data to obtain a high-dimensional component health state vector specifically includes: Mapping the time series performance indicator feature vector, the log feature vector, and the graph feature vector to the same dimension based on independent fully connected neural network layers, respectively, to obtain a first time series performance indicator feature vector, a first log feature vector, and a first graph feature vector of the same dimension; Adding corresponding position encoding vectors and modality type identifiers to the first time series performance indicator feature vector, the first log feature vector, and the first graph feature vector, respectively, to obtain a second time series performance indicator feature vector, a second log feature vector, and a second graph feature vector; Concatenating the second time series performance indicator feature vector, the second log feature vector, and the second graph feature vector to obtain a multimodal feature vector sequence; Inputting the multimodal feature vector sequence into a multi-layer multi-head attention mechanism encoder, so that the multi-layer multi-head attention mechanism encoder respectively calculates the cross-modal association weights in the multimodal feature vector sequence to generate a fused feature vector; A target fused feature vector is extracted from the fused feature vector, and the target fused feature vector is mapped to a target dimension based on a fully connected layer to obtain a high-dimensional component health status vector.

4. The method according to claim 1, wherein Inputting the high-dimensional component health state vector into a fault diagnosis model so that the fault diagnosis model outputs a target fault type and a root cause confidence level, specifically including: The high-dimensional component health state vector is input into the fault diagnosis large model, so that the fault diagnosis large model normalizes the high-dimensional component health state vector to obtain a normalized high-dimensional component health state vector; Performing hierarchical feature extraction on the normalized high-dimensional component health state vector based on the Transformer encoder to obtain deep features; Inputting the deep features into the fault classification head so that the fault classification head outputs a fault type raw score vector corresponding to each fault type; performing Softmax normalization on the fault type raw score vector to obtain a fault type probability value corresponding to each fault type, and selecting the fault type corresponding to the maximum fault type probability as the target fault type; The deep features are input into the confidence calculation head so that the confidence calculation head outputs a basic confidence, a risk weighting factor is determined based on the abnormal dimension value in the high-dimensional component health status vector, and a root cause confidence is calculated based on the risk weighting factor and the basic confidence.

5. The method according to claim 1, wherein: Based on the preset reinforcement learning strategy generation engine, candidate repair strategies are generated, including: Constructing a state space, wherein the state space includes a multi-dimensional real-time state vector; defining an action space, wherein the action space includes basic operations and operation combination rules; Design a reward function, wherein the reward function is calculated based on the strategy execution time, resource consumption, and repair results; The initial strategy is optimized using a proximal strategy optimization algorithm to obtain an optimized strategy, wherein the optimization includes a strategy network update and an acceleration mechanism; The optimization strategy is verified in a simulation environment. If the optimization strategy meets the preset success rate threshold and timeliness requirements, the optimization strategy is output as a candidate repair strategy.

6. The method according to claim 1, wherein: Matching the candidate repair strategy to a corresponding strategy level and executing the candidate repair strategy based on the strategy level specifically includes: Based on the fault type, mapping the candidate repair strategy to a corresponding strategy level, wherein the strategy level includes at least a first-level strategy, a second-level strategy, and a third-level strategy; Executing the candidate repair strategy based on the strategy level, and monitoring the strategy execution time of the candidate repair strategy; If the execution time of the policy exceeds the preset timeout threshold corresponding to the policy level, the corresponding timeout processing mechanism is triggered, wherein the timeout processing mechanism includes executing a backup policy of the same level or upgrading the policy level.

7. A self-healing operation and maintenance device for a big data component, characterized in that: include: Multimodal data processing module, fault diagnosis module, candidate repair strategy generation module and candidate repair strategy execution module; The multimodal data processing module is used to collect multimodal data of the big data component, preprocess the multimodal data to obtain preprocessed multimodal data, and perform feature fusion processing on the preprocessed multimodal data to obtain a high-dimensional component health status vector, wherein the multimodal data includes time series performance indicator data, log data, and component dependency graph; The fault diagnosis module is configured to input the high-dimensional component health state vector into a fault diagnosis model so that the fault diagnosis model outputs a target fault type and a root cause confidence, wherein the fault diagnosis model includes a Transformer encoder, a fault classification head, and a confidence calculation head; The candidate repair strategy generation module is configured to obtain cluster environment characteristic parameters of the current cluster environment when the root cause confidence meets a preset confidence threshold, and determine a historical repair strategy based on the target fault type; calculate cluster environment similarity based on the cluster environment characteristic parameters and the historical repair strategy; if the cluster environment similarity meets a preset cluster environment similarity threshold, use the historical repair strategy as a candidate repair strategy; if the cluster environment similarity does not meet the preset cluster environment similarity threshold, generate a candidate repair strategy based on a preset reinforcement learning strategy generation engine; The candidate repair strategy execution module is configured to match the candidate repair strategy with a corresponding strategy level and execute the candidate repair strategy based on the strategy level; The multimodal data processing module is used to collect multimodal data of the big data component and preprocess the multimodal data to obtain preprocessed multimodal data, specifically including: Performing graph embedding processing on the nodes in the component dependency graph to generate a graph feature vector of a preset dimension; The time series performance index feature vector, the log feature vector and the graph feature vector are used as preprocessed multimodal data; The candidate repair strategy generation module is configured to calculate the cluster environment similarity based on the cluster environment characteristic parameters and the historical repair strategy, specifically including: The cluster environment characteristic parameters include software version characteristics, core configuration parameter characteristics and cluster scale characteristics; Calculating the dimensional similarity of the software version feature, the core configuration parameter feature, the cluster size feature, and the historical repair strategy respectively, to obtain software version feature similarity, core configuration parameter feature similarity, and cluster size feature similarity; Based on a preset weight value, a weighted fusion process is performed on the software version feature similarity, the core configuration parameter feature similarity, and the cluster size feature similarity to obtain the cluster environment similarity.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 can be implemented.

Images