Multi-modal operation and maintenance data fault determination method and system based on large model

Abstract

The invention discloses a multi-modal operation and maintenance data method and system based on a large model, and the method comprises the steps: carrying out the preprocessing of obtained multi-modal operation and maintenance data, so as to obtain data feature sequences with the same feature dimension corresponding to different-modal operation and maintenance data; fusing the data feature sequences with the same feature dimension corresponding to the operation and maintenance data in different modes; processing the fusion vector, inputting the processed fusion vector into the fine-tuned large model, and outputting a semantic reasoning result; analyzing the semantic reasoning result into data in a structured form; and secondary verification is carried out on the abnormal event in the data in the structured form to determine final fault information, and the fault information is expressed in the structured form. A multi-source data fusion strategy is combined with large model reasoning, so that storage, retrieval and analysis of massive heterogeneous operation and maintenance data are both real-time and accurate. Depending on the reasoning ability of a large model, the system can timely give a problem analysis result, the fault processing speed is accelerated, and the service quality is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal operation and maintenance data fault determination method and system based on a large model. BACKGROUND

[0002] In recent years, AIOps (Intelligent Operation and Maintenance) technology has gradually developed. Traditional operation and maintenance analysis methods often use machine learning / rule engines to process different types of data, which requires a large amount of manual feature engineering and rule design, and lacks semantic association capabilities across data sources.

[0003] In related technologies, different modal operation and maintenance data are often modeled independently, and it is difficult to automatically link the performance in the same event, which affects the accuracy of fault location and root cause analysis. Different types of data require separate design of feature extraction and rules, and a large amount of manual intervention leads to poor scalability and high maintenance costs. Traditional methods are difficult to automatically abstract high-level semantic entities or events, and cannot accurately capture business impact and complex causal chains. When the model is migrated to a new system or scenario, a large amount of manual adjustment is required, making it difficult to adapt to changing application environments. A single model usually only targets a single task (such as only detecting or locating abnormalities), and cannot simultaneously meet multiple requirements such as detection, diagnosis, and report generation. SUMMARY

[0004] The main purpose of the present application is to provide a multi-modal operation and maintenance data fault determination method and system based on a large model to solve the problems in the related art.

[0005] In order to achieve the above purpose, according to the first aspect of the present application, a multi-modal operation and maintenance data fault determination method based on a large model is provided, which comprises: preprocessing the obtained multi-modal operation and maintenance data to obtain data feature sequences with the same feature dimension corresponding to different modal operation and maintenance data; fusing the data feature sequences with the same feature dimension corresponding to different modal operation and maintenance data, wherein the data feature sequences are first fused in the feature dimension during fusion; second fusion is performed based on a cross-attention mechanism; third fusion is performed based on an encoder coding method; the weights corresponding to different fusion results are calculated based on a preset network, and the final fusion vector is obtained based on the weights; the fusion vector is processed and input into a fine-tuned large model, and a semantic reasoning result is output; the semantic reasoning result is parsed into structured data; secondary verification is performed on abnormal events in the structured data to determine the final fault information, wherein the fault information is represented in a structured form.

[0006] Optionally, the preprocessing of the obtained multi-modal operation and maintenance data comprises: cleaning, desensitizing, time aligning and embedding the obtained multi-modal operation and maintenance data to ensure a unified feature dimension d, to obtain a feature sequence, wherein the multi-modal operation and maintenance data comprises log data, property index data and call chain data; and the feature sequence comprises a log feature sequence log_feats∈R^{B×L×d}, a metric feature sequence metric_feats∈R^{B×M×d}, and a call chain feature sequence trace_feats∈R^{B×T×d}, wherein B is a batch size, L is a log feature sequence length, M is a metric feature sequence length, and T is a call chain feature sequence length.

[0007] Optionally, the first fusion of the data feature sequences in the feature dimension during the fusion comprises: performing spatial aggregation on the obtained feature sequences to obtain three [batch, d] vectors, i.e., a vector x_log∈R^{B×d} corresponding to the log feature sequence, a vector x_metric∈R^{B×d} corresponding to the metric feature sequence, and a vector x_trace∈R^{B×d} corresponding to the call chain feature sequence; and concatenating the vectors in the feature dimension to obtain a [batch, 3d] vector, i.e., fused_raw = concat(x_log, x_metric, x_trace)∈R^{B×3d}, wherein, for an output dimension d, the [batch, 3d] vector is converted into a [batch, d] vector by full connection mapping, i.e., fused = W·fused_raw + b∈R^{B×d}, wherein W∈R^{d×3d}.

[0008] Optionally, the second fusion based on the cross-attention mechanism comprises: calculating multi-head attention between any two feature sequences; performing sequence dimension pooling on the multi-head attention; and combining the pooled vectors according to a specified strategy to obtain a second fusion result.

[0009] Optionally, the multi-head attention between any two feature sequences comprises: taking one of the two feature sequences as Query and the other as [key, value] to calculate the multi-head attention; wherein, taking the log feature sequence as Query and the metric feature sequence as [key, value] to calculate the multi-head attention, obtaining attn_log_metric∈R^{B×L×d}, and pooling along the sequence dimension to obtain v_log_metric∈R^{B×d}; taking the metric feature sequence as Query and the call chain feature sequence as [key, value] to calculate the multi-head attention, obtaining attn_metric_trace, and pooling along the sequence dimension to obtain v_metric_trace; taking the call chain feature sequence as Query and the log feature sequence as [key, value] to calculate the multi-head attention, obtaining attn_trace_log, and pooling along the sequence dimension to obtain v_trace_log.

[0010] Optionally, the combination of the pooled vectors according to the specified strategy to obtain the second fusion result comprises: combining each multi-head attention according to the strategy to obtain the final medium-term fusion output fused = combine(v_log_metric, v_metric_trace, v_trace_log)∈R^{B×d}, wherein the strategy comprises addition, concatenation and projection, or gated weighting.

[0011] Optionally, the third fusion based on the encoding manner of the encoder comprises: concatenating the log feature sequence log_feats, the metric feature sequence metric_feats and the scheduling chain feature sequence trace_feats at the beginning and end in the sequence dimension to obtain seq∈R^{B×(L+M+T)×d}; adding modal type embedding and time and position embedding to each sequence segment obtained; sending the seq with added embedding into a multi-layer Transformer encoder to obtain an encoding output out_seq∈R^{B×(L+M+T)×d}; and aggregating the encoding output to obtain a third fusion vector fused∈R^{B×d}.

[0012] Optionally, the preset network is used to calculate weights corresponding to different fusion results, and the weights are used for fusion to obtain a final fusion vector, including: the log feature sequence, the index feature sequence, and the call chain feature sequence are summarized respectively; the summarized feature sequences are spliced into a gate input vector gate_input = concat(x_log, x_metric, x_trace) e R^Bx3d; weights corresponding to different fusion modes are calculated based on the gate input vector weights = softmax(Gate(gate_input)) e R^BxE, where E is the number of experts; the weights, the first fusion result, the second fusion result, and the third fusion result expert_i_output e R^Bxd are weighted and summed to obtain a final fusion vector fused = Σ_i weights[:,i:i+1] * expert_i_output, where fused e R^Bxd and i = 1..E. Optionally, when the large model is fine-tuned, a sample fusion vector after the sample is fused is obtained; the sample fusion vector and a pre-constructed prompt are input into the large model to fine-tune the large model; the semantic reasoning result is parsed into structured data, including: the semantic reasoning result is parsed into recognized fault entities, event summaries, and causal chain relationships. According to the second aspect of the present application, a multi-modal operation and maintenance data system based on a large model is provided, including a preprocessing unit configured to preprocess acquired multi-modal operation and maintenance data to obtain data feature sequences corresponding to different modal operation and maintenance data and having the same feature dimension; a fusion unit configured to fuse the data feature sequences corresponding to different modal operation and maintenance data and having the same feature dimension, wherein the data feature sequences are first fused in the feature dimension during fusion; second fusion is performed based on a cross-attention mechanism; third fusion is performed based on an encoder encoding method; weights corresponding to different fusion results are calculated based on a preset network, and a final fusion vector is obtained based on the weights; a prediction unit configured to input the fusion vector into a fine-tuned large model after processing to output a semantic reasoning result; the semantic reasoning result is parsed into structured data; and an abnormal event in the structured data is secondarily verified to determine a final fault information, wherein the fault information is represented in a structured form.

[0013] The embodiment of the present application is a multi-modal operation and maintenance data method and system based on a large model, wherein the method comprises: pre-processing the obtained multi-modal operation and maintenance data to obtain data feature sequences corresponding to different modal operation and maintenance data with the same feature dimension; fusing the data feature sequences corresponding to different modal operation and maintenance data with the same feature dimension; inputting the fused vector after processing into the large model after fine-tuning, and outputting a semantic reasoning result; parsing the semantic reasoning result into structured data; performing secondary verification on the abnormal events in the structured data to determine the final fault information, wherein the fault information is represented in a structured form. The multi-source data fusion strategy combined with the large model reasoning makes the storage, retrieval and analysis of massive heterogeneous operation and maintenance data consider both real-time and accuracy. Relying on the reasoning capability of the large model, the system can give problem analysis results in time, speed up the fault processing speed, and ensure the service quality. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0015] Figure 1 is a flowchart of a multi-modal operation and maintenance data fault determination method based on a large model according to an embodiment of the present application; Figure 2 is an application schematic diagram of a multi-modal operation and maintenance data fault determination method based on a large model according to an embodiment of the present application.

[0016] Figure 3 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make those skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0019] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0020] According to an embodiment of the present application, a multi-modal operation and maintenance data fault determination method based on a large model is provided, as shown in Figure 1 The steps 101 to 103 are as follows: Step 101: Preprocessing the obtained multi-modal operation and maintenance data to obtain data feature sequences with the same feature dimension corresponding to different modal operation and maintenance data.

[0021] In this step, the multi-modal operation and maintenance data includes log data, which records system events, errors, user operations, etc. in the form of unstructured text, and the information is rich but the format is not the same. Including index data such as CPU, memory and other resource usage, which belongs to structured numerical time series data, reflecting the real-time change of system performance.

[0022] It also includes call chain data, which includes the description of the propagation path of requests between components in a distributed system, with time sequence and graph structure characteristics.

[0023] As an optional implementation manner of the present embodiment, preprocessing the obtained multi-modal operation and maintenance data includes: preprocessing the obtained multi-modal operation and maintenance data to ensure uniform feature dimension d after cleaning, desensitization, time alignment and embedding, to obtain a feature sequence, wherein the multi-modal operation and maintenance data includes log data, property index data, and call chain data; the feature sequence includes a log feature sequence log_feats∈R^{B×L×d}, wherein B is the batch size, and L is the length of the log feature sequence; an index feature sequence metric_feats∈R^{B×M×d}, wherein M is the length of the index feature sequence; and a call chain feature sequence trace_feats∈R^{B×T×d}, wherein T is the length of the call chain feature sequence.

[0024] The preprocessing of multi-modal operation and maintenance data can include the following: for log data, using a log parser (based on rules or lightweight ML) to clean and structure the logs, and converting unstructured text into an analyzable format; then using a pre-trained language model (such as S-BERT) to map the log text into a semantic vector (a method of converting text into a high-dimensional numerical vector representation); at the same time, extracting timestamps, component names, error codes, event descriptions, and other key information for subsequent processing.

[0025] For performance indicator data, collect various performance indicators of the system (such as CPU utilization, memory occupancy, network traffic, etc.), and perform standardization, normalization, and other processing on the indicator sequence; calculate statistical features (mean, peak, etc.) according to the time sequence characteristics; and extract the indicator snapshot corresponding to the log time to provide alignment basis for fusion analysis.

[0026] For behavior or call chain data, collect distributed tracking data or operation records, extract request ID, timestamp, and component call relationship; represent the call flow as a sequence or subgraph; for complex large systems, filter out key call chains to reduce computational complexity. For each type of data, the preprocessing module outputs the corresponding feature sequence tensor.

[0027] The corresponding data feature sequence obtained through preprocessing includes log feature sequence: log_feats∈R^{B×L×d}, where B is the batch size, L is the log sequence length, and d is the feature dimension.

[0028] Indicator feature sequence: metric_feats∈R^{B×M×d}, where M is the indicator sequence length.

[0029] Call chain feature sequence: trace_feats∈R^{B×T×d}, where T is the call chain sequence length (or the sequence length after subgraph pooling) In this step, after cleaning, desensitization, time alignment, and embedding (such as based on S-BERT, time series encoder, graph neural network, etc.) in preprocessing, the unified feature dimension d is ensured, so that subsequent fusion can be directly operated.

[0030] Log feature sequence: log_feats∈R^{B×L×d}, indicator feature sequence: metric_feats∈R^{B×M×d}, call chain feature sequence: trace_feats∈R^{B×T×d}, where the meanings of the parameters are as follows: batch: the number of samples processed in parallel, how many samples are input at a time; L / M / T: sequence length of each modality, different types of data have different lengths, L is the number of logs, M is the number of time points of indicators, and T is the number of nodes of call chains; d is the feature dimension: each sequence element (log / time point / call node) is encoded as a d-dimensional vector after the length. It is the "vector expression capacity" of semantic, timing, and graph information. Text log: use the sentence / embed model (such as sentence-BERT, lightweight Transformer) to directly output a d-dimensional vector, or output a higher dimension after linear projection to d; Indicator time point: use a time series encoder (1D-CNN, TCN, RNN, or a small MLP over sliding window statistics) to get a d-dimensional vector. Call chain node: use a graph neural network (GNN) or node encoding (node2vec) to get a d-dimensional node vector, or serialize the call chain and use a sequence encoder to get d.

[0031] Exemplarily, batch = 32 (32 samples are processed at a time); L = 50 (each sample has 50 logs); M = 100 (each sample has 100 indicator points); T = 20 (each sample has 20 nodes of call chain); d = 64 (each log / each indicator / each call chain node is represented by a 64-dimensional vector). Then the log feature sequence: [32, 50, 64]→32 samples, each sample has 50 logs, and each log is a 64-dimensional vector; the indicator feature sequence: [32, 100, 64]→32 samples, each sample has 100 time points, and each point is a 64-dimensional vector; the call chain feature sequence: [32, 20, 64]→32 samples, each sample has 20 call chain nodes, and each node is a 64-dimensional vector.

[0032] Step 102: fuse the data feature sequences corresponding to different modality operation and maintenance data with the same feature dimension, wherein the data feature sequences are first fused in the feature dimension when being fused; secondly fused based on a cross-attention mechanism; thirdly fused based on an encoder coding mode; and the weights corresponding to different fusion results are calculated based on a preset network, and the final fusion vector is obtained based on the weights.

[0033] In this step, at the first fusion, the pre-processed feature sequence of each data is first respectively spatially summarized (such as taking the mean along the sequence dimension), to obtain three [batch, d] vectors, and then they are directly spliced in the feature dimension to form a [batch, 3d] vector as the fusion output. If it is necessary to maintain a fixed dimension, it can be converted to [batch, d] through a fully connected mapping. This fusion method is simple and fast, but after direct splicing of each information, there is a lack of deep interaction, and it is difficult to automatically mine the internal relationship between modalities.

[0034] In the second fusion, a cross-attention mechanism is used to let different modal features "look" at each other. For example, the log feature sequence is taken as the Query, and the indicator feature sequence is taken as the [key, value], and the multi-head attention output between them is calculated to obtain the fusion representation of the log to the indicator. Taking the average of the output along the sequence dimension, a [batch, d] vector is obtained. Similarly, cross-attention calculation is performed for indicators to call chains, call chains to logs, etc. Finally, the vectors of these cross-attention outputs are combined (for example, added) to obtain the final fusion result [batch, d]. This way makes the information between different data start to intercommunicate, and can more accurately capture the key signals in each modality, but the computational complexity is increased compared to simple splicing.

[0035] In the third fusion, the feature sequences of logs, indicators and call chains are first spliced in the time / sequence dimension to form a sequence of length L+M+T (a modality type label can be added in front of each sequence); then the large sequence is input into a multi-layer Transformer encoder for overall processing. After encoding, the first label ([CLS]) or the corresponding aggregated vector in the encoding output can be taken as the fusion output [batch, d]. This way "strings together" all data information into a text for the model to understand together, and can capture deeper implicit relationships, but has a higher demand for computing power.

[0036] Further, a routing mechanism is introduced to automatically select or weight different fusion strategies according to the current scene. The specific method is as follows: first, the feature sequences of each modality are summarized (such as taking the mean), and they are connected into a [batch, 3d] gated input vector; then a small fully connected network is used to generate the weight distribution of each fusion "expert" strategy (early, middle, late); each expert outputs a [batch, d] vector, and finally the expert outputs are weighted and summed according to the gating weight to obtain the final fusion result [batch, d]. This method combines the advantages of multiple fusion methods, can use simple fusion when the data volume is large, and switch to deep fusion when the fault is complex, improving flexibility, but the design and parameter tuning are more complex.

[0037] As an optional implementation manner of the embodiment, the first fusion on the feature dimension of the data feature sequence when fusing includes performing spatial aggregation on each obtained feature sequence to obtain three [batch, d] vectors, i.e., a vector x_log corresponding to the log feature sequence, a vector x_metric corresponding to the metric feature sequence, and a vector x_trace corresponding to the call chain feature sequence; and performing concatenation on the vectors in the feature dimension to obtain a [batch, 3d] vector, i.e., fused_raw = concat(x_log, x_metric, x_trace) ∈ R^{B×3d}, wherein, for the output dimension d, the [batch, d] is converted by full connection mapping, i.e., fused = W·fused_raw + b ∈ R^{B×d}, wherein W ∈ R^{d×3d}.

[0038] In the optional implementation manner, the implementation step can include: a) performing pooling (average pooling or maximum pooling) on the log sequence along the sequence dimension to obtain x_log ∈ R^{B×d}. b) performing the same processing on the metric sequence to obtain x_metric ∈ R^{B×d}; performing the same processing on the call chain to obtain x_trace ∈ R^{B×d}. c) concatenating the three in the feature dimension: fused_raw = concat(x_log, x_metric, x_trace) ∈ R^{B×3d}. d) if the output dimension is still d, performing linear mapping or projection: fused = W·fused_raw + b ∈ R^{B×d}, wherein b is a bias size (obtained by initialization and training learning), and W ∈ R^{d×3d}.

[0039] Exemplarily, it is assumed that a log sequence has been well encoded by a model in the previous processing, each log is a 64-dimensional vector (a feature obtained by embedding), there are 100 logs in a log sequence, and therefore the feature matrix of the sequence is 100×64. If there are 32 samples processed together (batch = 32), it is [32, 100, 64].

[0040] Perform average pooling or maximum pooling on each sequence (L dimension = 100, i.e., there are 100 logs in a sequence). Average pooling (mean pooling, taking the average of 100 log vectors in each dimension): 1st dimension: take the average of the 1st number of the 100 vectors; Dimension 2: average the 2nd number of the 100 vectors; … up to Dimension 64.

[0041] Max pooling (for each dimension, take the max of the 100 values): Dimension 1: take the max of the 100 numbers; Dimension 2: take the max of the 100 numbers; … up to Dimension 64.

[0042] Regardless of whether average pooling or max pooling is used, the result is a new 1x64 vector, which represents the "average features" of the entire log sequence.

[0043] Suppose the scenario that there are 3 logs, and 3 features are extracted from each log: request duration, error code (yes / no), and CPU usage. The following table shows the features of the 3 logs: Average pooling logic: Duration: (100+200+300) / 3 = 200 Error code: (0+1+0) / 3 = 0.33 CPU: (30+50+80) / 3 = 53.3, resulting in the compressed 1x3 vector: [200, 0.33, 53.3] Similarly, the above steps, metrics_feats and trace_feats also get [32, 64] respectively. Concatenate the three types of features in d dimensions: [32, 64] + [32, 64] + [32, 64] -> [32, 194].

[0044] As an optional implementation manner of the embodiment, the second fusion based on the cross-attention mechanism comprises: calculating the multi-head attention between any two feature sequences; pooling the multi-head attention along the sequence dimension; combining the pooled vectors according to a specified strategy to obtain the result of the second fusion.

[0045] As an optional implementation manner of the embodiment, the multi-head attention between the two feature sequences comprises: taking one of the two feature sequences as Query and the other as [key, value] to calculate the multi-head attention; wherein, taking the log feature sequence as Query and the index feature sequence as [key, value] to calculate the multi-head attention, attn_log_metric∈R^{B×L×d} is obtained, and v_log_metric∈R^{B×d} is obtained by pooling along the sequence dimension; taking the index feature sequence as Query and the call chain feature sequence as [key, value] to calculate the multi-head attention, attn_metric_trace is obtained, and v_metric_trace is obtained by pooling along the sequence dimension; taking the call chain feature sequence as Query and the log feature sequence as [key, value] to calculate the multi-head attention, attn_trace_log is obtained, and v_trace_log is obtained by pooling along the sequence dimension.

[0046] As an optional implementation manner of the embodiment, the combined vector after the pooling is combined according to a specified strategy to obtain the second fusion result, which comprises: combining the multi-head attentions according to the strategy to obtain the final medium-term fusion output fused = combine(v_log_metric, v_metric_trace, v_trace_log)∈R^{B×d}, wherein the strategy comprises addition, concatenation and projection, or gated weighting.

[0047] In the above optional implementation manner, the implementation step comprises: a) taking the log sequence as Query and the index sequence as [key, value] to calculate the multi-head cross-attention to obtain attn_log_metric∈R^{B×L×d}; and pooling along the sequence dimension to obtain v_log_metric∈R^{B×d}. b) Similarly, attn_metric_trace (index to call chain) and attn_trace_log (call chain to log) are calculated, and v_metric_trace and v_trace_log are respectively pooled. c) The multi-path attention output is combined according to the strategy (for example, addition, concatenation and projection, or gated weighting) to obtain the final medium-term fusion output fused = combine(v_log_metric, v_metric_trace, v_trace_log)∈R^{B×d}.

[0048] As an optional implementation manner of the embodiment, the third fusion based on the encoding manner of the encoder includes concatenating the log feature sequence log_feats, the metric feature sequence metric_feats, and the trace feature sequence trace_feats at the sequence dimension to obtain seq∈R^{B×(L+M+T)×d}; adding the modal type embedding and the time and position embedding to each obtained sequence segment; sending the seq with the added embedding into a multi-layer Transformer encoder to obtain an encoding output out_seq∈R^{B×(L+M+T)×d}; and aggregating the encoding output to obtain a vector fused∈R^{B×d} of the third fusion.

[0049] In the optional implementation manner, the specific implementation steps can include: a) concatenating log_feats, metric_feats, and trace_feats at the sequence dimension, that is, splicing the three vector segments log_feats, metric_feats, and trace_feats into a long vector to obtain seq∈R^{B×(L+M+T)×d}. b) adding the modal type embedding (for example, 0 represents a log, 1 represents a metric, and 2 represents a call chain) and the time / position embedding to each sequence segment log_feats, metric_feats, and trace_feats to retain the source and time information. c) sending seq into a multi-layer Transformer encoder (preferably, N layers, and multi-head attention and Feed-Forward in each layer) to obtain an encoding output out_seq∈R^{B×(L+M+T)×d}. d) aggregating the encoding output (for example, taking out_seq[:,0,:] as a CLS vector, or performing intra-modal average / attention pooling) to obtain a fusion vector fused∈R^{B×d}.

[0050] As an optional implementation manner of the embodiment, the weight corresponding to each different fusion result is calculated based on a preset network, and the final fused vector is obtained by fusion based on the weight, which includes: the log feature sequence, the index feature sequence, and the call chain feature sequence are respectively summarized; the summarized feature sequences are spliced into a gate input vector gate_input = concat(x_log, x_metric, x_trace) ∈ R^{B×3d}; weights corresponding to different fusion manners are calculated based on the gate input vector weights = softmax(Gate(gate_input)) ∈ R^{B×E}, wherein E is the number of experts; and the final fused vector fused ∈ R^{B×d} is obtained by weighted summation based on the weights and the first fusion result, the second fusion result, and the third fusion result expert_i_output ∈ R^{B×d}, wherein i = 1..E. In the optional implementation manner, a routing mechanism is introduced, and different fusion strategies are automatically selected or weighted according to the current scene, which can include the following steps: a) Statistically summarize each modality first (for example, x_log = pool(log_feats), x_metric = pool(metric_feats), and x_trace = pool(trace_feats), all of which are R^{B×d}).

[0051] b) Splice the three into a gate input vector gate_input = concat(x_log, x_metric, x_trace) ∈ R^{B×3d}.

[0052] C) Calculate the expert weight distribution through a small fully connected network or a gate network Gate(·): weights = softmax(Gate(gate_input)) ∈ R^{B×E}, wherein E is the number of experts (for example, E = 3 corresponds to early / mid / late).

[0053] d) Each expert generates an output expert_i_output ∈ R^{B×d} (i = 1..E) in parallel.

[0054] e) Weighted summation according to the weight: fused = Σ_i weights[:,i:i+1] * expert_i_output, to obtain the final fused vector fused ∈ R^{B×d}.

[0055] Exemplarily, the features such as logs, indicators, call chains and the like are fused (concat, pooling, linear mapping and the like) to obtain a fused result fused_input. The fused result fused_input is sent in parallel to multiple expert networks (E experts, for example, early / middle / late three kinds of fusers). Each expert network respectively generates a respective output; then a gate generates a set of weights [w1..wE] for each sample (one complete data input into the model at a time). Finally, the expert outputs are weighted and summed according to these weights to obtain a fused result.

[0056] For example, there is a sample 1: d = 2, E = 3; Expert output: expert1_output = [1, 1] expert2_output = [2, 2] expert3_output = [3, 3] The weights given by the gate: weights = [0.1, 0.3, 0.6] Then finally: fused=0.1*[1,1]+0.3*[2,2]+0.6*[3,3]=[2.5,2.5]。

[0057] Through the above multi-level fusion strategy, the system can be from simple splicing to deep interaction to overall coding, and finally realize the scene adaptive hybrid strategy, effectively associate log, indicator and call chain information, and ensure efficient and accurate discovery of potential problems and mining of causal relationships in various operation scenarios.

[0058] Through a unified multi-modal fusion strategy, log, indicator and call chain information are interacted and fused in the same framework to realize cross-source association and comprehensive analysis. A hybrid architecture combining splicing, cross-attention and Transformer three-stage fusion technology covers different depth fusion needs. In the call chain modeling enhanced by graph neural network, the distributed call stack is converted into a subgraph and embedded using a graph neural network to realize accurate semantic expression of complex call processes. A routing mechanism is introduced to automatically select a fusion sub-module according to different scenarios, taking into account the calculation efficiency and analysis accuracy Step 103: input the processed fusion vector into the fine-tuned large model to output a semantic reasoning result; parse the semantic reasoning result into structured data; perform secondary verification on the abnormal events in the structured data to determine the final fault information, wherein the fault information is represented in a structured form.

[0059] In this step, the fused multi-source information is input into the large model in the form of a well-designed prompt for semantic reasoning. The prompt construction module generates text instructions based on specific tasks such as fault diagnosis, causal analysis, etc. to guide the large model to identify key entities, events, and their causal relationships. For example, a prompt can be designed as follows: "Please read the following log and performance indicator information and identify the key entities and events and their causal relationships". Then, the preprocessed fused features are converted into corresponding text or structured prompts and input into the model together with the original data. The selected large model can be a general large-scale language model (such as GPT series, Qwen, etc.) or a specialized model fine-tuned for operation and maintenance scenarios. In addition, the Retrieval-Augmented Generation (RAG) technology can be combined to introduce external knowledge base information to improve reasoning effectiveness. In the operation and maintenance scenario, entity recognition, anomaly detection, and causal reasoning are unified into a prompt pipeline to improve the consistency and efficiency of the output.

[0060] After the large model outputs, the results are post-processed: the text or structured answers generated by the model are parsed back into a structured form (such as identified fault entities, event summaries, and causal chain relationships); these information is stored in a knowledge graph or report system; combined with threshold or pre-set rules, the abnormal events are verified again; finally, the analyzed fault entities, event summaries, and causal relationships are fed back to the operation and maintenance personnel; in scenarios with automated execution capabilities, the analysis results can be further transmitted to the automated repair system for decision-making and handling.

[0061] The automatic semantic abstraction capability of large models on text and time series data reduces the dependence on manual feature extraction rules, significantly reducing system maintenance costs. Large models have built-in deep semantic understanding and reasoning capabilities, which can automatically extract high-level entities and events and infer causal chains, improving diagnostic accuracy. Combined with Prompt engineering and a small amount of fine-tuning technology, and with the integration of Retrieval-Augmented Generation (RAG), it supports rapid adaptation to new scenarios without the need to rewrite a large number of rules or retrain models. Through multi-task Prompt design, the collaborative output of entity recognition, anomaly detection, and causal reasoning is achieved, and multiple operation and maintenance requirements can be met with one reasoning.

[0062] The embodiment utilizes the powerful semantic understanding and generation capability of the large model to realize fully automated processing from data access to entity extraction, event correlation and causal inference, without the need for manual definition of a large number of rules or manual features, thereby significantly improving operation and maintenance efficiency. The large model is pre-trained on a large amount of corpus and has good generalization performance across systems and scenarios. Even when faced with new application systems or data formats, only a small amount of fine-tuning is needed without the need to redesign and develop special algorithms. The natural language output of the large model can simultaneously support multiple operation and maintenance tasks such as event detection, fault location and root cause report generation, avoiding the limitations of traditional single-model single-task, and being able to output multiple results at a time.

[0063] Reference Figure 2 By processing multi-source information through a unified model, the events described in the logs can be effectively correlated with the corresponding indicator fluctuations, and potential causal relationships can be mined. For example, when an error prompt appears in a log record, the model can automatically correlate it with a previous indicator anomaly (such as a CPU surge) and infer a possible causal chain.

[0064] Multi-source data fusion combined with large model reasoning enables the storage, retrieval and analysis of massive heterogeneous operation and maintenance data to balance real-time performance and accuracy. Relying on the reasoning capability of the large model, the system can provide timely problem analysis results, speed up fault processing and ensure service quality.

[0065] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0066] According to the embodiment of the application, a multi-modal operation and maintenance data system based on a large model is also provided, comprising a preprocessing unit configured to preprocess acquired multi-modal operation and maintenance data to obtain data feature sequences corresponding to different modal operation and maintenance data and having the same feature dimension; a fusion unit configured to fuse the data feature sequences corresponding to different modal operation and maintenance data and having the same feature dimension, wherein the data feature sequences are first fused in the feature dimension during fusion; secondly fused based on a cross-attention mechanism; thirdly fused based on an encoder coding method; weights corresponding to different fusion results are calculated based on a preset network, and a final fusion vector is obtained based on the weights; a prediction unit configured to input the fusion vector after processing to a fine-tuned large model, output semantic reasoning results, parse the semantic reasoning results into structured data, and perform secondary verification on abnormal events in the structured data to determine final fault information, wherein the fault information is represented in a structured form.

[0067] According to an embodiment of the present application, the present application further provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the method described in any of the above embodiments.

[0068] According to an embodiment of the present application, the present application further provides a readable storage medium, which stores computer instructions for enabling a computer to implement the method described in any of the above embodiments when the computer executes the computer instructions.

[0069] According to an embodiment of the present application, the present application further provides a computer program product, which enables the method described in any of the above embodiments when executed by a processor.

[0070] Figure 3 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices.

[0071] As shown in Figure 3 The electronic device 300 includes a computing unit 301 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 302 or a computer program loaded into a random access memory (RAM) 303 from a storage unit 308. Various programs and data required for the operation of the electronic device 300 can also be stored in the RAM 303. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0072] Various components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, a speaker, etc.; a storage unit 308, such as a magnetic disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0073] The computing unit 301 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs various methods and processes described above, such as the object matching method. For example, in some embodiments, the object matching method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded onto the RAM 303 and executed by the computing unit 301, one or more steps of the methods described above can be performed.

[0074] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0075] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0076] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine- readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A large model-based multi-modal operation and maintenance data fault determination method, characterized in that, The method comprises the following steps: Preprocessing the obtained multi-modal operation and maintenance data to obtain data feature sequences corresponding to different modal operation and maintenance data with the same feature dimension; Fusing the data feature sequences corresponding to different modal operation and maintenance data with the same feature dimension, wherein the first fusion is performed on the feature dimension of the data feature sequences during the fusion; the second fusion is performed based on the cross-attention mechanism; the third fusion is performed based on the encoding mode of the encoder; the weights corresponding to different fusion results are calculated based on the preset network, and the final fusion vector is obtained based on the weights; Processing the fusion vector and inputting it into the fine-tuned large model to output a semantic reasoning result; parsing the semantic reasoning result into structured data; performing secondary verification on the abnormal events in the structured data to determine the final fault information, wherein the fault information is represented in a structured form.

2. The large model-based multi-modal operation and maintenance data fault determination method according to claim 1, characterized in that, The preprocessing of the obtained multi-modal operation and maintenance data comprises: cleaning, desensitizing, time aligning and embedding the obtained multi-modal operation and maintenance data to ensure a unified feature dimension d and obtain feature sequences, wherein the multi-modal operation and maintenance data comprises log data, property index data and call chain data; the feature sequences comprise a log feature sequence log_feats∈R^{B×L×d}, wherein B is the batch size, and L is the length of the log feature sequence; an index feature sequence metric_feats∈R^{B×M×d}, wherein M is the length of the index feature sequence; and a call chain feature sequence trace_feats∈R^{B×T×d}, wherein T is the length of the call chain feature sequence.

3. The large model-based multi-modal operation and maintenance data fault determination method according to claim 2, characterized in that, The first fusion of the data feature sequences in the feature dimension during the fusion comprises spatially aggregating the obtained feature sequences to obtain three [batch, d] vectors, i.e., a vector x_log∈R^{B×d} corresponding to the log feature sequence, a vector x_metric∈R^{B×d} corresponding to the index feature sequence, and a vector x_trace∈R^{B×d} corresponding to the call chain feature sequence; The vectors are spliced in the feature dimension to obtain a [batch, 3d] vector, i.e., fused_raw = concat(x_log, x_metric, x_trace)∈R^{B×3d}, wherein, for the output dimension d, it is converted into [batch, d] by full connection mapping, i.e., fused = W·fused_raw + b∈R^{B×d}, wherein b is the bias size, and W∈R^{d×3d}.

4. The large model-based multi-modal operation and maintenance data fault determination method according to claim 2, characterized in that, The second fusion based on the cross-attention mechanism comprises: calculating the multi-head attention between any two feature sequences; and pooling the multi-head attention along the sequence dimension; The pooled vectors are combined according to a specified strategy to obtain the result of the second fusion.

5. The large model-based multi-modal operation and maintenance data fault determination method according to claim 4, characterized in that, The multi-head attention between any two feature sequences comprises: calculating the multi-head attention with one of the two feature sequences as Query and the other as a key-value pair [key, value]; Wherein, the multi-head attention is calculated with the log feature sequence as Query and the metric feature sequence as a key-value pair [key, value], obtaining attn_log_metric∈R^{B×L×d}, and the pooling along the sequence dimension obtains v_log_metric∈R^{B×d}; The multi-head attention is calculated with the metric feature sequence as Query and the call chain feature sequence as [key, value], obtaining attn_metric_trace, and the pooling along the sequence dimension obtains v_metric_trace; The multi-head attention is calculated with the call chain feature sequence as Query and the log feature sequence as [key, value], obtaining attn_trace_log, and the pooling along the sequence dimension obtains v_trace_log.

6. The large model-based multi-modal operation and maintenance data fault determination method according to claim 5, characterized in that, The combined vectors are combined according to the specified strategy to obtain the second fusion result, which comprises: combining each multi-head attention according to the strategy to obtain the final medium-term fusion output fused = combine(v_log_metric, v_metric_trace, v_trace_log)∈R^{B×d}, wherein, The strategy includes addition, concatenation and projection, or gated weighting.

7. The large model-based multi-modal operation and maintenance data fault determination method according to claim 2, characterized in that, The third fusion based on the encoding mode of the encoder comprises: The log feature sequence log_feats, the metric feature sequence metric_feats and the scheduling chain feature sequence trace_feats are concatenated at the beginning and end in the sequence dimension to obtain seq∈R^{B×(L+M+T)×d}; Add modal type embedding and time and position embedding to each sequence segment obtained; The seq with added embedding is sent to a multi-layer Transformer encoder to obtain an encoding output out_seq∈R^{B×(L+M+T)×d}; The encoding output is aggregated to obtain a third fusion vector fused∈R^{B×d}.

8. The large model-based multi-modal operation and maintenance data fault determination method according to claim 1, characterized in that, The preset network is used to calculate weights corresponding to different fusion results, and the weights are used for fusion to obtain a final fusion vector, including: the log feature sequence, the index feature sequence, and the call chain feature sequence are summarized respectively; the summarized feature sequences are spliced into a gate input vector gate_input = concat(x_log, x_metric, x_trace)∈R^{B×3d}; weights corresponding to different fusion modes are calculated based on the gate input vector weights = softmax(Gate(gate_input))∈R^{B×E}, wherein E is the number of experts; the weights, the first fusion result, the second fusion result, and the third fusion result expert_i_output∈R^{B×d} are weighted and summed to obtain a final fusion vector fused =Σ_iweights[:,i:i+1] * expert_i_output, wherein fused∈R^{B×d} and i = 1..E.

9. The large model-based multi-modal operation and maintenance data fault determination method according to claim 1, characterized in that, During the fine-tuning of the large model, a sample fusion vector after the fusion of the sample is obtained; The sample fusion vector and the pre-constructed prompt are input into the large model to fine-tune the large model; The semantic reasoning result is parsed into structured data, including: the semantic reasoning result is parsed into recognized fault entities, event summaries, and causal chain relationships.

10. A large model-based multi-modal operation and maintenance data system, characterized in that, Including: A preprocessing unit is configured to preprocess the obtained multi-modal operation and maintenance data to obtain data feature sequences corresponding to different modal operation and maintenance data and having the same feature dimension; A fusion unit is configured to fuse the data feature sequences corresponding to different modal operation and maintenance data and having the same feature dimension, wherein the data feature sequences are first fused in the feature dimension during the fusion; second fusion is performed based on a cross-attention mechanism; third fusion is performed based on an encoder encoding method; weights corresponding to different fusion results are calculated based on a preset network, and a final fusion vector is obtained based on the weights; a prediction unit is configured to input the fusion vector into a fine-tuned large model after processing, and output a semantic reasoning result; the semantic reasoning result is parsed into structured data; secondary verification is performed on abnormal events in the structured data to determine final fault information, wherein the fault information is represented in a structured form.

Images