An AI-based method and system for retrieving stored logs

Abstract

This invention relates to the field of data storage and intelligent operation and maintenance technology, and discloses an AI-based storage log retrieval method and system. The method includes: acquiring storage log data and query input; using a pre-trained semantic mapping model to extract features and construct a graph from the storage log data to obtain a semantic association graph; using a pre-trained language processing model to extract features from the query input to obtain a query vector; calculating cosine similarity and performing topological expansion based on the query vector and the feature vectors of each node in the semantic association graph to obtain an intent-related graph; processing the intent-related graph to obtain log reconstruction paths and optimized retrieval paths; performing density clustering on the optimized retrieval paths and combining them with the query vector to obtain time-related paths; extracting candidate entries based on the time-related paths and aggregating them to obtain merged entries; and reformatting the results to obtain a retrieval report. This method can achieve high-precision semantic retrieval of complex logs.

Description

Technical Field

[0001] This invention relates to the field of data storage and intelligent operation and maintenance technology, and in particular to an AI-based storage log retrieval method and system. Background Technology

[0002] In the information age, log storage and retrieval are crucial for ensuring stable system operation and quickly locating the root cause of faults. As systems grow in scale and business complexity increases, log data carries massive amounts of operational information, making the efficient extraction of valuable data a critical industry requirement.

[0003] In existing technologies, log data retrieval and filtering typically rely on fixed rules or simple keyword matching. This method, based on fixed rules or surface text matching, struggles to adapt to the dynamic changes and diverse expressions hidden within log data. Log content often contains a large amount of unstructured text, harboring complex business logic and temporal relationships. Without delving into the deeper information behind the logs, it's impossible to accurately determine the correlation between different log entries. This leads to the system's inability to accurately understand the user's true intent when faced with complex query requirements across systems and time periods, easily returning a large amount of irrelevant and redundant information or missing crucial clues. To overcome the limitations of traditional keyword matching, the industry urgently needs to introduce semantic retrieval mechanisms to achieve contextual logic mapping and noise reduction.

[0004] Existing technologies suffer from inaccurate matching of complex query intents and low retrieval accuracy when dealing with massive amounts of unstructured logs. Summary of the Invention

[0005] This invention provides an AI-based storage log retrieval method and system to solve the problems of inaccurate matching of complex query intents and low retrieval accuracy in the existing technology for massive unstructured logs.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides an AI-based method for retrieving stored logs, comprising:

[0007] Obtain stored log data, perform semantic mapping processing on the stored log data to obtain feature vectors, and perform graph construction processing based on the feature vectors to obtain a semantic association graph;

[0008] Obtain query input, extract features from the query input using a pre-trained language processing model to obtain a query vector, perform relevance retrieval processing on the query vector and the semantic association graph to obtain intent candidate nodes, and perform topology expansion processing on the intent candidate nodes to obtain an intent relevance graph;

[0009] The intent-related graph is subjected to node value evaluation and noise reduction processing to obtain a log denoised subgraph, and the log denoised subgraph is subjected to graph traversal processing to obtain the log reconstruction path.

[0010] An optimized retrieval path is obtained by performing intent path optimization processing based on the query vector and the log reconstruction path;

[0011] The optimized retrieval path is subjected to time series analysis to obtain time series features. Based on the time series features, the path clusters are divided into clusters. The time-related paths are obtained by calculating the similarity between the path clusters and the query vector.

[0012] The pre-trained language processing model is used to perform semantic evaluation on the time-related path to obtain candidate entries, the original log text is obtained, and fragment aggregation is performed on the candidate entries and the original log text to obtain merged entries;

[0013] The merged entries are completed using a preset field mapping table to obtain structured data, and the structured data is reformatted using a preset output template to obtain a retrieval report.

[0014] Secondly, the present invention provides an AI-based storage log retrieval system, comprising:

[0015] The graph construction module is used to acquire stored log data, perform semantic mapping processing on the stored log data to obtain feature vectors, and perform graph construction processing on the feature vectors to obtain a semantic association graph.

[0016] The subgraph extraction module is used to obtain query input, extract features from the query input using a pre-trained language processing model to obtain a query vector, perform relevance retrieval processing on the query vector and the semantic association graph to obtain intent candidate nodes, and perform topology expansion processing on the intent candidate nodes to obtain an intent-related graph.

[0017] The path denoising module is used to perform node value evaluation and denoising processing on the intent-related graph to obtain a log denoised subgraph, and to perform graph traversal processing on the log denoised subgraph to obtain the log reconstruction path.

[0018] The path optimization module is used to perform intent path optimization processing based on the query vector and the log reconstruction path to obtain an optimized retrieval path;

[0019] The temporal clustering module is used to perform temporal analysis on the optimized retrieval path to obtain temporal features, perform clustering based on the temporal features to obtain path clusters, and calculate the time-related path based on the similarity between the path clusters and the query vector.

[0020] The semantic aggregation module is used to perform semantic evaluation on the time-related path using the pre-trained language processing model to obtain candidate entries, acquire the original log text, and perform fragment aggregation on the candidate entries and the original log text to obtain merged entries.

[0021] The report generation module is used to complete the data of the merged entries using a preset field mapping table to obtain structured data, and to reorganize the structured data using a preset output template to obtain a retrieval report.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) This invention obtains stored log data and uses pre-trained models to extract features to construct a semantic association graph. Then, it calculates similarity and expands the topology based on the query vector to obtain an intent-related graph. This changes the traditional shallow retrieval method that relies on fixed rules or simple keyword matching. It transforms discrete unstructured log text into a graph structure network containing contextual association and global logic, enabling the system to deeply understand and map the true semantics of complex queries. This improves the intent matching accuracy of massive logs in complex cross-system query scenarios and effectively avoids irrelevant information flooding or key clues being missed due to keyword stuffing.

[0024] (2) This invention uses a random walk algorithm to estimate the betweenness centrality of nodes in the intent correlation graph, and removes nodes with low hub value based on the betweenness centrality to obtain the log denoising subgraph and log reconstruction path. It introduces an objective topology evaluation mechanism at the graph theory level, which can automatically identify and remove edge companion nodes that do not contribute to the propagation of fault logic in the complex log correlation network, such as noise reported in normal state, and extract the core trunk road that leads directly to the root cause. This greatly reduces the computational overhead caused by invalid nodes and significantly improves the interpretability of the fault investigation link and the efficiency of root cause location.

[0025] (3) This invention extracts the temporal features of the optimized retrieval path for density clustering and calculates the cosine similarity between the semantic vector mean of the nodes in the cluster and the query vector. It cleverly integrates the dual verification mechanism of temporal behavior pattern and isomorphic feature space. It not only uses temporal clustering to aggregate log links with the same delay or fluctuation characteristics in physical dimension, but also completes semantic verification in the same rigorous mathematical latent space. It effectively avoids the dimensionality curse caused by cross-model space calculation, solves the problem of misjudgment that is easily caused by simply relying on time or semantics, and realizes dynamic log filtering with high physical fidelity. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the AI-based storage log retrieval method provided in the first embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the structure of an AI-based storage log retrieval system provided in the second embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Reference Figure 1 The first embodiment of the present invention provides an AI-based method for retrieving stored logs, including the following steps:

[0030] S11, Obtain stored log data, perform semantic mapping processing on the stored log data to obtain feature vectors, and perform graph construction processing based on the feature vectors to obtain a semantic association graph;

[0031] S12, obtain query input, use a pre-trained language processing model to extract features from the query input to obtain a query vector, perform relevance retrieval processing on the query vector and the semantic association graph to obtain intent candidate nodes, and perform topology expansion processing on the intent candidate nodes to obtain an intent relevance graph;

[0032] S13, perform node value evaluation and noise reduction processing on the intent-related graph to obtain a log denoised subgraph, and perform graph traversal processing on the log denoised subgraph to obtain a log reconstruction path;

[0033] S14, Based on the query vector and the log reconstruction path, perform intent path optimization processing to obtain an optimized retrieval path;

[0034] S15, perform time series analysis on the optimized retrieval path to obtain time series features, perform clustering based on the time series features to obtain path clusters, and calculate the time-related path based on the similarity between the path clusters and the query vector.

[0035] S16, use the pre-trained language processing model to perform semantic evaluation on the time-related path to obtain candidate entries, obtain the original log text, and perform fragment aggregation based on the candidate entries and the original log text to obtain merged entries;

[0036] S17, use a preset field mapping table to complete the data of the merged entries to obtain structured data, and use a preset output template to reorganize the structured data to obtain a retrieval report.

[0037] In step S11, storage log data is obtained, semantic mapping processing is performed on the storage log data to obtain feature vectors, and graph construction processing is performed based on the feature vectors to obtain a semantic association graph.

[0038] The process includes semantic mapping of the stored log data to obtain feature vectors, and graph construction based on the feature vectors to obtain a semantic association graph, including:

[0039] The target text features are obtained by extracting unstructured text features from the stored log data using a preset log parsing module.

[0040] The target text features are mapped using a pre-trained semantic mapping model to obtain an initial vector, and the initial vector is padded with zeros to obtain a feature vector.

[0041] The feature vectors are subjected to cosine similarity calculation to obtain a semantic similarity matrix, and an initial graph structure is obtained by constructing a graph based on the elements in the semantic similarity matrix that are greater than a preset similarity threshold.

[0042] The node weights are obtained by calculating the magnitude of the feature vectors, and the semantic association graph is obtained by defining the edge relationships of the initial graph structure based on the node weights and the semantic similarity matrix.

[0043] In one implementation, this embodiment utilizes a pre-defined log parsing module to extract unstructured text features from the stored log data to obtain target text features. The log parsing module in this embodiment employs a text matching algorithm based on regular expressions. This module reads the original stored log data line by line and performs data cleaning. During data cleaning, the module uses masked regular expressions to match all physical media access control addresses and authentication password fields, uniformly replacing the matched characters with fixed asterisk placeholders, and scanning and removing invisible format control characters such as carriage returns, line feeds, and tabs from the text. After cleaning, the module extracts the log level identifier and text message body of each log entry through keyword matching. The module compares the log level identifiers, removing log entries with identifiers at the regular information level, and extracting the text message body corresponding to log entries with identifiers at the warning or error level, identifying these as target text features.

[0044] It should be noted that in this embodiment, a pre-trained semantic mapping model is used to perform feature mapping on the target text features to obtain an initial vector, and the initial vector is then padded with zeros to obtain a feature vector. The pre-trained semantic mapping model adopts a bidirectional encoder to represent the model architecture. The pre-training process of this model is as follows: A training corpus is constructed by collecting historical anonymized offline log text. Based on the training corpus, a masked language model task mechanism is used to randomly mask 15% of the words in the input text, constructing a cross-entropy loss function with the goal of predicting the masked words. An adaptive moment estimation optimization algorithm is used to iteratively update the network weight parameters inside the model. During this process, the initial learning rate is set to 2e-5, the batch size to 32, and the number of iterations to 10, until the value of the cross-entropy loss function converges to a stable state.

[0045] In one implementation, during feature mapping, this embodiment inputs the target text features into the trained semantic mapping model, extracts the output sequence of the last hidden state of the model, and obtains an initial vector. This embodiment extracts the maximum input sequence length from the semantic mapping model's structural definition and determines it as the standard dimension value, such as 512 dimensions. This embodiment calculates the actual dimension of the initial vector. When the actual dimension is less than the standard dimension value, zeros are appended digit by digit to the end of the initial vector until the vector's dimension is strictly equal to the standard dimension value, completing the zero-padded process and obtaining a feature vector of uniform length.

[0046] In one implementation, this embodiment calculates the cosine similarity of the feature vectors to obtain a semantic similarity matrix, and constructs an initial graph structure based on the elements in the semantic similarity matrix that are greater than a preset similarity threshold. This embodiment extracts any two feature vectors, calculates the sum of the products of their corresponding dimensions to obtain a dot product, calculates the square root of the sum of the squares of all elements of these two feature vectors to obtain two moduli, and divides the dot product by the product of the two moduli to obtain the cosine similarity value between the two vectors. This embodiment arranges the cosine similarity values ​​of all pairs of feature vectors according to their row and column indices to generate a complete semantic similarity matrix.

[0047] It should be noted that the preset similarity threshold is determined using the percentile method in this embodiment. This embodiment collects a historical benchmark log dataset, calculates the cosine similarity values ​​of all benchmark log pairs, sorts this set of values ​​in ascending order, plots a cumulative distribution function curve, extracts the cosine similarity value corresponding to the 90th percentile of the cumulative distribution function curve, and sets this value as the preset similarity threshold. When constructing the graph structure, this embodiment defines each feature vector as an independent node in the graph, traverses each element in the semantic similarity matrix, and determines whether the cosine similarity value corresponding to that element is greater than the preset similarity threshold; if it is greater, an undirected edge is established between the corresponding two nodes to generate the initial graph structure.

[0048] It is worth noting that in this embodiment, the node weights are obtained by calculating the modulus of the feature vector, and the semantic association graph is obtained by defining the edge relationships of the initial graph structure based on the node weights and the semantic similarity matrix. To eliminate the text length dimension bias introduced by the aforementioned zero-value padding operation, this embodiment does not perform modulo operation on all elements, but instead performs root mean square (RMS) calculation. This embodiment extracts all non-zero valid elements from the feature vector, calculates the sum of squares of all non-zero valid elements, divides the sum of squares by the absolute number of non-zero valid elements to obtain the mean square value, and performs a square root operation on the mean square value to obtain the root mean square (RMS) value. This embodiment strictly determines the RMS value as the node weight of the corresponding graph node. Subsequently, this embodiment traverses all established undirected edges in the initial graph structure, extracts the cosine similarity value between corresponding two nodes from the semantic similarity matrix, multiplies the cosine similarity value by the geometric mean of the corresponding node weights, assigns it as the edge weight of the undirected edge, completes the edge relationship definition, and outputs a semantic association graph containing node weights and edge weights.

[0049] For example, the log parsing module extracts two target text features: disk sector read timeout and data block synchronization failure. These two texts are input into a pre-trained semantic mapping model, yielding initial vectors of 300 and 350 dimensions respectively. During zero-padded processing, 212 and 162 zeros are appended to the end of each vector, respectively, bringing them to 512 dimensions. The sum of squares of the non-zero valid elements of each vector is extracted, and the root mean square is calculated, yielding values ​​of 1.5 and 1.8, which are used as the node weights of the two nodes. Subsequently, the cosine similarity between the two feature vectors is calculated, yielding a value of 0.88. The preset similarity threshold, determined through historical statistics, is 0.82. Since 0.88 is greater than 0.82, an undirected edge is established between these two nodes in the initial graph structure, and 0.88 is set as the edge weight of this undirected edge, generating a semantic association graph.

[0050] In step S12, a query input is obtained, and a query vector is obtained by using a pre-trained language processing model to extract features from the query input. Based on the query vector and the semantic association graph, a relevance retrieval process is performed to obtain an intent candidate node, and the intent candidate node is subjected to topology expansion processing to obtain an intent association graph.

[0051] Specifically, the process of obtaining intent candidate nodes by performing relevance retrieval processing based on the query vector and the semantic association graph, and then performing topology expansion processing on the intent candidate nodes to obtain an intent relevance graph, includes:

[0052] A set of candidate nodes is obtained by calculating the cosine similarity between the query vector and the feature vectors of each node in the semantic association graph.

[0053] The initial subgraph region is obtained by expanding the candidate node set outward to adjacent nodes, and the association strength is obtained by calculating the edge weights within the initial subgraph region.

[0054] Edge structures with association strength lower than a preset association strength threshold within the initial subgraph region are removed to obtain a set of connected subgraphs;

[0055] A path integrity coverage assessment is performed on the connected subgraph set to obtain a diversity matching metric. Subgraph regions with diversity matching metrics greater than a preset matching threshold are merged to obtain an intent-related graph.

[0056] In one implementation, this embodiment obtains a query input containing unstructured operation and maintenance description text, and uses a pre-trained language processing model to extract features from the query input to obtain a query vector. The pre-trained language processing model adopts a bidirectional encoder network based on the Transformer architecture. The pre-training process of this model is as follows: An operation and maintenance-specific corpus is constructed by acquiring open-source IT operation and maintenance knowledge base text and historical system error work orders; a self-supervised learning task is constructed using a masked language model mechanism; during this training process, the AdamW adaptive moment estimation optimization algorithm with weight decay is used, with an initial learning rate of 5e-5, a weight decay coefficient of 0.01, and a batch size of 64; the model weight parameters are iteratively updated using the backpropagation algorithm, with a maximum iteration count of 20 rounds, or until the perplexity index of the model on the validation set converges to a preset constant. In this embodiment, the word segmentation sequence of the query input is input into the trained language processing model, and the last layer global representation feature vector is extracted and determined as the query vector. Subsequently, this embodiment extracts the feature vectors of each node in the semantic association graph, calculates the dot product between the query vector and the feature vector of each node, and divides the dot product result by the product of the magnitudes of the two vectors to obtain the cosine similarity value. This embodiment sorts the values ​​in descending order. Regarding the determination of the preset extraction threshold ratio, this embodiment collects node recall data from historical similar queries, plots a function curve showing the change in recall rate with the node extraction ratio, calculates and extracts the curvature inflection point corresponding to the point of maximum decrease in recall rate growth in the function curve, and objectively sets the percentage value corresponding to this curvature inflection point as the preset extraction threshold ratio. This embodiment extracts the nodes with the highest cosine similarity values ​​within this extraction threshold ratio to construct a candidate node set.

[0057] It should be noted that in this embodiment, an initial subgraph region is obtained by expanding the candidate node set outwards to adjacent nodes, and the edge weights within the initial subgraph region are calculated to obtain the association strength. This embodiment takes each node in the candidate node set as the starting point and performs a breadth-first search operation based on a preset search depth in the semantic association graph. Regarding the determination of the preset search depth, this embodiment statistically analyzes the set of shortest path lengths from the root cause node to the apparent error node in historical real faults, calculates the arithmetic mean of this set of shortest path lengths, rounds it up, and strictly sets the rounded value as the preset search depth.

[0058] In one implementation, this embodiment extracts all adjacent nodes covered on the search path and the edges connecting these nodes to generate an initial subgraph region. This embodiment extracts the edge weight values ​​of each edge in the initial subgraph region and directly determines these edge weight values ​​as the association strength. This embodiment calculates the arithmetic mean and standard deviation of all edge weights in the initial subgraph region, subtracts one time the standard deviation from the arithmetic mean, and sets the calculated difference as a preset association strength threshold. This embodiment traverses all edge structures in the initial subgraph region, compares the corresponding association strength with the preset association strength threshold, removes edge structures with association strengths lower than the threshold, and extracts the largest interconnected subgraph component remaining in the graph after removing edge structures, generating a connected subgraph set.

[0059] In one implementation, this embodiment performs path integrity coverage assessment on the connected subgraph set to obtain a diversity matching metric, and merges subgraph regions with diversity matching metric values ​​greater than a preset matching threshold to obtain an intent-related graph. This embodiment parses each full-link path in the connected subgraph set from a source node with an in-degree of zero to a terminal node with an out-degree of zero. This embodiment counts the number of independent node types contained in each full-link path and divides this number by the total number of node types defined globally by the system to calculate the path coverage rate. This embodiment sums the path coverage rates of all full-link paths within the same subgraph to obtain the diversity matching metric value of that subgraph. This embodiment collects historical manually confirmed faulty link data to construct a positive and negative sample set, calculates the true positive rate and false positive rate of model classification under different threshold values, plots the receiver operating characteristic curve, extracts the matching metric value corresponding to the maximum value of the Youden index, and objectively labels it as the preset matching threshold. This embodiment extracts all subgraph regions with diversity matching metric values ​​greater than this threshold, performs a union operation on the node set and edge set contained in them, and generates an intent-related graph.

[0060] For example, a query input such as write timeout is obtained. After feature extraction through a language processing model, it is compared with a semantic association graph, and nodes with extremely high cosine similarity are extracted to form a candidate node set based on an extraction threshold ratio determined by the inflection point of the historical recall curve, such as 8%. The system determines a preset search depth of two hops based on the historical average shortest path and expands outward to capture adjacent nodes to form an initial subgraph region. The average weight of all edges in this subgraph is calculated to be 0.6, and the standard deviation is 0.15, so the association strength threshold is determined to be 0.45. The system removes interfering edges, retains connected components, and generates a set of connected subgraphs. Subsequently, the diversity matching metric is calculated to be 0.82. This value is greater than the matching degree threshold determined based on the Youden index, such as 0.75, so the system merges the qualifying subgraphs and outputs an intent-related graph.

[0061] In step S13, the intent-related graph is subjected to node value evaluation and noise reduction processing to obtain a log denoised subgraph, and the log denoised subgraph is subjected to graph traversal processing to obtain a log reconstruction path, including:

[0062] Betweenness centrality is obtained by estimating the hub value of each node in the intent-related graph using a random walk algorithm;

[0063] Determine whether the betweenness centrality is lower than a preset structural threshold;

[0064] If the betweenness centrality is lower than a preset structural threshold, the corresponding node is removed from the intent-related graph to obtain a log denoising subgraph.

[0065] The log reconstruction path is obtained by performing a graph traversal on the log denoising subgraph.

[0066] In one implementation, this embodiment uses a random walk algorithm to estimate the hub value of each node in the intent correlation graph to obtain betweenness centrality. This embodiment employs an approximate walk mechanism based on Monte Carlo sampling. Regarding the basis for setting the preset total number of walks and the maximum single walk step size, this embodiment uses a dynamic adaptive mechanism to extract the total number of nodes and the network diameter of the intent correlation graph, i.e., the maximum value of the shortest path between any two nodes; this embodiment objectively sets the maximum single walk step size to the value of the network diameter, and sets the preset total number of walks to the product of the total number of nodes and a preset coverage constant, for example, set to 50 to ensure the stable convergence of the Markov chain distribution.

[0067] In one implementation, this embodiment constructs a node access counter in memory, with its initial value set to zero. In each traversal round, this embodiment randomly selects a node as the starting node in the intent-related graph, extracts all adjacent undirected edges of the current node and their corresponding edge weights, and divides the edge weight of each edge by the sum of the weights of all adjacent edges to obtain the state transition probability of each path. Based on this state transition probability, this embodiment randomly selects an edge to move to the next node, and increments the value of the next node in the access counter by one. The current round ends when the number of traversal steps reaches the maximum single traversal step size or when the endpoint node with an out-degree of zero is reached. After all traversal rounds have accumulated to the preset total number of traversals, this embodiment extracts the absolute frequency value recorded for each node in the access counter, divides this absolute frequency value by the preset total number of traversals, calculates a floating-point number between zero and one, and determines this floating-point number as the betweenness centrality of the corresponding node.

[0068] It should be noted that this embodiment determines whether the betweenness centrality is lower than a preset structural threshold, and if it is determined to be lower, the corresponding node is removed from the intent correlation graph to obtain a log denoising subgraph. Regarding the preset structural threshold, this embodiment uses the maximum inter-class variance algorithm for objective determination. This embodiment extracts the betweenness centrality values ​​of all nodes in the intent correlation graph and constructs a one-dimensional numerical distribution histogram with a fixed numerical interval step size. This embodiment traverses all possible split points in the histogram, dividing the node values ​​into high centrality and low centrality categories. This embodiment calculates the intra-class variance and inter-class variance for these two categories respectively. This embodiment compares the inter-class variance values ​​corresponding to all split points, extracts the optimal split point that maximizes the inter-class variance value, and strictly sets the betweenness centrality value corresponding to the optimal split point as the preset structural threshold. This embodiment reads the betweenness centrality of each graph node and compares it with the preset structural threshold. If the betweenness centrality of a node is lower than the threshold, this embodiment performs a topology modification operation, deleting the node along with all its directly connected edges from the intent correlation graph. After traversal is complete, the remaining graph structure is output as a log denoising subgraph.

[0069] In one implementation, this embodiment performs a graph traversal on the log denoising subgraph to obtain the log reconstruction path. This embodiment uses the topological sorting rules of a directed acyclic graph to hierarchically divide the log denoising subgraph. This embodiment scans all nodes in the log denoising subgraph, extracts all nodes with an in-degree of zero, and determines this as the root node for the search. This embodiment uses a depth-first search algorithm to traverse along the connecting edges of the log denoising subgraph. During the traversal, this embodiment uses a stack data structure to push each node along the path sequentially until a leaf node with an out-degree of zero is reached. At this point, the sequentially arranged node sequence in the stack is output as a complete path. Subsequently, this embodiment performs a backtracking operation, pops the top node from the stack, and explores other unvisited adjacent edges, repeating the above pushing and output process until all unidirectional node sequences from the source node to the destination node in the log denoising subgraph are exhaustively enumerated. These node sequences are then used to generate the log reconstruction path.

[0070] For example, in the intent-related graph, there exists a node whose main text is a regular state report. The system executes a Monte Carlo random walk algorithm, setting the preset total number of walks to 100,000. After the traversal, the access counter shows that the regular state report node has been visited 2,000 times. The system divides 2,000 by 100,000, calculating its betweenness centrality to be 0.02. The system uses the Otsu's inter-class variance algorithm to calculate the variance of the betweenness centrality distribution of the entire graph, finding the optimal split point value of 0.15 that maximizes the between-class variance, and sets it as the preset structure threshold. The system determines that 0.02 is lower than 0.15, performs a topology modification, completely removing the regular state report node and its connecting edges from the graph, resulting in a log denoising subgraph. Subsequently, the system performs a depth-first search and backtracking on the log denoising subgraph to extract the complete fault path after eliminating the redundant node, outputting the log reconstruction path.

[0071] In step S14, an optimized retrieval path is obtained by performing intent path optimization processing based on the query vector and the log reconstruction path, including:

[0072] The feature vectors of each node in the log reconstruction path are subjected to feature pooling to obtain the path representation vector.

[0073] A semantic matching score is obtained by performing vector space operations on the query vector and the path representation vector.

[0074] Highly relevant paths are obtained by extracting paths whose semantic matching scores are greater than a preset intent matching threshold;

[0075] Tracing the source log nodes mapped by the highly relevant paths, extracting the log timestamps and process call chains from the metadata associated with the source log nodes to obtain context dependency data, and then splicing and merging the highly relevant paths based on the context dependency data to obtain an optimized retrieval path.

[0076] In one implementation, this embodiment performs feature pooling on the feature vectors of each node in the log reconstruction path to obtain a path representation vector. This embodiment extracts the feature vectors of all nodes included in a single log reconstruction path, sums the element values ​​corresponding to the feature vectors of each node according to the feature dimension, and divides the sum by the total number of nodes included in the log reconstruction path to complete the average feature pooling process. This outputs a single feature vector with the same dimension as the node feature vectors, and this single feature vector is determined as the path representation vector of the log reconstruction path.

[0077] It should be noted that in this embodiment, a semantic matching score is obtained by performing vector space operations on the query vector and the path representation vector, and highly relevant paths are obtained by extracting paths whose semantic matching scores are greater than a preset intent matching threshold. In this embodiment, the query vector and the path representation vector are extracted, the sum of the products of their corresponding dimension elements is calculated to obtain a dot product, the vector magnitudes of both are calculated, the dot product is divided by the product of the two vector magnitudes, and the resulting cosine similarity value is determined as the semantic matching score.

[0078] It should be noted that, regarding the determination of the preset intent matching threshold, this embodiment collects manually labeled positive sample path sets and negative sample path sets from the historical operation and maintenance system. Semantic matching scores are calculated for each set, generating positive and negative probability density distribution curves. The intersection point of these two probability density distribution curves is extracted, and the score value corresponding to this intersection point is strictly set as the preset intent matching threshold. This embodiment compares the semantic matching score of each log reconstruction path with the preset intent matching threshold one by one. Log reconstruction paths with semantic matching scores lower than or equal to the preset intent matching threshold are eliminated, and paths with semantic matching scores greater than the preset intent matching threshold are retained, forming a highly relevant path set.

[0079] In one implementation, this embodiment traces the source log nodes mapped by the highly relevant path, extracts the log timestamps and process call chains from the metadata associated with the source log nodes to obtain context dependency data, and then concatenates and merges the highly relevant paths based on the context dependency data to obtain an optimized retrieval path. This embodiment traces the source physical logs corresponding to each node in the highly relevant path using a log parsing engine, reads the metadata fields associated with the source physical log node, extracts the log timestamp recording the physical occurrence time and the process call chain identifier (such as Trace ID) recording the global request number, and combines these two types of data to form context dependency data. This embodiment groups highly relevant paths with the same process call chain identifier into the same group. Within each group, all log nodes are sorted chronologically according to the order of the log timestamps. Adjacent nodes in the sorted node sequence are connected in a forward chronological order to generate a single, continuous log path that eliminates out-of-order behavior, and this path is output as the optimized retrieval path.

[0080] For example, for a log reconstruction path containing 5 log nodes, this embodiment extracts 512-dimensional feature vectors corresponding to these 5 nodes, calculates the arithmetic mean of these 5 feature vectors in each dimension, and generates a 512-dimensional path representation vector. Then, the cosine similarity between this path representation vector and the query vector is calculated, resulting in a semantic matching score of 0.82. The preset intent matching threshold, determined by the intersection of historical sample distributions, is 0.75. Since 0.82 is greater than 0.75, this embodiment retains this path as a highly relevant path. Further, the process call chain identifier of the nodes in this path is extracted as Req-9921, and the node sequence is rearranged according to log timestamps. The rearranged ordered node link is output as the optimized retrieval path.

[0081] In step S15, time-series analysis is performed on the optimized retrieval path to obtain time-series features. Based on the time-series features, path clusters are obtained by clustering. Based on the path clusters and the query vector, time-related paths are obtained by similarity calculation.

[0082] The time-series analysis of the optimized retrieval path yields time-series features, including:

[0083] The starting log node time of the optimized retrieval path is set as the baseline zero point, and the relative time offset of subsequent nodes relative to the baseline zero point is calculated to obtain the timestamp alignment information.

[0084] The temporal characteristics are obtained by calculating the rate of change of time difference between adjacent nodes based on the timestamp alignment information.

[0085] In one implementation, this embodiment sets the starting log node time of the optimized retrieval path as a baseline zero point, and calculates the relative time offset of subsequent nodes relative to the baseline zero point to obtain timestamp alignment information. This embodiment extracts the timestamp of the first node in the optimized retrieval path and marks it as zero. For the second node and all subsequent nodes in the optimized retrieval path, this embodiment extracts their actual timestamps, subtracts the timestamp of the first node from the actual timestamp, and obtains a time difference sequence in milliseconds. This embodiment determines this sequence as the timestamp alignment information for the optimized retrieval path.

[0086] It should be noted that this embodiment calculates the rate of change of time difference between adjacent nodes based on the timestamp alignment information to obtain the temporal features. This embodiment reads the timestamp alignment information sequence, calculates the first difference between the time offset corresponding to the current node and the time offset corresponding to the previous node, and simultaneously calculates the second difference between the time offset corresponding to the previous node and the time offset corresponding to the node before that. The first difference is subtracted from the second difference to obtain a third difference, and the third difference is divided by the second difference to obtain the rate of change of adjacent time differences at the current node position. For the first time interval in the path, the rate of change is not calculated, and its value is directly set to zero. For any adjacent time interval, if the second difference is zero, the rate of change at the current node position is set to zero. This embodiment traverses the entire path to calculate the rate of change of all node positions, constructing a one-dimensional feature vector, which is determined as the temporal feature of the optimized retrieval path. If the optimized retrieval path contains fewer than three nodes, the temporal feature vector of the path is set to a zero vector.

[0087] In one implementation, this embodiment uses the Euclidean distance algorithm to calculate the path similarity score by measuring the distance between the temporal features of different paths, and then constructs a similarity matrix based on the path similarity score. This embodiment obtains the temporal feature vectors of any two optimized retrieval paths in the system. When the lengths of the temporal feature vectors of the two paths are inconsistent, this embodiment uses the longer feature vector as a reference and fills the end of the shorter feature vector with zeros digit by digit until the two lengths are equal. This embodiment calculates the sum of squares of the differences between corresponding dimension elements and performs a square root operation on the sum of squares to obtain the Euclidean distance value. Subsequently, this embodiment calculates the quotient of the value divided by the sum of the value and the Euclidean distance value, and determines this quotient as the path similarity score between the two optimized retrieval paths. This embodiment iterates through and calculates the path similarity score for all optimized retrieval path pairs, using the first path index as the row coordinate and the second path index as the column coordinate, filling the score into the corresponding two-dimensional grid to generate a real symmetric matrix, which is then output as the similarity matrix.

[0088] For example, there exists an optimized retrieval path containing three nodes with absolute timestamps of 10:05:00.100 milliseconds, 10:05:00.150 milliseconds, and 10:05:00.300 milliseconds. In this embodiment, the first node is set as the baseline zero point, i.e., 0 milliseconds. The relative time offsets of subsequent nodes are calculated to be 50 milliseconds and 200 milliseconds, respectively, resulting in a timestamp alignment information sequence of [0, 50, 200]. The first adjacent time difference is 50 milliseconds, and the second adjacent time difference is 150 milliseconds. Calculating 150 minus 50 and then dividing by 50 yields a time difference change rate of 2.0. The sequential feature value is then extracted and used to calculate the sum of squares of the differences with the feature values ​​of another path, taking the square root to obtain an Euclidean distance of 0.5. Further calculation yields a path similarity score of 0.67, which is then filled into the similarity matrix.

[0089] Specifically, path clusters are obtained by clustering based on the temporal features, and time-related paths are obtained by calculating the similarity between the path clusters and the query vector, including:

[0090] The Euclidean distance algorithm is used to calculate the distance between the temporal features of different paths to obtain path similarity scores, and a similarity matrix is ​​constructed based on the path similarity scores.

[0091] The similarity matrix is ​​divided into path clusters by density clustering.

[0092] The centroid virtual path vector is obtained by calculating the mean of the path representation vectors corresponding to all paths within the path cluster.

[0093] Calculate the cosine similarity between the centroid virtual path vector and the query vector;

[0094] If the cosine similarity is greater than the preset consistency threshold, then all paths within the corresponding path cluster are extracted to obtain the time-related paths.

[0095] In one implementation, this embodiment performs density clustering on the similarity matrix to obtain path clusters. This embodiment uses the density-based spatial clustering algorithm DBSCAN to process the similarity matrix. Regarding the determination of the preset minimum sample number parameter upon which this algorithm relies, this embodiment statistically analyzes the average concurrent number of abnormal logs within the same preset time window (e.g., one minute) when a real system failure occurs in history. This average concurrent number is rounded down and objectively set as the preset minimum sample number parameter. Regarding the preset neighborhood radius parameter, this embodiment uses the K-distance graph inflection point method for calibration. The Euclidean distance from each path to its Kth nearest neighbor path is calculated, where K equals the preset minimum sample number parameter. All calculated distance values ​​are arranged in descending order to plot a curve. The second derivative of the curve is calculated, and the distance value corresponding to the point of maximum curvature (i.e., the inflection point) is extracted. This distance value is strictly set as the preset neighborhood radius parameter. This embodiment uses the neighborhood radius parameter and the minimum sample number parameter to perform density reachability traversal on the similarity matrix, grouping density-connected paths into the same group to form multiple independent path clusters.

[0096] It should be noted that in this embodiment, the centroid virtual path vector is obtained by calculating the mean of the semantic vectors of all nodes corresponding to all paths within the path cluster. This embodiment extracts a single path representation vector generated in step S14 by aggregating all optimized retrieval paths contained within the same path cluster. In this embodiment, the arithmetic mean of the dimension elements corresponding to all path representation vectors within the cluster is calculated in the vector space to obtain a single mean vector with the same dimension as the path representation vector. This single mean vector is output as the centroid virtual path vector.

[0097] It is worth noting that in this embodiment, the cosine similarity between the centroid virtual path vector and the query vector is calculated. When the cosine similarity is greater than a preset consistency threshold, all paths within the corresponding path cluster are extracted to obtain time-related paths. The cosine similarity is calculated by dividing the dot product of the two vectors by the product of their magnitudes. Regarding the determination of the preset consistency threshold, this embodiment collects a dataset of benchmark related paths from historical manual troubleshooting records that are confirmed to be completely consistent with the actual query intent. The cosine similarity set between the centroid virtual path vectors corresponding to these benchmark related paths and the historical query vectors is calculated. A normal distribution function is used to fit the probability density of the cosine similarity set, and the mean and standard deviation of the fitted distribution are calculated. The mean is subtracted by twice the standard deviation, and the calculated difference is objectively set as the preset consistency threshold. This embodiment determines whether the currently calculated cosine similarity is greater than the preset consistency threshold. If it is greater than the preset consistency threshold, this embodiment extracts all physical log paths contained in the path cluster and outputs them as filtered time-related paths.

[0098] In step S16, the pre-trained language processing model is used to perform semantic evaluation on the time-related path to obtain candidate entries, the original log text is obtained, and fragment aggregation is performed on the candidate entries and the original log text to obtain merged entries.

[0099] The process of using the pre-trained language processing model to perform semantic evaluation on the time-related path to obtain candidate entries includes:

[0100] The key text entities are obtained by parsing and extracting the time-related path using natural language processing technology, and the key text entities are vectorized and mapped using the pre-trained language processing model to obtain entity semantic vectors.

[0101] A semantic score is obtained by calculating the cosine similarity between the entity semantic vector and the query vector.

[0102] A preliminary search result list is obtained by extracting the key text entities whose semantic scores are greater than a preset score threshold;

[0103] According to the preset sorting and adjustment rules, the items in the preliminary search result list are weighted according to time proximity and position frequency distribution to obtain a priority weight sequence, and the items in the preliminary search result list are rearranged according to the priority weight sequence to obtain candidate items.

[0104] In one implementation, this embodiment utilizes natural language processing technology to parse and extract key text entities from the time-related path. This embodiment employs a named entity recognition algorithm based on conditional random fields (CRF) to perform sequence labeling on the unstructured text contained in the time-related path, extracting noun phrases representing system error states or anomaly types, and identifying them as key text entities. Regarding the model training process of the CRF algorithm, this embodiment extracts a historical IT operations corpus, uses a BIO tagging system to manually label entities containing error states and module names to construct a training set; extracts the word segmentation parts of speech and context boundary characters of the text as state feature functions and transition feature functions; and uses a quasi-Newton method, i.e., the L-BFGS optimizer, to iteratively solve for the maximum likelihood estimation of the network weight parameters of the CRF until the loss function converges, obtaining the trained model. Subsequently, this embodiment uses the pre-trained language processing model to vectorize the key text entities to obtain entity semantic vectors. The network architecture and weight parameters of the pre-trained language processing model are completely consistent with the model used when extracting the query vector. In this embodiment, the extracted key text entities are input into the model, the global representation features output by the last layer of the model are extracted, and a high-dimensional entity semantic vector is generated that is strictly located in the same mapping feature space as the query vector.

[0105] It should be noted that in this embodiment, a semantic score is obtained by calculating the cosine similarity between the entity semantic vector and the query vector. Key text entities with semantic scores greater than a preset score threshold are extracted to obtain a preliminary search result list. This embodiment extracts the sum of the products of the corresponding dimensions of the entity semantic vector and the query vector to obtain a dot product result. The vector magnitudes of both are calculated, and the dot product result is divided by the product of the two vector magnitudes. The resulting cosine similarity value is used as the semantic score. Regarding the determination of the preset score threshold, this embodiment collects historical manually labeled fault diagnosis datasets to construct a validation set. The harmonic mean of precision and recall for entity extraction under different threshold values ​​is calculated, i.e., the F1 score. The threshold value that maximizes this harmonic mean is extracted and strictly defined as the preset score threshold. This embodiment iterates through the semantic scores of all key text entities, removes entities with scores lower than or equal to the preset score threshold, and summarizes the remaining key text entities to generate a preliminary search result list.

[0106] In one implementation, this embodiment assigns a priority weight sequence to the entries in the preliminary search result list according to a preset sorting and adjustment rule based on time proximity and position frequency distribution. The entries in the preliminary search result list are then rearranged according to the priority weight sequence to obtain candidate entries. Regarding the quantitative calculation of time proximity, this embodiment extracts the timestamp of the last occurrence of the key text entity in the time-related path, calculates the time difference between this timestamp and the path's end timestamp, adds this time difference to a preset smoothing constant (e.g., set to 1), and divides this sum by the sum to obtain the normalized time proximity. The preset smoothing constant is used to strictly prevent calculation overflow errors caused by a zero denominator when the time difference is zero. Regarding the quantitative calculation of position frequency distribution, this embodiment counts the absolute number of times the key text entity appears in the entire time-related path, divides this absolute number by the total number of nodes in the path, and obtains the normalized position frequency distribution.

[0107] It is worth noting that the first and second weight coefficients involved in calculating the final weights are objectively calibrated using a hyperparameter optimization algorithm in this embodiment. This embodiment extracts historical search logs to construct a validation set, and uses a grid search to traverse all possible combinations of the first and second weight coefficients within the range of zero to one with a fixed step size, such as 0.05, ensuring that the sum of the two is equal to one. This embodiment calculates the average reciprocal ranking index of the system output results for each set of parameter combinations, extracts the set of parameter combinations that maximizes this average reciprocal ranking index, and strictly calibrates it as the first and second weight coefficients. This embodiment extracts the previously calculated temporal proximity and multiplies it by the first weight coefficient; it also extracts the previously calculated positional frequency distribution and multiplies it by the second weight coefficient; the products of these two are added together to obtain a single priority weight value, and the set of values ​​for all entries generates a priority weight sequence. This embodiment performs a descending sort operation on the preliminary search result list based on the value magnitude in this sequence, extracts the top-ranked entity set, and obtains candidate entries.

[0108] For example, suppose two key text entities, "disk sector corruption" and "regular heartbeat detection," are extracted from the time-related path. After inputting a pre-trained language processing model that is from the same source as the query vector, the semantic scores for these two entities are calculated to be 0.92 and 0.15, respectively. A preset score threshold of 0.80 is determined based on the F1 score maximization criterion of the historical validation set. The system compares the scores and finds that 0.15 is less than 0.80, thus removing the regular heartbeat detection and extracting "disk sector corruption" for inclusion in the preliminary search results list. In the weight allocation calculation, the system extracts the time difference between "disk sector corruption" and the path error endpoint as 10 seconds. Since the smoothing constant is 1, its temporal proximity is calculated to be 0.09. The first weight coefficient, objectively determined based on grid search, is 0.7, and the second weight coefficient is 0.3. The system combines frequency distribution to calculate the final priority weight values, completes the descending order sorting, and outputs the candidate entries.

[0109] This includes obtaining the original log text and performing fragment aggregation based on the candidate entries and the original log text to obtain merged entries, including:

[0110] Obtain the original log text, and scan and extract the initial matching log fragments based on the specific node identifiers in the candidate entries and the original log text.

[0111] Calculate the time difference between two adjacent initial matching log segments. If the time difference is less than a preset time threshold, then connect and splice the two adjacent initial matching log segments to obtain an aggregated log block.

[0112] The correlation strength is calculated by the overlap of the context parameters carried in the aggregated log block. If the correlation strength is greater than a preset correlation threshold, the aggregated log block is determined as a consistent log sequence.

[0113] Duplicate records generated by system retries in the consistency log sequence are cleaned up and removed to obtain a single text, and the single text is identified as the merged entry.

[0114] In one implementation, this embodiment obtains the original log text and scans and extracts initial matching log fragments based on the specific node identifiers in the candidate entries and the original log text. This embodiment traces and extracts the source log nodes mapped to the candidate entries in the time-related path, reads the metadata associated with the source log nodes, and parses out the specific node identifiers containing error status codes, physical node interconnection protocol addresses, or global transaction numbers. This embodiment employs a multi-pattern string matching algorithm, such as the Aho-Corasick algorithm, to perform parallel scanning on massive amounts of original log text, locates complete log text lines containing the specific node identifiers, extracts them in chronological order of physical occurrence, and generates multiple initial matching log fragments.

[0115] It should be noted that in this embodiment, the time difference between two adjacent initial matching log segments is calculated. If the time difference is less than a preset time threshold, the two adjacent initial matching log segments are concatenated to obtain an aggregated log block. In this embodiment, the initial matching log segments are traversed in time order, and the occurrence timestamp of the subsequent segment is extracted and the difference is calculated with the occurrence timestamp of the previous segment. Regarding the determination of the preset time threshold, this embodiment extracts statistical logs of network transmission latency and disk seek latency under historical stable operating conditions, constructs a cumulative distribution histogram of latency values, extracts the latency value corresponding to the 95th percentile in the histogram, and objectively labels it as the preset time threshold. If the currently calculated time difference between adjacent segments is less than the preset time threshold, this embodiment directly concatenates the two segments in the original text string order, merges them into the same block, and generates an aggregated log block.

[0116] In one implementation, this embodiment calculates the correlation strength by assessing the overlap of context parameters carried in the aggregated log block. If the correlation strength is greater than a preset correlation threshold, the aggregated log block is identified as a consistent log sequence. This embodiment uses regular expressions to extract request identifier and session identifier fields as context parameters from each log segment contained in the aggregated log block. This embodiment counts the number of context parameters with identical values ​​in two adjacent segments, divides this number by the total number of context parameters carried by these two segments, and obtains the overlap of context parameters. This overlap is directly determined as the correlation strength. Regarding the determination of the preset correlation threshold, this embodiment extracts consecutive fragmented logs from the historical database that are confirmed to belong to the same business logic chain. The arithmetic mean and standard deviation of the overlap of context parameters between adjacent segments of these historical logs are calculated. The arithmetic mean is subtracted by half of the standard deviation, and the calculated difference is objectively set as the preset correlation threshold. If the currently calculated correlation strength is greater than the preset correlation threshold, this embodiment identifies it as a consistent log sequence.

[0117] It is worth noting that in this embodiment, duplicate records generated by system retries in the consistent log sequence are cleaned up and removed to obtain a single text, which is then identified as the merged entry. This embodiment reads the consistent log sequence and removes the timestamp field from the header of each log text line. This embodiment calculates a secure hash algorithm, such as SHA-256, for each line of the remaining plain text after removing the timestamp. This embodiment compares the hash values ​​of adjacent log text lines sequentially. When multiple identical hash values ​​are detected consecutively, it is determined to be a circular error record caused by the distributed system's retry mechanism. This embodiment retains the first log text line with that hash value, deletes all subsequent redundant records with the same hash value, combines the cleaned, non-redundant log sequence into a single text, and outputs it as the merged entry.

[0118] In step S17, the merged entries are completed using a preset field mapping table to obtain structured data, and the structured data is reformatted using a preset output template to obtain a retrieval report.

[0119] In one implementation, this embodiment utilizes a preset field mapping table to complete the data of the merged entries, obtaining structured data. Regarding the construction method of the preset field mapping table, this embodiment reads the static configuration file of the distributed storage system, the asset management database, and the alarm level definition dictionary, extracts the mapping relationship between the device node number and its corresponding physical interconnect protocol address, gateway mask, business module name, and default alarm level, and stores the above correspondence as a key-value hash table structure to establish the preset field mapping table. This embodiment calls predefined field parsing rules to traverse the input merged entries, checking whether there are any empty basic fields in the entries, such as an empty physical address field. If a missing attribute is detected, this embodiment extracts the existing device node number in the merged entry as the search key, performs a hash addressing query in the preset field mapping table, and extracts the corresponding physical address value and alarm level attribute. This embodiment appends the extracted attribute values ​​to the corresponding empty positions of the merged entries, generating structured data with complete field elements.

[0120] It should be noted that this embodiment uses a preset output template to restructure the structured data to obtain a retrieval report. The preset output template is a pre-written JavaScript object schema standard document or an Extensible Markup Language (Extreme Markup Language) structure document. This preset output template predefines hierarchical nested parsing rules and data node placeholders, including dimensions of time evolution, node topology, and request chain. This embodiment performs an extraction operation, reading the occurrence time, the completed node physical address, the core event description, and the global request number from the structured data as independent variables.

[0121] During the data format reorganization process, this embodiment pre-constructs a business dictionary containing precise mapping relationships between regular expression keywords and scene classification labels. For example, this dictionary defines that keywords such as disconnection or timeout matched by regular expressions are mapped to network connectivity anomaly labels, and keywords such as synchronization failure are mapped to data replica consistency labels. This embodiment inputs the extracted core event descriptions into this business dictionary to perform regular expression comparisons and outputs the corresponding scene classification labels. Subsequently, this embodiment strictly follows the spatial positions of the corresponding placeholders in the preset output template to perform key-value alignment and data filling of the above-mentioned occurrence time, node physical address, core event description, global request number, and generated scene classification labels. This embodiment calls the hierarchical layout algorithm built into the template to allocate the event sequence arranged in chronological order to the root node level of the data block, and allocate the node topology relationship and physical address to the subordinate child node level, generating a formatted data document with a strict tree structure, which is then output as a retrieval report.

[0122] For example, after receiving a merged entry, the system detects that the entry text only contains the words "Node_3," while its physical address and alarm level fields are empty. The system extracts "Node_3" as an index key and performs a query in a preset field mapping table, such as a hash table, to obtain the physical address of node 3 as 192.168.1.103 and the alarm level as critical. The system writes this supplementary data into the current entry, generating structured data. Subsequently, the system inputs the core event description text data block for synchronization failure into a preset business dictionary and performs regular expression matching, successfully outputting a data replica consistency scenario classification label. The system calls a preset JSON Schema output template, precisely aligning the occurrence time 10:12:05, node address 192.168.1.103, event description text, and scenario classification label to the corresponding hierarchical key-value pairs defined in the template, completing the layout and assembly, and finally outputting a retrieval report with machine readability and structural standardization.

[0123] In summary, this invention discloses an AI-based storage log retrieval method and system. This invention extracts high-dimensional features from unstructured logs using a pre-trained model and constructs a global semantic association graph. Combined with query vectors, it performs spatial similarity calculations and topological expansion, achieving accurate understanding and mapping of complex cross-system query intents, breaking through the shallow physical limitations of traditional keyword matching. By introducing a random walk algorithm to evaluate node betweenness centrality to eliminate topological edge noise, and performing density clustering and virtual centroid semantic verification based on temporal features, it achieves dual dynamic filtering of the physical time dimension and the mathematical latent space dimension, effectively avoiding the dimensionality curse caused by cross-model spatial computation and accurately extracting pure links directly to the root cause of faults. Furthermore, by combining entity semantic scores and spatiotemporal distribution features for weight reordering, and utilizing fragment aggregation and a global field mapping table for data completion and structured reorganization, it automatically converges and assembles massive, fragmented, and highly repetitive low-level machine errors into a standardized retrieval analysis report with complete contextual elements. This enables accurate retrieval and intelligent noise reduction based on deep semantic understanding in massive unstructured logs, effectively solving the technical problems of inaccurate matching of complex query intent and low retrieval accuracy in traditional methods, and providing high-quality data support for intelligent troubleshooting and efficient decision-making in business systems.

[0124] Reference Figure 2 The second embodiment of the present invention provides an AI-based storage log retrieval system, comprising:

[0125] The graph construction module is used to acquire stored log data, perform semantic mapping processing on the stored log data to obtain feature vectors, and perform graph construction processing on the feature vectors to obtain a semantic association graph.

[0126] The subgraph extraction module is used to obtain query input, extract features from the query input using a pre-trained language processing model to obtain a query vector, perform relevance retrieval processing on the query vector and the semantic association graph to obtain intent candidate nodes, and perform topology expansion processing on the intent candidate nodes to obtain an intent-related graph.

[0127] The path denoising module is used to perform node value evaluation and denoising processing on the intent-related graph to obtain a log denoised subgraph, and to perform graph traversal processing on the log denoised subgraph to obtain the log reconstruction path.

[0128] The path optimization module is used to perform intent path optimization processing based on the query vector and the log reconstruction path to obtain an optimized retrieval path;

[0129] The temporal clustering module is used to perform temporal analysis on the optimized retrieval path to obtain temporal features, perform clustering based on the temporal features to obtain path clusters, and calculate the time-related path based on the similarity between the path clusters and the query vector.

[0130] The semantic aggregation module is used to perform semantic evaluation on the time-related path using the pre-trained language processing model to obtain candidate entries, acquire the original log text, and perform fragment aggregation on the candidate entries and the original log text to obtain merged entries.

[0131] The report generation module is used to complete the data of the merged entries using a preset field mapping table to obtain structured data, and to reorganize the structured data using a preset output template to obtain a retrieval report.

[0132] It should be noted that the AI-based storage log retrieval system provided in this embodiment of the invention is used to execute all the process steps of the AI-based storage log retrieval method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0133] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0134] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. An AI-based method for retrieving stored logs, characterized in that, include: Obtain stored log data, perform semantic mapping processing on the stored log data to obtain feature vectors, and perform graph construction processing based on the feature vectors to obtain a semantic association graph; Obtain query input, extract features from the query input using a pre-trained language processing model to obtain a query vector, perform relevance retrieval processing on the query vector and the semantic association graph to obtain intent candidate nodes, and perform topology expansion processing on the intent candidate nodes to obtain an intent relevance graph; The intent-related graph is subjected to node value evaluation and noise reduction processing to obtain a log denoised subgraph, and the log denoised subgraph is subjected to graph traversal processing to obtain the log reconstruction path. An optimized retrieval path is obtained by performing intent path optimization processing based on the query vector and the log reconstruction path; The optimized retrieval path is subjected to time series analysis to obtain time series features. Based on the time series features, the path clusters are divided into clusters. The time-related paths are obtained by calculating the similarity between the path clusters and the query vector. The pre-trained language processing model is used to perform semantic evaluation on the time-related path to obtain candidate entries, the original log text is obtained, and fragment aggregation is performed on the candidate entries and the original log text to obtain merged entries; The merged entries are completed using a preset field mapping table to obtain structured data, and the structured data is reformatted using a preset output template to obtain a retrieval report.

2. The AI-based storage log retrieval method according to claim 1, characterized in that, The step of performing semantic mapping processing on the stored log data to obtain feature vectors, and then performing graph construction processing based on the feature vectors to obtain a semantic association graph, includes: The target text features are obtained by extracting unstructured text features from the stored log data using a preset log parsing module. The target text features are mapped using a pre-trained semantic mapping model to obtain an initial vector, and the initial vector is padded with zeros to obtain a feature vector. The feature vectors are subjected to cosine similarity calculation to obtain a semantic similarity matrix, and an initial graph structure is obtained by constructing a graph based on the elements in the semantic similarity matrix that are greater than a preset similarity threshold. The node weights are obtained by calculating the magnitude of the feature vectors, and the semantic association graph is obtained by defining the edge relationships of the initial graph structure based on the node weights and the semantic similarity matrix.

3. The AI-based storage log retrieval method according to claim 1, characterized in that, The step of obtaining intent candidate nodes by performing relevance retrieval processing based on the query vector and the semantic association graph, and then performing topology expansion processing on the intent candidate nodes to obtain an intent relevance graph, includes: A set of candidate nodes is obtained by calculating the cosine similarity between the query vector and the feature vectors of each node in the semantic association graph. The initial subgraph region is obtained by expanding the candidate node set outward to adjacent nodes, and the association strength is obtained by calculating the edge weights within the initial subgraph region. Edge structures with association strength lower than a preset association strength threshold within the initial subgraph region are removed to obtain a set of connected subgraphs; A path integrity coverage assessment is performed on the connected subgraph set to obtain a diversity matching metric. Subgraph regions with diversity matching metrics greater than a preset matching threshold are merged to obtain an intent-related graph.

4. The AI-based storage log retrieval method according to claim 1, characterized in that, The process of performing node value evaluation and noise reduction on the intent-related graph to obtain a log denoised subgraph, and then performing graph traversal on the log denoised subgraph to obtain the log reconstruction path, includes: Betweenness centrality is obtained by estimating the hub value of each node in the intent-related graph using a random walk algorithm; Determine whether the betweenness centrality is lower than a preset structural threshold; If the betweenness centrality is lower than a preset structural threshold, the corresponding node is removed from the intent-related graph to obtain a log denoising subgraph. The log reconstruction path is obtained by performing a graph traversal on the log denoising subgraph.

5. The AI-based storage log retrieval method according to claim 1, characterized in that, The step of optimizing the retrieval path by performing intent path optimization processing based on the query vector and the log reconstruction path includes: The feature vectors of each node in the log reconstruction path are subjected to feature pooling to obtain the path representation vector. A semantic matching score is obtained by performing vector space operations on the query vector and the path representation vector. Highly relevant paths are obtained by extracting paths whose semantic matching scores are greater than a preset intent matching threshold; Tracing the source log nodes mapped by the highly relevant paths, extracting the log timestamps and process call chains from the metadata associated with the source log nodes to obtain context dependency data, and then splicing and merging the highly relevant paths based on the context dependency data to obtain an optimized retrieval path.

6. The AI-based storage log retrieval method according to claim 1, characterized in that, The time-series analysis of the optimized retrieval path to obtain time-series features includes: The starting log node time of the optimized retrieval path is set as the baseline zero point, and the relative time offset of subsequent nodes relative to the baseline zero point is calculated to obtain the timestamp alignment information. The temporal characteristics are obtained by calculating the rate of change of time difference between adjacent nodes based on the timestamp alignment information.

7. The AI-based storage log retrieval method according to claim 1, characterized in that, The step of clustering based on the temporal features to obtain path clusters, and calculating the time-related path based on the similarity between the path clusters and the query vector, includes: The Euclidean distance algorithm is used to calculate the distance between the temporal features of different paths to obtain path similarity scores, and a similarity matrix is ​​constructed based on the path similarity scores. The similarity matrix is ​​divided into path clusters by density clustering. The centroid virtual path vector is obtained by calculating the mean of the path representation vectors corresponding to all paths within the path cluster; the path representation vector is obtained by performing feature pooling on the log reconstruction path. Calculate the cosine similarity between the centroid virtual path vector and the query vector; If the cosine similarity is greater than the preset consistency threshold, then all paths within the corresponding path cluster are extracted to obtain the time-related paths.

8. The AI-based storage log retrieval method according to claim 1, characterized in that, The step of using the pre-trained language processing model to perform semantic evaluation on the time-related path to obtain candidate entries includes: The key text entities are obtained by parsing and extracting the time-related path using natural language processing technology, and the key text entities are vectorized and mapped using the pre-trained language processing model to obtain entity semantic vectors. A semantic score is obtained by calculating the cosine similarity between the entity semantic vector and the query vector. A preliminary search result list is obtained by extracting the key text entities whose semantic scores are greater than a preset score threshold; According to the preset sorting and adjustment rules, the items in the preliminary search result list are weighted according to time proximity and position frequency distribution to obtain a priority weight sequence, and the items in the preliminary search result list are rearranged according to the priority weight sequence to obtain candidate items.

9. The AI-based storage log retrieval method according to claim 1, characterized in that, The step of obtaining the original log text and performing fragment aggregation based on the candidate entries and the original log text to obtain merged entries includes: Obtain the original log text, and scan and extract the initial matching log fragments based on the specific node identifiers in the candidate entries and the original log text. Calculate the time difference between two adjacent initial matching log segments. If the time difference is less than a preset time threshold, then connect and splice the two adjacent initial matching log segments to obtain an aggregated log block. The correlation strength is calculated by the overlap of the context parameters carried in the aggregated log block. If the correlation strength is greater than a preset correlation threshold, the aggregated log block is determined as a consistent log sequence. Duplicate records generated by system retries in the consistency log sequence are cleaned up and removed to obtain a single text, and the single text is identified as the merged entry.

10. An AI-based storage log retrieval system, characterized in that, include: The graph construction module is used to acquire stored log data, perform semantic mapping processing on the stored log data to obtain feature vectors, and perform graph construction processing on the feature vectors to obtain a semantic association graph. The subgraph extraction module is used to obtain query input, extract features from the query input using a pre-trained language processing model to obtain a query vector, perform relevance retrieval processing on the query vector and the semantic association graph to obtain intent candidate nodes, and perform topology expansion processing on the intent candidate nodes to obtain an intent-related graph. The path denoising module is used to perform node value evaluation and denoising processing on the intent-related graph to obtain a log denoised subgraph, and to perform graph traversal processing on the log denoised subgraph to obtain the log reconstruction path. The path optimization module is used to perform intent path optimization processing based on the query vector and the log reconstruction path to obtain an optimized retrieval path; The temporal clustering module is used to perform temporal analysis on the optimized retrieval path to obtain temporal features, perform clustering based on the temporal features to obtain path clusters, and calculate the time-related path based on the similarity between the path clusters and the query vector. The semantic aggregation module is used to perform semantic evaluation on the time-related path using the pre-trained language processing model to obtain candidate entries, acquire the original log text, and perform fragment aggregation on the candidate entries and the original log text to obtain merged entries. The report generation module is used to complete the data of the merged entries using a preset field mapping table to obtain structured data, and to reorganize the structured data using a preset output template to obtain a retrieval report.

Images