Container fault tracing method and device based on cross-community knowledge fusion and medium

Abstract

The invention relates to a container fault tracing method and device based on cross-community knowledge fusion and a medium. The method comprises the steps that a container fault traceability model is constructed, wherein the container fault traceability model comprises a data processing module, a knowledge extraction module, a knowledge fusion engine module and a fault traceability module, and key entity extraction is conducted on processed data in the knowledge extraction module through the NLP technology to construct a unified knowledge representation model; in a knowledge fusion engine module, fault-related entities in the unified knowledge representation model and the causal relationship between the entities are aligned through a defined cross-community entity alignment function to construct an initial cross-community knowledge graph, and then dynamic graph updating is performed on the initial cross-community knowledge graph by using a time sequence graph convolutional network. And a causal directed acyclic graph is generated in a fault tracing module according to the final cross-community knowledge graph, and container fault tracing is performed on the causal directed acyclic graph by using a root cause positioning algorithm. By adopting the method, the timeliness of fault diagnosis can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and particularly to a container fault tracing method, device and medium based on cross-community knowledge fusion. Background Art

[0002] In a large-scale Internet software development project, the team adopted container technology for application deployment and management to achieve efficient resource utilization and rapid service delivery. In the cloud-native environment of the entire project, containerized applications run on a cluster managed by Kubernetes, involving multiple microservice modules covering different functions such as user interface, backend logic, and database access. During daily operations, the project team relies on tools such as Prometheus and ELK Stack to monitor the containers. If users report problems such as response latency and partial data loading failures when using specific functions of the application, the operations and maintenance team starts to investigate. By viewing the performance metrics of the containers through Prometheus, it is found that the CPU usage of a certain microservice container has increased abnormally. At the same time, the logs in ELK Stack show that the container has encountered multiple errors when processing specific requests. However, relying solely on these local logs and metric data, it is difficult for the team to determine the root cause of the problem. Using existing methods, a rule engine and machine learning models are used to locate the fault. However, since these methods do not fully utilize cross-community heterogeneous knowledge resources, the problem-solving has reached a deadlock. For example, the project team did not find a solution related to this failure in the enterprise internal knowledge base. Although the enterprise internal monitoring tools record detailed container operation data, they know nothing about the possible similar problems and solutions in the open source community. Although a certain container monitoring system used in the project detected the fault through log aggregation and metric analysis, due to its reliance on structured data and lack of the ability to integrate unstructured cross-community knowledge, the analysis is not comprehensive enough. In technical communities such as Stack Overflow, there may be discussions and solutions shared by other developers about similar container failures, but this monitoring system cannot obtain and utilize this information. Although the enterprise internal logs record the container operation status in detail, they cannot provide effective clues for known compatibility problems in the open source community, such as the incompatibility of some new software library versions with the existing container environment. This is because the existing container fault tracing technology relies on a single community and cannot integrate heterogeneous data across communities such as GitHub and Stack Overflow, resulting in a limited coverage of fault diagnosis. In terms of data processing, there is a huge format difference between structured logs and metric data and unstructured technical documents and community discussions, and the phenomenon of knowledge islands is serious. For example, the open source project documents obtained from GitHub are in plain text format, while the logs generated by the container monitoring system are in structured JSON format. Integrating the two requires a lot of manual intervention for data alignment, which not only leads to low tracing efficiency but also greatly reduces the accuracy. Traditional fault tracing methods rely on static knowledge bases and cannot update cross-community knowledge in real time. As the container environment is frequently updated, new fault patterns keep emerging, and the old knowledge bases cannot identify and solve these new problems in a timely manner. In a project, due to the regular updates of container images and dependent software libraries, a new fault pattern appeared after a certain update, but the root cause analysis method based on the static knowledge base failed to detect the problem in time, resulting in an extended service interruption time. Existing technologies do not design a unified representation framework compatible with multi-modal data, making it difficult to fuse heterogeneous data such as text, logs, and metrics. Traditional ETL tools and rule engines are difficult to perform semantic alignment and real-time reasoning on unstructured data. When faced with a large amount of information from different communities and data sources, it is impossible to quickly and effectively extract useful knowledge, affecting the speed and accuracy of fault tracing. At the same time, traditional methods do not incorporate dynamic update technologies such as temporal graph neural networks (T-GCN) and cannot adapt to the rapid changes in the container environment. In a container cluster, the call relationships and resource allocation situations between services change in real time, and traditional methods are difficult to capture these dynamic information in real time and use them for fault analysis, resulting in insufficient timeliness of fault diagnosis. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide a container fault tracing method, device, and medium based on cross-community knowledge fusion that can improve the timeliness of fault diagnosis.

[0004] A container fault tracing method based on cross-community knowledge fusion, the method includes: Obtain structured data and unstructured data; construct a container fault tracing model; the container fault tracing model includes a data processing module, a knowledge extraction module, a knowledge fusion engine module, and a fault tracing module; According to the data processing module, standardize the structured data and vectorize the unstructured data to obtain processed data; In the knowledge extraction module, use NLP technology to extract key entities from the processed data to construct a unified knowledge representation model; In the knowledge fusion engine module, align the fault-related entities and the causal relationships between entities in the unified knowledge representation model through a defined cross-community entity alignment function to construct an initial cross-community knowledge graph, and then use a temporal graph convolutional network to perform dynamic graph updates on the initial cross-community knowledge graph to obtain a final cross-community knowledge graph; In the fault tracing module, generate a causal directed acyclic graph according to the final cross-community knowledge graph, and use a root cause location algorithm to perform container fault tracing on the causal directed acyclic graph.

[0005] A computer device, comprising a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: Obtain structured data and unstructured data; construct a container fault traceability model; the container fault traceability model includes a data processing module, a knowledge extraction module, a knowledge fusion engine module, and a fault traceability module; According to the data processing module, standardize the structured data and vectorize the unstructured data for the structured data and unstructured data to obtain processed data; In the knowledge extraction module, use NLP technology to extract key entities from the processed data to construct a unified knowledge representation model; In the knowledge fusion engine module, align the fault-related entities and the causal relationships between entities in the unified knowledge representation model through a defined cross-community entity alignment function to construct an initial cross-community knowledge graph, and then use a temporal graph convolutional network to perform dynamic graph updates on the initial cross-community knowledge graph to obtain a final cross-community knowledge graph; In the fault traceability module, generate a causal directed acyclic graph according to the final cross-community knowledge graph, and use a root cause localization algorithm to perform container fault traceability on the causal directed acyclic graph.

[0006] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtain structured data and unstructured data; construct a container fault traceability model; the container fault traceability model includes a data processing module, a knowledge extraction module, a knowledge fusion engine module, and a fault traceability module; According to the data processing module, standardize the structured data and vectorize the unstructured data for the structured data and unstructured data to obtain processed data; In the knowledge extraction module, use NLP technology to extract key entities from the processed data to construct a unified knowledge representation model; In the knowledge fusion engine module, align the fault-related entities and the causal relationships between entities in the unified knowledge representation model through a defined cross-community entity alignment function to construct an initial cross-community knowledge graph, and then use a temporal graph convolutional network to perform dynamic graph updates on the initial cross-community knowledge graph to obtain a final cross-community knowledge graph; In the fault traceability module, generate a causal directed acyclic graph according to the final cross-community knowledge graph, and use a root cause localization algorithm to perform container fault traceability on the causal directed acyclic graph.

[0007] The above-mentioned container fault tracing method, device, and medium based on cross-community knowledge fusion. In this application, a container fault tracing model is constructed. The container fault tracing model includes a data processing module, a knowledge extraction module, a knowledge fusion engine module, and a fault tracing module. The data processing module standardizes structured data and vectorizes unstructured data, unifying the data format, reducing processing obstacles, and accelerating data conversion. The knowledge extraction module uses NLP technology to quickly extract key entities to build a model, providing effective basic data for tracing. In the knowledge fusion engine module, through the entity alignment function, multi-source knowledge is quickly integrated to form an initial graph, and then it is dynamically updated using T-GCN. T-GCN combines the time dimension to capture environmental changes in real-time. The lightweight incremental GNN only locally trains the newly added subgraphs, avoiding repeated training, enabling the graph to keep up with environmental changes and providing the latest knowledge for tracing. The fault tracing module generates a causal directed acyclic graph based on the latest knowledge graph, using algorithms for automated reasoning to reduce manual intervention. Based on comprehensive and real-time updated knowledge, the root cause of the fault can be quickly located. Compared with traditional methods, this application breaks down data and knowledge barriers, updates knowledge in real-time, automates the processing flow, and avoids inefficiencies caused by inconsistent data formats, outdated knowledge, and excessive manual intervention. It adapts to the rapid changes in the container environment and, through the collaborative effect of each link, significantly improves the timeliness of fault diagnosis and tracing. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic flowchart of a container fault tracing method based on cross-community knowledge fusion in an embodiment; Figure 2 It is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0009] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further details this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0010] In one embodiment, as Figure 1 shown, a container fault tracing method based on cross-community knowledge fusion is provided, including the following steps: Step 102, obtain structured data and unstructured data; construct a container fault tracing model. The container fault tracing model includes a data processing module, a knowledge extraction module, a knowledge fusion engine module, and a fault tracing module.

[0011] Step 104, according to the data processing module, standardize the structured data and vectorize the unstructured data to obtain the processed data.

[0012] Define a unified Schema for structured data, map log fields to standard triples <Timestamp, LogLevel, Message>, standardize the data format, facilitate subsequent rapid processing and analysis, reduce the processing time consumed by inconsistent data formats, enable subsequent modules to use this data faster, and thus improve the overall efficiency of fault diagnosis and traceability. Convert unstructured data into vector form, and use a pre-trained model to generate text embedding vectors so that it can be processed in the same framework as structured data. Break the gap between data types, avoid the complex processes that need to be processed separately due to different data forms, save time, and accelerate the conversion speed from raw data to available analysis data.

[0013] Step 106: In the knowledge extraction module, use NLP technology to extract key entities from the processed data and construct a unified knowledge representation model.

[0014] Use NLP technology to extract key entities and construct a unified knowledge representation model, quickly extract important information related to faults from the processed data, such as error codes, solutions, etc., without manually screening the data one by one, reduce the manual analysis time, enable the rapid extraction and integration of fault-related knowledge, and provide timely and effective basic data for subsequent fault traceability.

[0015] Step 108: In the knowledge fusion engine module, align the fault-related entities and the causal relationships between entities in the unified knowledge representation model through the defined cross-community entity alignment function to construct an initial cross-community knowledge graph, and then use the temporal graph convolutional network to perform dynamic graph updates on the initial cross-community knowledge graph to obtain the final cross-community knowledge graph.

[0016] By defining a cross-community entity alignment function, quickly determine equivalent entities from different community sources, align the fault-related entities and causal relationships to construct an initial cross-community knowledge graph, enable the rapid integration of knowledge from different places, avoid the cumbersome process of manually searching and matching knowledge from different sources, and accelerate the first step of knowledge fusion. Use the temporal graph convolutional network (T-GCN) to perform dynamic updates on the initial graph. T-GCN can combine the time dimension and capture dynamic change information such as service call relationships and resource allocation in the container environment in real time. A lightweight incremental GNN that only performs local training on the newly added subgraph, avoids repeated training of the entire graph, quickly updates the knowledge graph according to new data, ensures that the knowledge graph always reflects the latest container environment state, provides the latest knowledge support for fault traceability, and improves the timeliness of fault diagnosis.

[0017] Step 110: In the fault traceability module, generate a causal directed acyclic graph based on the final cross-community knowledge graph, and use the root cause location algorithm to perform container fault traceability on the causal directed acyclic graph.

[0018] Generate a causal directed acyclic graph based on the final cross - community knowledge graph, and use the root cause localization algorithm for fault tracing. The automated data integration and reasoning process reduces manual intervention. Based on the latest and comprehensive knowledge graph, the root cause of the fault can be quickly located, avoiding repeated troubleshooting caused by incomplete information or outdated knowledge, and greatly shortening the time for fault diagnosis and tracing.

[0019] In the above - mentioned container fault tracing method based on cross - community knowledge fusion, this application constructs a container fault tracing model. The container fault tracing model includes a data processing module, a knowledge extraction module, a knowledge fusion engine module, and a fault tracing module. The data processing module standardizes structured data and vectorizes unstructured data, unifying the data format, reducing processing obstacles, and accelerating data conversion. The knowledge extraction module uses NLP technology to quickly extract key entities to build a model, providing effective basic data for tracing. In the knowledge fusion engine module, through the entity alignment function, multi - source knowledge is quickly integrated to form an initial graph, and then T - GCN is used for dynamic update. T - GCN combines the time dimension to capture environmental changes in real - time. The lightweight incremental GNN only locally trains the newly added sub - graph, avoiding repeated training, enabling the graph to keep up with environmental changes, and providing the latest knowledge for tracing. The fault tracing module generates a causal directed acyclic graph based on the latest knowledge graph, and uses the algorithm for automated reasoning, reducing manual intervention. Based on comprehensive and real - time updated knowledge, the root cause of the fault can be quickly located. Compared with traditional methods, this application breaks data and knowledge barriers, updates knowledge in real - time, automates the processing flow, avoids inefficiencies caused by inconsistent data formats, outdated knowledge, and excessive manual intervention, adapts to the rapid changes in the container environment, and significantly improves the timeliness of fault diagnosis and tracing under the synergistic effect of each link.

[0020] In one of the embodiments, the process of obtaining structured data and unstructured data includes: Use DaemonSet to collect Kubernetes Pod logs and Prometheus monitoring metrics as structured data; Prometheus monitoring metrics include CPU / memory time - series data, etc.; Crawl GitHub Issue descriptions, Stack Overflow Q&A texts, and enterprise internal documents related to container faults as unstructured data according to web crawler technology; in a specific embodiment, container - fault - related unstructured data that meets the requirements can be screened by keyword matching of document titles or tags (such as "container", "fault", etc.); Perform data standardization operations on the structured data, define a unified Schema, where the log fields are mapped to a <Timestamp, LogLevel, Message> triple; Perform data vectorization on unstructured data and use a pre-trained model to generate text embedding vectors.

[0021] In one embodiment, the knowledge extraction module uses NLP technology to extract key entities from the processed data and construct a unified knowledge representation model, including: Identify fault-related entities in the processed data according to the pre-trained model to construct an entity set; the fault-related entities include error codes, service names, and operation steps; Use dependency syntax analysis to generate relationship triples for the causal relationships between fault-related entities: R ={( e i , r ij , e j )}; Among them, e i and e j respectively represent two entities, r ij represents the relationship between entities; Construct a unified knowledge representation model according to the entity set, relationship triples, and attribute set (such as timestamp, confidence).

[0022] In one embodiment, the defined cross-community entity alignment function is: ; Among them, , represents entities in different communities, represents 's vector representation, represents 's vector representation, represents the norm.

[0023] In a specific embodiment, equivalent entities are determined for alignment through a cosine similarity threshold.

[0024] In one embodiment, the final cross-community knowledge graph includes multiple nodes; use a temporal graph convolutional network to perform dynamic graph update on the initial cross-community knowledge graph to obtain the final cross-community knowledge graph, including: According to the temporal graph convolutional network, use the node features of the previous moment to retain historical context, combine the new data at the current moment with the node features of the previous moment, and update the node features at the current moment to obtain the final cross-community knowledge graph.

[0025] In one embodiment, the newly added data at the current moment is combined with the node features at the previous moment to update the node features of the initial cross-community knowledge graph at the current moment, including: The newly added data at the current moment is combined with the node features at the previous moment, and the node features at the current moment are updated to: ; Wherein, represents the activation function, represents the historical state weight matrix, represents the node features at the previous moment, represents the newly added data weight matrix, represents the newly added data.

[0026] In one embodiment, a root cause localization algorithm is used to trace the container faults in the causal directed acyclic graph, including: Generate a causal directed acyclic graph based on the cross-community knowledge graph, where the nodes are fault entities, and the edge weights are quantified by the conditional probability Use the root cause localization algorithm to trace the container faults in the causal directed acyclic graph, and the root cause node obtained is: ; Wherein, and respectively represent different nodes in the causal directed acyclic graph, represents the conditional probability, indicating the probability that node occurs under the condition that node occurs, and is used to measure the likelihood of causal association between nodes, represents the LSTM prediction error, represents the scoring function for node , which is obtained by accumulating the relevant calculation results of all its descendant nodes, represents the set of all descendant nodes of node .

[0027] It should be understood that although the steps in the flowchart of Figure 1 are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, Figure 1At least some of the steps may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed and completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least some of the sub-steps or stages of other steps or other steps.

[0028] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 2 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for tracing the source of container failures based on cross-community knowledge fusion. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the outer shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0029] Those skilled in the art can understand that Figure 2 the structure shown in

[0030] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0031] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0032] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A container fault tracing method based on cross-community knowledge fusion, characterized in that: The method comprises: Acquire structured data and unstructured data; construct a container fault tracing model; the container fault tracing model includes a data processing module, a knowledge extraction module, a knowledge fusion engine module and a fault tracing module; Performing structured data standardization and unstructured data vectorization on the structured data and unstructured data according to the data processing module to obtain processed data; In the knowledge extraction module, NLP technology is used to extract key entities from the processed data to build a unified knowledge representation model; In the knowledge fusion engine module, the fault-related entities and the causal relationships between entities in the unified knowledge representation model are aligned by a defined cross-community entity alignment function to construct an initial cross-community knowledge graph, and then the initial cross-community knowledge graph is dynamically updated by using a temporal graph convolutional network to obtain a final cross-community knowledge graph; In the fault tracing module, a causal directed acyclic graph is generated according to the final cross-community knowledge graph, and a root cause location algorithm is used to trace the container fault on the causal directed acyclic graph.

2. The method according to claim 1, characterized in that The process of acquiring structured and unstructured data includes: Use DaemonSet to collect Kubernetes Pod logs and Prometheus monitoring indicators as structured data; the Prometheus monitoring indicators include CPU / memory time series data; Use crawler technology to crawl GitHub Issue descriptions, Stack Overflow Q&A texts, and internal corporate documents related to container failures as unstructured data.

3. The method according to claim 1, characterized in that: The structured data and the unstructured data are normalized and vectorized according to the data processing module to obtain processed data, including: Perform data standardization on structured data and define a unified Schema, where the log field is mapped as<Timestamp, LogLevel, Message> Triples; Perform data vectorization operations on unstructured data and use pre-trained models to generate text embedding vectors.

4. The method according to claim 1, characterized in that: In the knowledge extraction module, NLP technology is used to extract key entities from the processed data to build a unified knowledge representation model, including: Identify fault-related entities in the processed data according to the pre-trained model to construct an entity set; the fault-related entities include error codes, service names, and operation steps; The causal relationship between the fault-related entities is analyzed using dependency syntax to generate a relationship triple: R ={( e i , r ij , e j )} in, e i and e j Represent two entities respectively. r ij Represents relationships between entities; A unified knowledge representation model is constructed according to the entity set, relationship triples and attribute set.

5. The method according to claim 1, characterized in that The cross-community entity alignment function defined is: in, , Entities representing different communities, express The vector representation of express The vector representation of Represents the norm.

6. The method according to claim 1, characterized in that The final cross-community knowledge graph includes multiple nodes; the initial cross-community knowledge graph is dynamically updated using a temporal graph convolutional network to obtain the final cross-community knowledge graph, including: According to the temporal graph convolutional network, the node features of the previous moment are used to retain the historical context, the new data at the current moment is combined with the node features of the previous moment, the node features of the current moment are updated, and the final cross-community knowledge graph is obtained.

7. The method according to claim 6, characterized in that Combine the newly added data at the current moment with the node features at the previous moment to update the node features of the initial cross-community knowledge graph at the current moment, including: Combine the newly added data at the current moment with the node features at the previous moment, and update the node features at the current moment to: in, represents the activation function, represents the historical state weight matrix, represents the node features at the previous moment, represents the weight matrix of new data, Indicates newly added data.

8. The method according to claim 1, characterized in that The root cause location algorithm is used to trace the cause and effect directed acyclic graph to the container fault, including: Generate a causal directed acyclic graph based on the cross-community knowledge graph, where nodes are fault entities and edge weights are determined by conditional probabilities. Quantitatively, the root cause location algorithm is used to trace the container failure on the causal directed acyclic graph, and the root cause node is obtained as follows: in, and They represent different nodes in the causal directed acyclic graph, Represents the conditional probability, indicating that at the node Under the conditions that occur, the node The probability of occurrence is used to measure the possibility of causal relationship between nodes. represents the LSTM prediction error, Represents the node The scoring function is obtained by accumulating the relevant calculation results of all its descendant nodes. Representation Node The collection of all descendant nodes of .

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images