Unified log and monitoring method and system of federal multi-cluster

Abstract

The invention relates to the technical field of operation and maintenance management in a distributed computing environment, and discloses a federated multi-cluster unified log and monitoring method and system, and the method comprises a log data collection module, a feature analysis module, a monitoring strategy generation module and a monitoring execution module. The log data acquisition module acquires historical log records, performance indexes and abnormal event data through a cross-cluster interface; the feature analysis module performs cleaning and feature extraction on the data by adopting a time window method and quantile calculation to generate log, performance and abnormal feature parameters; the monitoring strategy generation module performs strategy retrieval based on the knowledge graph space, and dynamically matches an optimal monitoring strategy; and the monitoring execution module carries out monitoring according to the strategy and analyzes the result. Uniform collection, intelligent feature analysis and self-adaptive monitoring of log data in a multi-cluster environment are achieved, the operation and maintenance efficiency and monitoring precision of a cross-cluster system are improved, and the method is suitable for distributed scenes such as cloud computing and big data.

Description

Technical Field

[0001] This invention relates to the field of operation and maintenance management technology in distributed computing environments, specifically to a unified logging and monitoring method and system for federated multi-cluster systems. Background Technology

[0002] In modern distributed computing environments, federated multi-cluster architectures are widely used in scenarios such as cloud computing, big data processing, and microservice management. This architecture achieves resource sharing and task collaboration by uniting multiple independent clusters, but its complexity also brings significant operational challenges. Logging and monitoring are core aspects of operations and maintenance; however, traditional methods in multi-cluster environments suffer from the following problems: Cross-cluster log data is scattered and heterogeneous. Different clusters may use different log formats, storage systems, and collection tools, making it difficult to manage and analyze the data uniformly. Traditional solutions typically rely on manual configuration or monitoring tools for a single cluster, which cannot efficiently integrate cross-cluster data, resulting in monitoring blind spots and delays.

[0003] Monitoring strategies lack dynamic adaptability. Existing systems mostly trigger alarms based on static rules or simple thresholds, failing to dynamically adjust strategies according to historical log characteristics, performance metrics, and abnormal events. For example, fixed thresholds may generate a large number of false alarms or missed alarms due to cluster load fluctuations, reducing monitoring effectiveness.

[0004] The automation level of feature analysis and policy generation is insufficient. Traditional methods typically require manual intervention to extract features or design policies, which is not only inefficient but also difficult to handle the real-time monitoring needs of large-scale clusters. Although some research has attempted to introduce machine learning, model training relies on large amounts of labeled data, and the logic of policy generation is difficult to interpret, limiting practical applications.

[0005] The identification and feedback mechanism for monitoring results is inadequate. Periodic monitoring data often lacks effective classification and prioritization, making it difficult for operations and maintenance personnel to quickly locate critical issues. Existing identifiers are mostly based on simple rules and cannot combine multi-dimensional features for comprehensive judgment, affecting the timeliness of fault handling.

[0006] Therefore, a system is needed that can automatically integrate cross-cluster log data, dynamically generate monitoring policies, and efficiently identify monitoring results to solve the above-mentioned technical problems. Summary of the Invention

[0007] The purpose of this invention is to provide a unified logging and monitoring method and system for federated multi-cluster systems to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a unified logging and monitoring method and system for federated multi-cluster systems, the method comprising: The log data acquisition module is configured to perform cross-cluster interface data retrieval to obtain the historical log record set, historical performance index set, and historical abnormal event set of the cross-cluster interface for tracking the target cluster within a preset observation period. The feature analysis module is configured to traverse the set of historical log records, the set of historical performance indicators, and the set of historical abnormal events to perform feature dimension parsing and determine log feature parameters, performance feature parameters, and abnormal feature parameters. The monitoring strategy generation module is configured to use the log feature parameters, performance feature parameters, and anomaly feature parameters as index keys to perform a centralized search within the monitoring strategy configuration space to determine the target monitoring strategy. The monitoring execution module is configured to perform monitoring status control on the target cluster according to the target monitoring strategy and a preset monitoring rule set, obtain a periodic monitoring dataset, and use a monitoring result recognizer to identify the periodic monitoring dataset to obtain the target monitoring result.

[0009] Preferably, the feature analysis module is further configured as follows: The historical log record set is parsed using the time window method to obtain the log time window; Collect historical log records within the log time window that fall between the effective upper limit and the effective lower limit, and obtain a cleaned log record set. The effective upper limit is the boundary value formed by adjusting the maximum value in the window downward by a preset ratio, and the effective lower limit is the boundary value formed by adjusting the minimum value in the window upward by a preset ratio. The discreteness of the cleaned log record set is calculated to obtain the log feature parameters; The same feature dimension parsing logic is applied to the set of historical performance indicators and the set of historical abnormal events to obtain the performance feature parameters and abnormal feature parameters.

[0010] Preferably, the feature analysis module is further configured as follows: The historical log records are arranged in order of timestamp, and the historical log record in the middle position is used as the window baseline. Calculate the first quarter time quantile and the second quarter time quantile of the historical log record set; The log time window is constructed by using the first quarter time quantile as the lower boundary value and the second quarter time quantile as the upper boundary value, combined with the window baseline.

[0011] Preferably, the monitoring strategy generation module is further configured to: Multiple sample log feature parameters, multiple sample performance feature parameters, and multiple sample anomaly feature parameters, as well as corresponding multiple sample monitoring strategies, are collected as construction data. A pre-constructed knowledge graph space is provided, wherein the root node of the knowledge graph space is R, the first branch is log feature parameters, the second branch is performance feature parameters, and the third branch is anomaly feature parameters. The constructed data is input into the knowledge graph space to obtain multiple sample knowledge nodes, and the multiple sample knowledge nodes are labeled using the multiple sample monitoring strategies to obtain the monitoring strategy configuration space.

[0012] Preferably, the monitoring strategy generation module is further configured to: The branch path in the monitoring strategy configuration space that passes through the log feature parameters and is parallel to the performance-anomaly branch plane is taken as the first path; The branch path in the monitoring strategy configuration space that passes through the performance characteristic parameters and is parallel to the log-anomaly branch plane is taken as the second path; The branch path in the monitoring strategy configuration space that passes through the abnormal feature parameters and is parallel to the log-performance branch plane is taken as the third path; The subspace enclosed by the performance-anomaly branch plane, the log-anomaly branch plane, the log-performance branch plane, the first path, the second path, and the third path is used as a configuration subgraph, wherein the configuration subgraph contains multiple configuration sample knowledge nodes; A centralized search is performed on the multiple configuration sample knowledge nodes to determine the target configuration sample knowledge node, and the sample monitoring strategy corresponding to the target configuration sample knowledge node is used as the target monitoring strategy.

[0013] Preferably, the monitoring strategy generation module is further configured to: Extract the central configuration sample knowledge node of the configuration subgraph, and construct a central neighborhood graph with the central configuration sample knowledge node as the starting point and according to a preset centralized retrieval range. The central neighborhood graph is a tree-like subgraph constructed with the central configuration sample knowledge node as the core and the preset centralized retrieval range as the expansion radius. The number of configured sample knowledge nodes in the central neighborhood graph is counted, and the counted value is compared with the node density of the central neighborhood graph to obtain the central neighborhood concentration. Randomly select a configuration sample knowledge node from the edge of the central neighborhood graph as the first retrieval configuration node, and construct the first retrieval neighborhood concentration of the first retrieval configuration node; Determine whether the concentration of the first search neighborhood is greater than or equal to the concentration of the central neighborhood. If so, update the first search configuration node as the starting point and continue to carry out centralized search until the preset number of centralized searches is met. Then, take the search configuration node obtained in the last search as the target configuration sample knowledge node.

[0014] Preferably, the monitoring strategy generation module is further configured to: If this is not the case, the initial value of the retrieval update failure count of 0 will be updated to 1, and a configuration sample knowledge node will be randomly selected from the edge of the central neighborhood graph as the first retrieval configuration node to carry out centralized retrieval analysis. When the retrieval update failure count is greater than the preset maximum retrieval update failure count, the central configuration sample knowledge node will be used as the target configuration sample knowledge node.

[0015] Preferably, the present invention also includes a unified logging and monitoring method for federated multi-cluster systems, the method comprising: Perform cross-cluster interface data retrieval to obtain the historical log record set, historical performance index set, and historical abnormal event set of the cross-cluster interface for tracking the target cluster within a preset observation period; The historical log record set, historical performance index set, and historical abnormal event set are traversed to perform feature dimension parsing to determine log feature parameters, performance feature parameters, and abnormal feature parameters; Using the log feature parameters, performance feature parameters, and anomaly feature parameters as index keys, a centralized search is conducted within the monitoring policy configuration space to determine the target monitoring policy. Based on the target monitoring strategy, the target cluster is monitored and its status is adjusted according to a preset monitoring rule set to obtain a periodic monitoring dataset. The periodic monitoring dataset is then identified using a monitoring result recognizer to obtain the target monitoring result.

[0016] Preferably, the method further includes the step of traversing the historical log record set, the historical performance indicator set, and the historical abnormal event set to perform feature dimension parsing and determine log feature parameters, performance feature parameters, and abnormal feature parameters, specifically including: The historical log record set is parsed using the time window method to obtain the log time window; Collect historical log records within the log time window that fall between the effective upper limit and the effective lower limit, and obtain a cleaned log record set. The effective upper limit is the boundary value formed by adjusting the maximum value in the window downward by a preset ratio, and the effective lower limit is the boundary value formed by adjusting the minimum value in the window upward by a preset ratio. The discreteness of the cleaned log record set is calculated to obtain the log feature parameters; The same feature dimension parsing logic is applied to the set of historical performance indicators and the set of historical abnormal events to obtain the performance feature parameters and abnormal feature parameters.

[0017] Preferably, the step of using the time window method to perform data parsing on the historical log record set and obtaining the log time window specifically includes: The historical log records are arranged in order of timestamp, and the historical log record in the middle position is used as the window baseline. Calculate the first quarter time quantile and the second quarter time quantile of the historical log record set; The log time window is constructed by using the first quarter time quantile as the lower boundary value and the second quarter time quantile as the upper boundary value, combined with the window baseline.

[0018] Compared with the prior art, the beneficial effects of the present invention are: The unified logging and monitoring system and method for federated multi-cluster systems proposed in this invention fundamentally solves many problems existing in traditional technologies through the collaborative work of multiple innovative modules and processes, and demonstrates significant advantages in improving the management efficiency and stability of federated multi-cluster systems. Regarding log data collection, the system's log data collection module performs cross-cluster interface data retrieval, enabling it to efficiently acquire complete and accurate historical log records, historical performance metrics, and historical anomaly events for the target cluster within a preset observation period. This process fully considers the differences in data interfaces and formats across clusters, employing an adaptive data retrieval strategy to ensure the comprehensiveness and accuracy of data collection, laying a solid data foundation for subsequent analysis and monitoring. In the feature analysis phase, the innovative design of the feature analysis module significantly improves the quality and efficiency of data processing. By employing a time window method to analyze the historical log record set, and combining this with the setting of effective upper and lower limits for data cleaning, noisy data is effectively eliminated, resulting in a more representative cleaned log record set. Based on this, dispersion calculations are performed to accurately determine log feature parameters. Simultaneously, the same logic is used to analyze the feature dimensions of historical performance indicator sets and historical anomaly event sets, thereby comprehensively and accurately obtaining performance and anomaly feature parameters. These precise feature parameters provide a reliable basis for the formulation of subsequent monitoring strategies, more realistically reflecting the cluster's operating status and potential problems. The monitoring strategy generation module is one of the core innovations of this invention. By collecting multiple sample data sets to construct a knowledge graph space and labeling knowledge nodes, a rich and structured monitoring strategy configuration space is formed. Based on this, a unique path planning and subspace construction method is used to determine the configuration subgraph, and a concentration-based centralized retrieval strategy is employed to quickly and accurately locate the target monitoring strategy from a massive pool of monitoring strategies. This dynamic and intelligent monitoring strategy generation method can flexibly adjust according to the real-time operating status of the cluster and changing business needs, significantly improving the targeting and effectiveness of monitoring strategies. During the monitoring execution phase, the monitoring execution module adjusts the monitoring status of the target cluster based on the target monitoring strategy and preset monitoring rule set. By acquiring periodic monitoring datasets and using a monitoring result recognizer for accurate identification, it can obtain target monitoring results in a timely and accurate manner. This enables the system to monitor the cluster's operational status in real time, provide rapid early warnings and responses to potential problems and faults, and effectively improve the stability and reliability of the federated multi-cluster system. This invention optimizes the entire process from log data collection, feature analysis, monitoring strategy generation to monitoring execution. It not only improves the efficiency and accuracy of data processing and enhances the flexibility and effectiveness of monitoring strategies, but also significantly reduces system operation and maintenance costs, providing strong technical support for the stable operation and efficient management of federated multi-cluster systems. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the working principle of the unified logging and monitoring system for federated multi-cluster systems described in this invention. Figure 2 Build a design diagram for the log time window; Figure 3 Design diagram for configuring sub-map retrieval; Figure 4 A schematic diagram illustrating the principle of neighborhood retrieval. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figures 1-4 This invention relates to a unified logging and monitoring method and system for federated multi-cluster systems. The system includes a log data acquisition module, a feature analysis module, a monitoring policy generation module, and a monitoring execution module. The specific implementation steps are as follows: The log data acquisition module is configured to pull data from cross-cluster interfaces, obtaining a set of historical log records, historical performance metrics, and historical anomaly events tracked by the cross-cluster interfaces within a preset observation period. Specifically, this module interacts with the interfaces of each cluster and extracts relevant historical data from the target cluster according to a set time period. For example, the preset observation period could be one day, one week, or other suitable timeframes, set according to actual needs. When pulling data, it is necessary to ensure data integrity and accuracy to avoid data loss or errors.

[0022] The feature analysis module is configured to traverse the historical log record set, historical performance indicator set, and historical anomaly event set to perform feature dimension parsing, determining log feature parameters, performance feature parameters, and anomaly feature parameters. This module performs comprehensive analysis of the collected data, extracting feature parameters from different dimensions using various algorithms and methods. For example, for the historical log record set, it may be necessary to analyze dimensions such as log time, type, and content to determine log feature parameters.

[0023] The monitoring strategy generation module is configured to use log feature parameters, performance feature parameters, and anomaly feature parameters as index keys to perform a centralized search within the monitoring strategy configuration space to determine the target monitoring strategy. This module uses the extracted feature parameters as indexes to find the most suitable monitoring strategy within the pre-built monitoring strategy configuration space. The monitoring strategy configuration space stores a large number of sample monitoring strategies, allowing for quick location of the appropriate strategy using the index key.

[0024] The monitoring execution module is configured to adjust the monitoring status of the target cluster according to a preset monitoring rule set based on the target monitoring strategy, acquire periodic monitoring datasets, and use a monitoring result recognizer to identify the periodic monitoring datasets and obtain the target monitoring results. This module monitors the target cluster according to the determined target monitoring strategy and the preset monitoring rule set. During the monitoring process, monitoring data is acquired periodically to form periodic monitoring datasets, which are then analyzed and identified by the monitoring result recognizer to obtain the final target monitoring results. Example

[0025] In this embodiment, the feature analysis module performs feature dimension analysis on the historical log record set, historical performance index set, and historical abnormal event set to determine the corresponding feature parameters. Specifically, the module first performs data analysis on the historical log record set using the time window method. The historical log record set is arranged in timestamp order, with the timestamp used to identify the specific time each log record was generated. Sorting allows the log records to present a chronological order. Next, the historical log record in the middle position is found in the sorted set and used as the window baseline. The determination of the middle position needs to be calculated based on the number of log records in the set. If the number is odd, the log record in the middle position is the very middle one; if the number is even, the log record in the middle position can be either of the two middle records or determined according to a specific rule.

[0026] Calculate the first and second quarter time quantiles of the historical log record set. The calculation of time quantiles involves dividing the entire time range into four equal parts. The first quarter time quantile is the time point located at the first 25% of the time range, and the second quarter time quantile is the time point located at the last 25% of the time range. During the calculation, the start and end times of the time range must first be determined, then the log records are arranged in chronological order, and finally, the corresponding time points are found based on the proportion of records.

[0027] By using the first quarter-time quantile as the lower boundary value and the second quarter-time quantile as the upper boundary value, combined with the window baseline, a log time window is constructed. This constructed log time window can focus on log records within a certain period of time, facilitating subsequent analysis and processing. The window baseline is used to determine the center position of the window, while the lower and upper boundary values ​​define the time range of the window.

[0028] After obtaining the log time window, historical log records within that window that fall between the effective upper and lower limits are counted, resulting in a cleaned log record set. The effective upper limit is a boundary value formed by adjusting the maximum value of log records within the window downwards by a preset percentage, while the effective lower limit is a boundary value formed by adjusting the minimum value of log records within the window upwards by a preset percentage. The preset percentage can be set according to actual needs and data characteristics, such as 5% or 10%. This method filters out some outliers or extreme values ​​within the window, making the cleaned log record set more representative of log data under normal circumstances.

[0029] After cleaning, the dispersion of the cleaned log record set is calculated to obtain log characteristic parameters. Dispersion measurement measures the degree of dispersion of log record data; common dispersion indicators include variance, standard deviation, and range. By calculating these indicators, characteristic parameters that characterize the dispersion of log data can be obtained, which can be used to describe the distribution of log records.

[0030] The feature analysis module performs the same feature dimension parsing logic on the historical performance indicator set and the historical outlier event set to obtain performance feature parameters and outlier feature parameters. For the historical performance indicator set, a time window is first determined, and then the performance indicator data is cleaned within the window to remove outliers before calculating feature parameters such as dispersion. For the historical outlier event set, a similar process is followed: determining the time window, cleaning the data, and extracting feature parameters, thereby comprehensively analyzing the characteristics of various types of data.

[0031] Constructing a time window is a crucial step in the entire feature analysis process. It helps focus on data within a specific time period, improving the relevance and effectiveness of the analysis. Data cleaning ensures the quality of data input into subsequent analyses, reducing the impact of outliers on the results. Dispersion measurement quantitatively characterizes the distribution features of the data, providing a reliable basis for generating subsequent monitoring strategies.

[0032] By performing detailed feature dimension analysis on the historical log record set, historical performance index set, and historical anomaly event set, the feature analysis module can accurately determine log feature parameters, performance feature parameters, and anomaly feature parameters. These parameters will serve as an important basis for generating subsequent monitoring strategies, laying the foundation for the effective operation of the unified logging and monitoring system across the entire federated multi-cluster environment. In practical applications, the parameters and processing methods in each step can be flexibly adjusted according to different cluster characteristics and monitoring needs to adapt to different scenarios. For example, the preset ratio can vary for different types of log records; the method for calculating the dispersion of different performance indices can also be selected based on their data characteristics. This flexible processing approach enables the feature analysis module to better adapt to complex federated multi-cluster environments and extract more representative feature parameters. Example

[0033] In this embodiment, the main function of the monitoring policy generation module is to construct a monitoring policy configuration space so that target monitoring policies can be retrieved based on feature parameters. The specific implementation of this module is as follows: it requires collecting multiple sample log feature parameters, multiple sample performance feature parameters, and multiple sample anomaly feature parameters, while simultaneously acquiring multiple sample monitoring policies corresponding to these feature parameters. These data are then used together as the construction data. The sample data sources are diverse, collected from different federated multi-cluster environments, covering clusters of different sizes and business types, to ensure the broad applicability of the subsequently constructed monitoring policy configuration space. For example, sample log feature parameters may include the time distribution of different types of logs, the keyword frequency of log content, etc.; sample performance feature parameters may involve the changing trends of indicators such as CPU utilization, memory usage, and network bandwidth; sample anomaly feature parameters may include the frequency and duration of various abnormal events. The sample monitoring policy is a specific monitoring scheme formulated for different combinations of feature parameters, such as an alarm policy for specific log anomalies, or threshold settings for performance indicator fluctuations.

[0034] After acquiring the data, a knowledge graph space needs to be pre-constructed. The root node of this knowledge graph space is set as R, from which three main branches extend: log feature parameters, performance feature parameters, and anomaly feature parameters. This tree-like structure design follows the classification logic of data features in federated multi-cluster monitoring, using logs, performance, and anomalies as the three main categories of key features as the primary dimensions, facilitating subsequent structured organization and retrieval of the data. The root node R, as the core of the entire knowledge graph, unifies all branches, while the three branches correspond to different data domains that the monitoring system needs to focus on, forming a hierarchical spatial architecture.

[0035] The constructed data is input into a pre-constructed knowledge graph space. During the input process, each sample log feature parameter, sample performance feature parameter, and sample anomaly feature parameter is mapped to the corresponding branch in the knowledge graph space according to its category. For example, a sample log feature parameter will be associated with a specific position under the log feature parameter branch, and the same applies to sample performance feature parameters and sample anomaly feature parameters. Through this mapping method, each piece of constructed data generates a corresponding sample knowledge node in the knowledge graph space. These sample knowledge nodes do not exist in isolation; they form specific connections through the correlation of feature parameters, thereby constructing a complex network structure in the knowledge graph space.

[0036] After generating sample knowledge nodes, these nodes need to be labeled using corresponding sample monitoring strategies. The labeling process essentially establishes a mapping relationship between knowledge nodes and monitoring strategies, that is, it clarifies the sample monitoring strategy corresponding to each sample knowledge node. For example, when a sample knowledge node represents a specific combination of log feature parameters, performance feature parameters, and anomaly feature parameters, the labeling operation will bind the sample monitoring strategy for that combination to that node. Through this labeling, each node in the knowledge graph space has a clear monitoring strategy orientation, thus forming a monitoring strategy configuration space.

[0037] In constructing the monitoring strategy configuration space, it is crucial to ensure the completeness and accuracy of the sample data. On one hand, the amount of sample data collected must be large enough to cover various combinations of characteristic parameters that may occur in a federated multi-cluster environment. If the amount of sample data is too small, the monitoring strategy configuration space may be incomplete and unable to cope with complex real-world monitoring scenarios. On the other hand, the formulation of sample monitoring strategies needs to be combined with actual monitoring needs and experience to ensure that the labeled nodes can provide effective monitoring guidance. For example, during the labeling process, the sample monitoring strategies need to be reviewed by personnel with extensive cluster management experience to ensure the rationality and effectiveness of the strategies.

[0038] The construction of a knowledge graph space also requires consideration of data update mechanisms. As the federated multi-cluster environment continues to change, new feature parameters and monitoring strategies will constantly emerge. Therefore, the monitoring strategy configuration space needs to be dynamically updated. For example, when new sample data is collected, it needs to be promptly input into the knowledge graph space to generate new sample knowledge nodes and annotate them, thereby updating and expanding the monitoring strategy configuration space. This dynamic update mechanism ensures that the monitoring strategy configuration space remains synchronized with the actual cluster environment, thereby improving the adaptability and effectiveness of the monitoring system.

[0039] The monitoring policy configuration space constructed through the above steps can store a large number of feature parameter combinations and their corresponding monitoring policies in a structured manner. During the subsequent monitoring policy generation process, the monitoring policy generation module can quickly retrieve suitable target monitoring policies from this configuration space using log feature parameters, performance feature parameters, and anomaly feature parameters as index keys. This knowledge graph-based construction method leverages the advantages of graph structures in representing complex relationships and enabling rapid retrieval, efficiently handling the monitoring policy generation problem in a federated multi-cluster environment, and providing solid policy support for achieving unified logging and monitoring. Example

[0040] In this embodiment, after constructing the monitoring policy configuration space, the monitoring policy generation module needs to further determine the target monitoring policy from it. The specific implementation is as follows: Three special branch paths need to be defined in the monitoring policy configuration space. The first path is a branch path that passes through log feature parameters and is parallel to the performance-anomaly branch plane, denoted as the first path. Here, "passing through log feature parameters" means that the path uses log feature parameters as one of its key nodes, and "parallel to the performance-anomaly branch plane" means that the path's direction in the knowledge graph space is consistent with the extension direction of the performance-anomaly branch plane, without intersecting or forming an angle with it.

[0041] The second path is a branch path that passes through the performance characteristic parameters and is parallel to the log-anomaly branch plane; it is called the second path. Similarly, this path uses the performance characteristic parameters as key nodes, and its extension direction is parallel to the log-anomaly branch plane, meaning it maintains a relatively fixed distance and direction relationship with this plane in spatial structure. The third path is a branch path that passes through the anomaly characteristic parameters and is parallel to the log-performance branch plane; it is called the third path. It uses the anomaly characteristic parameters as key nodes, and its path direction is parallel to the log-performance branch plane.

[0042] After defining the three paths, three branch planes need to be determined: the performance-anomaly branch plane, the log-anomaly branch plane, and the log-performance branch plane. The performance-anomaly branch plane consists of performance feature parameter branches and anomaly feature parameter branches, encompassing the relationship between performance features and anomaly features. The log-anomaly branch plane consists of log feature parameter branches and anomaly feature parameter branches, reflecting the association between log features and anomaly features. The log-performance branch plane consists of log feature parameter branches and performance feature parameter branches, embodying the interrelationship between log features and performance features.

[0043] The subspace enclosed by these three branch planes and the three previously defined paths (the first path, the second path, and the third path) is defined as the configuration subgraph. In this process, the spatial relationship between each plane and the path needs to be clearly defined to ensure that the enclosed subspace has clear boundaries and limits. The configuration subgraph is a specific region within the monitoring strategy configuration space, containing multiple configuration sample knowledge nodes. These nodes are selected from the existing sample knowledge nodes and are located within this subspace.

[0044] After obtaining the configuration subgraph, a centralized search needs to be conducted on multiple configuration sample knowledge nodes to determine the target configuration sample knowledge node. This centralized search process must follow certain rules and methods. For example, a similarity-based search can be used, calculating the similarity between the input log feature parameters, performance feature parameters, and anomaly feature parameters and the nodes in the configuration subgraph, finding the node with the highest similarity as the target configuration sample knowledge node. Alternatively, a hierarchical search can be used, filtering according to the importance of the feature parameters, gradually narrowing the search scope until the target node is finally determined.

[0045] When defining paths and planes, it's crucial to accurately grasp the structural characteristics of the knowledge graph space. For instance, determining whether the first path is parallel to the performance-anomaly branch plane requires analyzing the relationships between nodes on the path and nodes within the performance-anomaly branch plane, starting from the knowledge graph's topology, to ensure the path remains parallel to the plane in spatial extension. This necessitates a clear understanding of the knowledge graph's construction methods and node connection relationships to avoid deviations in the configuration of subgraphs due to inaccurate path definitions.

[0046] The construction of the configuration subgraph needs to ensure that it accurately reflects the relationships between feature parameters. For example, the subspace enclosed by the performance-anomaly branch plane and the first, second, and third paths should cover all possible combinations of log feature parameters with other feature parameters while maintaining parallelism with the performance-anomaly feature plane. In this way, the configuration sample knowledge nodes in the configuration subgraph can represent a class of monitoring scenarios with similar combinations of feature parameters, providing a more targeted scope for subsequent retrieval.

[0047] The effectiveness of a centralized retrieval process depends on the rationality of the retrieval algorithm. When designing a retrieval algorithm, the weighting of feature parameters needs to be considered. Different feature parameters may have varying degrees of importance in actual monitoring scenarios. Therefore, when calculating similarity or performing filtering, each feature parameter needs to be assigned an appropriate weight to ensure that the retrieval results better reflect actual needs. For example, in some scenarios, the importance of anomaly feature parameters may be higher than that of log feature parameters and performance feature parameters. Therefore, during retrieval, the weight of anomaly feature parameters can be increased, making the retrieval results more likely to match nodes with similar anomaly feature parameters.

[0048] The update mechanism for the configuration subgraph also needs to be considered. As the monitoring strategy configuration space is continuously updated, the nodes in the configuration subgraph will also change. Therefore, the configuration subgraph needs to be maintained and updated regularly to ensure that the configuration sample knowledge nodes within it reflect the latest feature parameter combinations and monitoring strategies. For example, when a new sample knowledge node is added to the monitoring strategy configuration space, it needs to be determined whether it falls within the scope of the configuration subgraph. If it does, it should be included in the configuration subgraph to enrich the retrieval resources.

[0049] Through the above steps, the monitoring strategy generation module can accurately locate the configuration subgraph within the monitoring strategy configuration space, and determine the target configuration sample knowledge node through centralized retrieval. Then, the sample monitoring strategy corresponding to that node is used as the target monitoring strategy. This method narrows the retrieval scope and improves retrieval efficiency by dividing the complex knowledge graph space into specific subspaces. Simultaneously, by utilizing the spatial relationships between feature parameters and paths / planes, it ensures the relevance and accuracy of the retrieval results, better meeting the unified logging and monitoring needs of federated multi-cluster environments. In practical applications, the definition of paths and planes, as well as the parameter settings of the retrieval algorithm, need to be flexibly adjusted according to the specific cluster environment and monitoring requirements to optimize the target monitoring strategy generation process. Example

[0050] In this embodiment, after determining the configuration subgraph, the monitoring strategy generation module further determines the target configuration sample knowledge node by constructing a central neighborhood graph and performing centralized retrieval. The specific implementation is as follows: The central configuration sample knowledge node is extracted from the configuration subgraph. This node is typically determined based on the distribution of nodes in the configuration subgraph. For example, the node located at the geometric center of the configuration subgraph, or the node with the highest correlation to other nodes in the feature parameter space, can be used as the central configuration sample knowledge node. For instance, assuming there are multiple nodes in the configuration subgraph, each corresponding to a set of log feature parameters, performance feature parameters, and anomaly feature parameters, the node corresponding to the central location can be determined as the central configuration sample knowledge node by calculating the average coordinates of these nodes in the feature parameter space.

[0051] Starting with the central configuration sample knowledge node, a central neighborhood graph is constructed according to a preset centralized retrieval range. This preset centralized retrieval range can be a distance threshold in the feature parameter space, such as setting an Euclidean distance threshold to include all nodes whose distance to the central node is less than this threshold in the central neighborhood graph; or it can be a hierarchical range in the graph, such as expanding N layers of nodes outward from the central node to form a neighborhood graph. The central neighborhood graph presents a tree-like subgraph structure, radiating outward from the central configuration sample knowledge node and including configuration sample knowledge nodes that have a certain correlation with the central node.

[0052] After constructing the central neighborhood graph, it is necessary to count the number of configured sample knowledge nodes within it. The number of nodes directly reflects the size of the neighborhood graph. Simultaneously, the node density of the central neighborhood graph needs to be calculated. Node density is typically calculated as the ratio of the number of nodes to the volume of the feature parameter space occupied by the neighborhood graph. This metric measures the density of nodes in the neighborhood graph. Dividing the number of nodes by the node density yields the central neighborhood concentration. This concentration metric reflects the degree of concentration of node distribution in the central neighborhood graph; a higher value indicates a more concentrated distribution of nodes within the neighborhood.

[0053] A knowledge node from the edge of the central neighborhood graph is randomly selected as the first retrieval configuration node. Edge nodes in the central neighborhood graph are those far from the central node in the feature parameter space or located on the outermost layer of the tree structure. Selecting an edge node as the starting point helps explore the node distribution in different regions of the neighborhood graph. After selecting the first retrieval configuration node, the first retrieval neighborhood concentration is constructed using this node as the center, following the same rules and scope as when constructing the central neighborhood graph. This involves counting the number of nodes in the first retrieval neighborhood graph, calculating its node density, and thus obtaining the first retrieval neighborhood concentration.

[0054] Determine if the concentration of the first search neighborhood is greater than or equal to the concentration of the central neighborhood. If so, it indicates that the concentration of node distribution in the neighborhood graph centered on the first search configuration node is not less than that in the central neighborhood graph, suggesting that the node may be in a more favorable search region. In this case, update the first search configuration node as the new search starting point and continue the concentrated search as described above. This involves selecting nodes from the edge of the neighborhood graph of the new starting point, calculating the new neighborhood concentration, and making a judgment. This process is repeated until the preset number of concentrated searches is met. The preset number of concentrated searches can be set according to actual needs, such as 5 or 10 times. When this number is reached, the search configuration node obtained in the last search is used as the target configuration sample knowledge node.

[0055] For example, assuming the preset number of centralized searches is 3, the first time node A is selected from the edge of the central neighborhood graph, and its neighborhood concentration is calculated to be greater than the central neighborhood concentration, so node A is taken as the new starting point; the second time node B is selected from the neighborhood edge of node A, and its neighborhood concentration is calculated to be greater than the neighborhood concentration of the current starting point, so node B is updated; the third time node C is selected from the neighborhood edge of node B, and after the preset number of searches is reached, node C is taken as the target configuration sample knowledge node.

[0056] Throughout the retrieval process, it is essential to ensure that the construction rules of the central neighborhood map are consistent with those of the retrieved neighborhood map. For example, the distance threshold or hierarchical range should remain the same to guarantee the comparability of neighborhood concentration calculations. Simultaneously, the selection of edge nodes should be randomized to avoid retrieval bias caused by fixed selection methods, ensuring that the retrieval process comprehensively explores different regions of the neighborhood map.

[0057] Furthermore, the setting of the preset centralized search range needs to comprehensively consider the size of the feature parameter space and the node distribution density. If the range is set too small, the neighborhood graph may contain too few nodes, making it impossible to fully explore suitable nodes; if the range is set too large, the neighborhood graph may become too large, increasing computational complexity. Therefore, the preset centralized search range needs to be reasonably adjusted according to the actual situation of the configured subgraph to balance search efficiency and search quality.

[0058] By employing this retrieval method based on a central neighborhood graph and neighborhood concentration comparison, the monitoring strategy generation module can progressively explore better retrieval areas within the configuration subgraph, ultimately determining the target configuration sample knowledge node. This retrieval method utilizes the concentration of node distribution as a guide, avoiding blind searching and improving the targeting and efficiency of the retrieval. It can quickly locate monitoring strategy nodes with a high degree of matching with current feature parameters within a complex configuration subgraph, providing precise policy support for unified logging and monitoring of federated multi-cluster systems. In practical applications, parameters such as the method for determining the central configuration sample knowledge node, the preset centralized retrieval range, and the number of retrievals can be dynamically optimized according to changes in the cluster environment and adjustments to monitoring needs, adapting to different monitoring scenarios. Example

[0059] In this embodiment, when the monitoring strategy generation module performs centralized retrieval, if it encounters a situation where the concentration of the retrieval neighborhood does not meet expectations, it needs to execute a corresponding failure handling mechanism. The specific implementation is as follows: When it is determined that the concentration of the first retrieval neighborhood is not greater than or equal to the concentration of the central neighborhood, it indicates that the concentration of node distribution in the neighborhood graph centered on the current first retrieval configuration node is lower than that of the central neighborhood graph. At this time, the retrieval update failure count, which was initially set to 0, needs to be updated to 1. This count is used to record the number of consecutive retrieval failures so as to determine whether the retrieval strategy needs to be adjusted later.

[0060] After updating the failure count, a new configuration sample knowledge node is randomly selected from the edge of the central neighborhood graph as the new first retrieval configuration node, and the centralized retrieval analysis is carried out again. For example, assuming that the neighborhood concentration of the first selected edge node A is less than the central neighborhood concentration, the failure count is recorded as 1. Then, another unselected node B on the edge of the central neighborhood graph is selected as the new retrieval starting point, the neighborhood concentration of node B is calculated and judged.

[0061] Each time edge nodes are reselected, it is necessary to ensure that the selected nodes are random, avoiding repeated selection of nodes from the same region, so as to ensure that the retrieval process can cover different edge regions of the central neighborhood map. For example, if the central neighborhood map has a tree structure and edge nodes are distributed at the ends of different branches, each selection can be made randomly from different branches to increase the comprehensiveness of the retrieval.

[0062] When the search update failure count exceeds the preset maximum search update failure count, it indicates that a node with better neighborhood concentration has not been found after multiple attempts. In this case, the central configuration sample knowledge node is used as the target configuration sample knowledge node. The preset maximum search update failure count can be set according to actual needs, such as 3 or 5 times. The setting of this value needs to comprehensively consider search efficiency and search accuracy. For example, if the preset maximum search update failure count is 3 times, when the failure count reaches 4 times (exceeding the preset value), the edge node search is stopped, and the central node is directly used as the target.

[0063] To illustrate with a specific example: Assume that the central configuration sample knowledge node in the configuration subgraph is C, and the preset maximum retrieval update failure count is 3. First, node D is selected from the edge of the central neighborhood graph. Its neighborhood concentration is less than the central neighborhood concentration, and the failure count is recorded as 1. Second, node E is selected, but the neighborhood concentration still does not meet the requirement, and the failure count is recorded as 2. Third, node F is selected, but its neighborhood concentration is still lower than the central one, and the failure count is recorded as 3. Fourth, node G is selected. At this point, the failure count reaches 4 (exceeding the preset 3), so the central node C is selected as the target configuration sample knowledge node.

[0064] Throughout the failure handling process, it is necessary to accurately record the retrieval update failure count to ensure the correctness of the counting logic. When reselecting edge nodes after each failure, a random selection principle must be followed to avoid the retrieval getting stuck in localized areas due to a fixed selection method, thus failing to cover more possibilities. For example, in a tree-structured central neighborhood graph, edge nodes may be distributed across different levels and branches; random selection ensures that each attempt explores different regions.

[0065] Setting the preset maximum search update failure count requires balancing search efficiency and strategy accuracy. If this value is set too low, it may lead to premature adoption of central nodes without sufficient exploration, affecting the accuracy of the strategy; if set too high, it will increase the number of searches and reduce efficiency. Therefore, in practical applications, this value can be dynamically adjusted based on factors such as the node distribution density of the configuration subgraph and the complexity of the feature parameter space. For example, when the node distribution in the configuration subgraph is relatively uniform, the preset value can be appropriately reduced; when the node distribution varies significantly, the preset value can be increased to increase the number of explorations.

[0066] Furthermore, after designating the central configuration sample knowledge node as the target node, it is also necessary to consider whether the sample monitoring strategy corresponding to the central node meets the actual monitoring requirements. Although the central node may not be optimal in terms of neighborhood concentration, as the center of the configuration subgraph, it usually represents a typical combination of feature parameters, and the corresponding monitoring strategy has a certain degree of universality. For example, the central node may correspond to the feature parameter combinations of most common monitoring scenarios, and its associated monitoring strategy can be applied as a general strategy in general situations.

[0067] When implementing a failure handling mechanism, it is also necessary to pay attention to its coordination with other retrieval steps. For example, if the failure count does not exceed a preset value, the steps of edge node selection and neighborhood concentration calculation should continue to ensure the continuity of the retrieval process. At the same time, each time a new edge node is selected, its neighborhood graph should be reconstructed and its concentration calculated to ensure the accuracy of the data.

[0068] Through this failure handling mechanism, the monitoring strategy generation module can address situations with suboptimal neighborhood concentration during centralized retrieval, avoiding policy generation stagnation caused by multiple retrieval failures. This mechanism ensures exploration and attempts at nodes with better strategies while limiting the time cost of invalid retrievals by setting a maximum number of failures, ultimately achieving a balance between efficiency and accuracy. In actual monitoring scenarios with federated multi-cluster environments, due to the complexity and dynamism of the cluster environment, various combinations of characteristic parameters may occur. This mechanism can flexibly handle different retrieval situations, ensuring the stability and reliability of monitoring strategy generation. In practical applications, the parameters in the failure handling mechanism (such as the preset maximum retrieval update failure count and the edge node selection method) can be optimized by combining historical monitoring data and real-time status of the cluster to further improve the effectiveness of monitoring strategy generation.

[0069] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0070] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A unified logging and monitoring system for federated multi-cluster systems, characterized in that, The system includes: The log data acquisition module is configured to perform cross-cluster interface data retrieval to obtain the historical log record set, historical performance index set, and historical abnormal event set of the cross-cluster interface for tracking the target cluster within a preset observation period. The feature analysis module is configured to traverse the set of historical log records, the set of historical performance indicators, and the set of historical abnormal events to perform feature dimension parsing and determine log feature parameters, performance feature parameters, and abnormal feature parameters. The monitoring strategy generation module is configured to use the log feature parameters, performance feature parameters, and anomaly feature parameters as index keys to perform a centralized search within the monitoring strategy configuration space to determine the target monitoring strategy. The monitoring execution module is configured to perform monitoring status control on the target cluster according to the target monitoring strategy and a preset monitoring rule set, obtain a periodic monitoring dataset, and use a monitoring result recognizer to identify the periodic monitoring dataset to obtain the target monitoring result.

2. The unified logging and monitoring system for federated multi-cluster systems as described in claim 1, characterized in that, The feature analysis module is also configured to: The historical log record set is parsed using the time window method to obtain the log time window; Collect historical log records within the log time window that fall between the effective upper limit and the effective lower limit, and obtain a cleaned log record set. The effective upper limit is the boundary value formed by adjusting the maximum value in the window downward by a preset ratio, and the effective lower limit is the boundary value formed by adjusting the minimum value in the window upward by a preset ratio. The discreteness of the cleaned log record set is calculated to obtain the log feature parameters; The same feature dimension parsing logic is applied to the set of historical performance indicators and the set of historical abnormal events to obtain the performance feature parameters and abnormal feature parameters.

3. The unified logging and monitoring system for federated multi-cluster systems as described in claim 2, characterized in that, The feature analysis module is also configured to: The historical log records are arranged in order of timestamp, and the historical log record in the middle position is used as the window baseline. Calculate the first quarter time quantile and the second quarter time quantile of the historical log record set; The log time window is constructed by using the first quarter time quantile as the lower boundary value and the second quarter time quantile as the upper boundary value, combined with the window baseline.

4. The unified logging and monitoring system for federated multi-cluster systems as described in claim 1, characterized in that, The monitoring strategy generation module is also configured to: Multiple sample log feature parameters, multiple sample performance feature parameters, and multiple sample anomaly feature parameters, as well as corresponding multiple sample monitoring strategies, are collected as construction data. A pre-constructed knowledge graph space is provided, wherein the root node of the knowledge graph space is R, the first branch is log feature parameters, the second branch is performance feature parameters, and the third branch is anomaly feature parameters. The constructed data is input into the knowledge graph space to obtain multiple sample knowledge nodes, and the multiple sample knowledge nodes are labeled using the multiple sample monitoring strategies to obtain the monitoring strategy configuration space.

5. The unified logging and monitoring system for federated multi-cluster systems as described in claim 4, characterized in that, The monitoring strategy generation module is also configured to: The branch path in the monitoring strategy configuration space that passes through the log feature parameters and is parallel to the performance-anomaly branch plane is taken as the first path; The branch path in the monitoring strategy configuration space that passes through the performance characteristic parameters and is parallel to the log-anomaly branch plane is taken as the second path; The branch path in the monitoring strategy configuration space that passes through the abnormal feature parameters and is parallel to the log-performance branch plane is taken as the third path; The subspace enclosed by the performance-anomaly branch plane, the log-anomaly branch plane, the log-performance branch plane, the first path, the second path, and the third path is used as a configuration subgraph, wherein the configuration subgraph contains multiple configuration sample knowledge nodes; A centralized search is performed on the multiple configuration sample knowledge nodes to determine the target configuration sample knowledge node, and the sample monitoring strategy corresponding to the target configuration sample knowledge node is used as the target monitoring strategy.

6. The unified logging and monitoring system for federated multi-cluster systems as described in claim 5, characterized in that, The monitoring strategy generation module is also configured to: Extract the central configuration sample knowledge node of the configuration subgraph, and construct a central neighborhood graph with the central configuration sample knowledge node as the starting point and according to a preset centralized retrieval range. The central neighborhood graph is a tree-like subgraph constructed with the central configuration sample knowledge node as the core and the preset centralized retrieval range as the expansion radius. The number of configured sample knowledge nodes in the central neighborhood graph is counted, and the counted value is compared with the node density of the central neighborhood graph to obtain the central neighborhood concentration. Randomly select a configuration sample knowledge node from the edge of the central neighborhood graph as the first retrieval configuration node, and construct the first retrieval neighborhood concentration of the first retrieval configuration node; Determine whether the concentration of the first search neighborhood is greater than or equal to the concentration of the central neighborhood. If so, update the first search configuration node as the starting point and continue to carry out centralized search until the preset number of centralized searches is met. Then, take the search configuration node obtained in the last search as the target configuration sample knowledge node.

7. The unified logging and monitoring system for federated multi-cluster systems as described in claim 6, characterized in that, The monitoring strategy generation module is also configured to: If this is not the case, the initial value of the retrieval update failure count of 0 will be updated to 1, and a configuration sample knowledge node will be randomly selected from the edge of the central neighborhood graph as the first retrieval configuration node to carry out centralized retrieval analysis. When the retrieval update failure count is greater than the preset maximum retrieval update failure count, the central configuration sample knowledge node will be used as the target configuration sample knowledge node.

8. A unified logging and monitoring method for federated multi-cluster systems, characterized in that: The method includes: Perform cross-cluster interface data retrieval to obtain the historical log record set, historical performance index set, and historical abnormal event set of the cross-cluster interface for tracking the target cluster within a preset observation period; The historical log record set, historical performance index set, and historical abnormal event set are traversed to perform feature dimension parsing to determine log feature parameters, performance feature parameters, and abnormal feature parameters; Using the log feature parameters, performance feature parameters, and anomaly feature parameters as index keys, a centralized search is conducted within the monitoring policy configuration space to determine the target monitoring policy. Based on the target monitoring strategy, the target cluster is monitored and its status is adjusted according to a preset monitoring rule set to obtain a periodic monitoring dataset. The periodic monitoring dataset is then identified using a monitoring result recognizer to obtain the target monitoring result.

9. The unified logging and monitoring method for federated multi-cluster networks as described in claim 8, characterized in that, The step of traversing the historical log record set, historical performance indicator set, and historical abnormal event set to perform feature dimension parsing and determine log feature parameters, performance feature parameters, and abnormal feature parameters specifically includes: The historical log record set is parsed using the time window method to obtain the log time window; Collect historical log records within the log time window that fall between the effective upper limit and the effective lower limit, and obtain a cleaned log record set. The effective upper limit is the boundary value formed by adjusting the maximum value in the window downward by a preset ratio, and the effective lower limit is the boundary value formed by adjusting the minimum value in the window upward by a preset ratio. The discreteness of the cleaned log record set is calculated to obtain the log feature parameters; The same feature dimension parsing logic is applied to the set of historical performance indicators and the set of historical abnormal events to obtain the performance feature parameters and abnormal feature parameters.

10. The unified logging and monitoring method for federated multi-cluster networks as described in claim 9, characterized in that, The step of using the time window method to parse the historical log record set and obtain the log time window specifically includes: The historical log records are arranged in order of timestamp, and the historical log record in the middle position is used as the window baseline. Calculate the first quarter time quantile and the second quarter time quantile of the historical log record set; The log time window is constructed by using the first quarter time quantile as the lower boundary value and the second quarter time quantile as the upper boundary value, combined with the window baseline.

Images