A unified method and system for observable log metrics and tracking data

Abstract

This invention discloses a unified method and system for observable log metrics and tracking data. The method includes: collecting various types of raw observation data, constructing a columnar storage structure after unified formatting, and performing hierarchical storage; subsequently optimizing the data layout using column-level metadata and a query pattern feature library, and implementing a multi-level indexing strategy; simultaneously establishing a causal graph between heterogeneous observation data through data cleaning, structure enhancement, and contextual information association, generating observation data with rich context; and finally constructing a high-performance query processing system based on a unified query engine and sublinear algorithms to achieve progressive result return and visual analysis. This invention significantly improves data retrieval efficiency and analytical depth, enabling operations and maintenance personnel to more quickly locate the root causes of system problems and effectively reducing the average time for fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a unified log metrics and tracking data observability method and system, applicable to the monitoring, log management and performance analysis of large-scale distributed systems. Background Technology

[0002] Observability technology is a core capability for the operation and maintenance of modern IT infrastructure and application systems, mainly comprising three pillars: log management, metric monitoring, and distributed tracing. With the widespread adoption of cloud-native architectures, microservices, and containerization technologies, enterprise IT environments are becoming increasingly complex, interactions between system components are increasing, and the amount of observational data generated is growing exponentially.

[0003] Currently, common observability solutions in the industry are mainly divided into three categories of independent systems: ELK Stack (Elasticsearch, Logstash, Kibana) or Loki for log data; time-series databases such as Prometheus and InfluxDB for metric data; and distributed tracing systems such as Jaeger and Zipkin for tracking data. These systems each provide specialized storage and query capabilities for different types of observation data, but they lack deep integration with each other.

[0004] In existing technologies, some solutions attempt to correlate different observational data through API aggregation or data forwarding, but the underlying infrastructure remains heterogeneous storage engines and query interfaces. In this architecture, different types of observational data employ their own storage formats and compression algorithms, use specialized query languages, and are often deployed on independent infrastructures. While this specialized design can provide optimization for specific data types, it also introduces significant technical drawbacks.

[0005] First, the heterogeneous observation systems create severe data silos, hindering in-depth correlation analysis and requiring switching between multiple systems for problem localization. Second, multiple systems lead to storage redundancy and resource waste, resulting in extremely high storage costs at petabyte-scale data volumes. Third, existing systems lack intelligent data lifecycle management and adaptive optimization capabilities, failing to dynamically adjust storage strategies based on data access patterns. Finally, most systems fail to effectively integrate front-end user experience data, resulting in a disconnect between back-end observations and actual user experience, making end-to-end performance analysis and problem diagnosis difficult. Summary of the Invention

[0006] This invention provides a unified method and system for the observability of log metrics and tracking data, enabling unified collection, processing, storage and analysis of observation data from different sources and types, reducing system complexity and storage costs, and improving the efficiency of problem diagnosis and performance analysis.

[0007] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0008] This invention provides a method for making unified log metrics and tracking data observable, including:

[0009] Acquire observation data from different sources, collect and parse the observation data, map the parsed observation data to a unified internal data model and establish correlation mappings between different types of data to generate a unified observation data stream;

[0010] Based on the unified observation data stream, the data is partitioned and columnarly divided, the data characteristics of each column are analyzed and the encoding method and compression algorithm are automatically selected, column-level metadata is created and maintained, and a columnar storage structure is generated.

[0011] Based on the columnar storage structure, data access is monitored and recorded, data access patterns are identified, data value scores are calculated, and data is divided into hot data, warm data, and cold data according to the data value scores and stored in different storage media to generate a hierarchical storage dataset.

[0012] Based on the hierarchical storage dataset, user query behavior is analyzed, query pattern characteristics are identified, indexing strategies are formulated for different data layers, multiple types of index structures are implemented and maintained, data layout is optimized, and an optimized data storage structure is generated.

[0013] Based on the unified observation data stream, cleaning and structure enhancement operations are performed on the observation data to realize the conversion from logs to indicators, add contextual information to the observation data, and generate observation data with rich context.

[0014] Based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, a unified query engine is implemented to optimize queries and manage resources, provide a visual interface, and generate a visual analysis platform.

[0015] Preferably, the step of collecting and parsing the observation data, mapping the parsed observation data to a unified internal data model, establishing correlation mappings between different types of data, and generating a unified observation data stream includes:

[0016] Based on different types of observation data, raw observation data is received through multiple interfaces to form an initial multi-source data stream; wherein, the observation data includes log data, indicator data, and tracking data;

[0017] Based on the initial multi-source data stream, the data format is automatically identified and key fields and metadata are extracted. For semi-structured logs, heuristic algorithms are applied to perform automatic pattern recognition and field extraction to generate a structured intermediate dataset.

[0018] Based on the structured intermediate dataset, different types of observation data are mapped to a unified internal data model. The unified internal data model includes timestamp fields, data source identifiers, resource attributes, tag sets, indicator values, log content, and tracking information, generating observation data entities in a unified format.

[0019] Based on the observation data entities in the unified format, metadata is automatically enhanced and a correlation mapping is established between different types of data to generate the unified observation data stream.

[0020] Preferably, the step of applying heuristic algorithms to semi-structured logs for automatic pattern recognition and field extraction to generate a structured intermediate dataset includes:

[0021] Based on the original log data stream in the initial multi-source data stream, a multi-dimensional feature space is established and multiple sorting methods are predefined to form multiple sorting sets;

[0022] Based on the multiple sorting sets, a parallel binary search strategy is executed in multiple sorting directions. Each binary search query determines a log subset half-space that conforms to a specific pattern feature, generating multiple half-space query results.

[0023] Based on the query results of the multiple half-spaces, the intersection of the multiple half-spaces is calculated to form a log pattern description and generate a half-space intersection model.

[0024] Based on the semi-space intersection model, the subsequent query strategy is dynamically adjusted. When the confidence of pattern recognition reaches a preset threshold, the query is terminated, and a log pattern model is generated.

[0025] Based on the log pattern model, the original log data stream is parsed and fields are extracted to generate the structured intermediate dataset.

[0026] Preferably, generating a columnar storage structure based on the unified observation data stream includes:

[0027] Based on the unified observation data stream, intelligent partitioning is performed according to the timestamp field and data source identifier in the unified internal data model. The data is divided into logical partitions and blocks by column to form a set of columnar data blocks.

[0028] Based on the set of columnar data blocks, the data characteristics of each column are analyzed in real time and the encoding method is automatically selected to generate the encoded column dataset;

[0029] Based on the encoded column dataset, compression algorithms are dynamically selected and applied according to data with different characteristics and system resource conditions to generate compressed columnar data blocks.

[0030] Based on the compressed columnar data blocks, column-level metadata is created and maintained. The column-level metadata includes data type, encoding method, compression algorithm and data block location, thereby generating the columnar storage structure.

[0031] Preferably, the step of intelligent partitioning based on the timestamp field and data source identifier in the unified internal data model, dividing the data into logical partitions and column-based blocks to form a set of columnar data blocks, includes:

[0032] Based on the unified observation data stream, the correlation between observation data in the unified internal data model is modeled as a graph structure, realizing a graph sampling algorithm with sublinear complexity and generating graph sampling results;

[0033] Based on the graph sampling results, a sublinear time edge counting algorithm and a community detection algorithm are applied to generate graph structure feature estimates.

[0034] Based on the graph structure feature estimation, a topology-aware data partitioning strategy is generated to ensure that highly correlated data identified in the graph structure feature estimation are assigned to the same or adjacent partitions, thereby generating a topology-aware partitioning scheme.

[0035] Based on the topology-aware partitioning scheme, the optimal size of each partition is adaptively determined and a progressive partitioning adjustment mechanism is implemented to generate the columnar data block set.

[0036] Preferably, the step of performing real-time analysis of the data characteristics of each column and automatically selecting the encoding method to generate the encoded column dataset includes:

[0037] Based on the column data in the columnar data block set, a multidimensional description space of data characteristics is constructed, and a characteristic space structure is generated.

[0038] Based on the feature space structure, an active learning strategy is implemented and a query sequence is designed to perform binary search in parallel across multiple feature dimensions, generating parallel search results.

[0039] Based on the parallel search results, the search results in different sorting directions are merged into a unified data characteristic description model to generate a comprehensive characteristic model;

[0040] Based on the comprehensive characteristic model, the optimal encoding scheme is automatically selected for each column of data in the columnar data block set to generate the encoded columnar dataset.

[0041] Preferably, the process of monitoring and recording data access, identifying data access patterns, calculating data value scores, and classifying data into hot data, warm data, and cold data based on the data value scores and storing them in different storage media to generate a hierarchical storage dataset includes:

[0042] Based on the columnar storage structure, the column-level metadata is used to continuously monitor and record data access, and data access patterns and trends are identified through time series analysis and machine learning algorithms to generate a data access feature model.

[0043] Based on the data access feature model, a data value score is calculated for each data partition and data block in the columnar storage structure. Based on the data value score, the data is divided into hot data, warm data, and cold data, and a data hierarchical configuration scheme is generated.

[0044] According to the data tiering configuration scheme, a multi-level storage architecture is implemented, which stores hot data, warm data, and cold data in different storage media. Data blocks in the columnar storage structure are automatically moved between different storage media according to changes in data tiers to generate the tiered storage dataset.

[0045] Based on the hierarchical storage dataset and the data access feature model, an intelligent retention strategy is implemented to automatically perform downsampling, aggregation, or cleaning operations on the cold data, generating an optimized dataset that has undergone intelligent lifecycle management.

[0046] Preferably, the step of identifying data access patterns and trends through time series analysis and machine learning algorithms to generate a data access feature model includes:

[0047] Based on the recorded data access information, a network topology describing the inter-block access relationships in the columnar storage structure is constructed, and a data access network topology is generated.

[0048] Based on the data access network topology, the data access behavior is parameterized into the Ising model, and an energy function is defined so that common access patterns in the data access network topology correspond to low-energy states, thereby generating a parameterized representation of the Ising model.

[0049] Based on the parameterized representation of the Ising model, an efficient learning algorithm based on very few samples is implemented to learn model parameters from a limited number of access logs and generate the learned model parameters.

[0050] Based on the learned model parameters, the access pattern prediction is optimized under hard constraints, and the future access probability prediction of data blocks in the columnar storage structure is output to generate the data access feature model.

[0051] Preferably, the step of calculating a data value score for each data partition and data block in the columnar storage structure, and dividing the data into hot data, warm data, and cold data based on the data value score to generate a data tiering configuration scheme includes:

[0052] Based on the columnar storage structure, the interrelationships between the observation data recorded in the column-level metadata are modeled as a network structure, and a triangle counting algorithm with sublinear complexity is implemented to generate triangle structure detection results.

[0053] Based on the triangular structure detection results, the local density and centrality of the data in the columnar storage structure are calculated, and the data with high density and high centrality in the triangular structure detection results are initially ranked in terms of value to generate a topological value assessment.

[0054] Based on the topological value assessment, a sublinear time feature extraction algorithm is designed to extract key features from the observation data in the columnar storage structure. The key features and the topological value assessment are combined to construct a comprehensive data value model and generate a data value score.

[0055] Based on the distribution of the data value scores, the optimal threshold for data stratification is dynamically determined through sublinear statistical estimation. The capacity allocation of each layer is optimized by comprehensively considering storage costs, access patterns in the data access feature model, and the distribution of the data value scores, thereby generating the data stratification configuration scheme.

[0056] Preferably, the step of analyzing user query behavior, identifying query pattern characteristics, formulating indexing strategies for different data layers, implementing and maintaining multiple types of index structures, optimizing data layout, and generating an optimized data storage structure includes:

[0057] Based on the hierarchical storage dataset and historical query logs, in-depth analysis of user query behavior is performed to identify query pattern features. Clustering algorithms are used to group similar queries and generate a query pattern feature library.

[0058] Based on the query pattern feature library and the hierarchical storage dataset, differentiated indexing strategies are formulated for different data layers, and dynamic indexing strategy schemes are generated.

[0059] Based on the dynamic indexing strategy, multiple types of index structures are implemented and maintained, the performance of each index structure is automatically monitored and the index parameters are dynamically adjusted to generate optimized and diversified index structures.

[0060] Based on the query pattern feature library and the optimized diversified index structure, a co-occurrence matrix of columns and fields in the query is constructed, and the co-occurrence frequency and relevance score between columns and fields are calculated. When the relevance score exceeds a preset association threshold, it is identified as a highly relevant data pair. The columns and fields in the highly relevant data pair are arranged in consecutive storage locations, the layout of time-series data in the hierarchical storage dataset is optimized, and the optimized data storage structure is generated.

[0061] Preferably, the step of performing in-depth analysis of user query behavior, identifying query pattern features, and using a clustering algorithm to group similar queries to generate a query pattern feature library includes:

[0062] Based on historical query logs, a relational network describing the relationships between query operations is constructed, the temporal pattern of queries is identified, and the sequential dependencies between query operations are captured to generate a relational network of query operations.

[0063] Based on the query operation association network, the query pattern is represented as an Ising model. An energy function reflecting the probability of the query pattern is defined so that common query patterns in the query operation association network correspond to low-energy state configurations, thereby generating the Ising model representation of the query pattern.

[0064] Based on the Ising model representation of the query pattern, an efficient learning algorithm based on minimum sample size is implemented to extract key pattern features from a limited number of query logs and generate a learned query pattern model.

[0065] Based on the learned query pattern model, a representative set of query patterns is generated under the premise of satisfying the constraints, realizing deep inference of query intent, predicting future query needs, optimizing the corresponding index and data layout strategies in the hierarchical storage dataset, and outputting the query pattern feature library containing intent annotations.

[0066] Preferably, the step of performing cleaning and structure enhancement operations on the observation data based on the unified observation data stream, realizing the transformation from logs to indicators, adding contextual information to the observation data, and generating observation data with rich context includes:

[0067] Based on the unified observation data stream, a configurable data processing pipeline framework is designed, and a configurable data pipeline definition is generated.

[0068] Based on the configurable data pipeline definition, cleaning and structuring enhancement operations are performed on the observation data in the unified observation data stream to generate a cleaned structured data stream.

[0069] Based on the cleaned structured data stream, the real-time conversion from logs to metrics is realized. Rules are defined to extract numerical metrics from the logs in the cleaned structured data stream and generate a log-derived metric dataset.

[0070] Based on the cleaned structured data stream and the indicator dataset derived from the logs, data association and context enrichment are achieved, service topology information and metadata are added to the observation data, and the observation data with rich context is generated.

[0071] Preferably, the process of implementing data association and enriching context by adding service topology information and metadata to the observation data to generate the observation data with rich context includes:

[0072] Construct a network structure that describes the potential correlation between different types of observation data in the cleaned structured data stream and the log-derived indicator dataset. Initialize the correlation strength using the timestamp field and data source identifier in the unified observation data stream to generate a heterogeneous observation data correlation network.

[0073] Based on the heterogeneous observation data association network, the association relationship between heterogeneous data is parameterized into an Ising model, prior constraints are introduced and modeled as hard constraints, and an Ising model representation of the association relationship is generated.

[0074] Based on the Ising model representation of the aforementioned association relationship, a small-sample association pattern learning algorithm is implemented to learn association patterns from a small number of known association instances and generate association pattern learning results.

[0075] Based on the learning results of the association patterns, association inference is performed under hard constraints. The probability that there is a real association between each pair of data entities in the heterogeneous observation data association network is calculated, and a consistent association network is generated.

[0076] Based on the consistent correlation network, a causal graph of heterogeneous observation data is constructed. The causal graph is then used to automatically add contextual information to the observation data in the cleaned structured data stream and the log-derived indicator dataset, thereby generating the observation data with rich context.

[0077] Preferably, the visualization and analysis platform for generating high-value observation data with rich context, based on the columnar storage structure, the optimized data storage structure, and the high-value observation data with rich context, includes:

[0078] Based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, a unified query engine supporting multiple query languages ​​is realized. The optimal execution path is selected according to the data characteristics in the column-level metadata and the optimized data storage structure to generate query execution capabilities.

[0079] Based on the query execution capability, multi-level query optimization is achieved. The query plan is automatically rewritten based on the query conditions and the data statistics in the column-level metadata, realizing vectorized execution and parallel processing. The delayed materialization technology of the columnar storage structure is adopted to realize the intelligent caching mechanism to cache hot data in the hierarchical storage dataset, thereby generating a high-performance query processing system.

[0080] Based on the high-performance query processing system, an adaptive query resource management mechanism is implemented, which dynamically allocates computing resources according to query complexity, the amount of data in the columnar storage structure, and system load, supports query truncation and progressive result return, and obtains a stable query service.

[0081] Based on the stable query service, an integrated front-end visualization interface is provided, enabling data exploration tools to allow users to interactively analyze the observation data with rich context. An alarm configuration interface and an investigation and analysis toolchain are provided to generate the visualization analysis platform.

[0082] Preferably, the multi-level query optimization, which automatically rewrites the query plan based on query conditions and data statistics in the column-level metadata, enables vectorized execution and parallel processing, employs the delayed materialization technique of the columnar storage structure, and implements an intelligent caching mechanism to cache hot data in the hierarchical storage dataset, thereby generating a high-performance query processing system, includes:

[0083] Before executing the full query, a sublinear algorithm is applied to estimate the size of the query results. An approximate value of the result set size is obtained by performing a random query on the optimized diversified index structure. The query execution strategy is optimized based on the estimated result set size, and a query strategy optimization scheme is generated.

[0084] Based on the query strategy optimization scheme, for analytical queries, an approximate query processing based on sublinear sampling is implemented. The sampling strategy is designed according to the query conditions and the statistical characteristics in the column-level metadata. When processing a very small proportion of data in the columnar storage structure, an approximate result is provided, and an approximate query result is generated.

[0085] Based on the query strategy optimization scheme, an optimization algorithm for sub-linear complexity is implemented for aggregation operations. The data statistical characteristics in the column-level metadata and the optimized diversified index structure are used to avoid full table scans and generate aggregation optimization results.

[0086] According to the query strategy optimization scheme, for complex queries involving the association of multiple observation data in the observation data with rich context, the sublinear graph algorithm is applied to accelerate the association query, realize the rapid identification of key paths and abnormal patterns in the causal graph, and generate accelerated association query results.

[0087] Based on the query strategy optimization scheme, the approximate query results, the aggregation optimization results, and the correlation query acceleration results, an adaptive query execution strategy is implemented, and a progressive processing method is adopted for complex queries to generate the high-performance query processing system.

[0088] This invention also provides an observable system for unified log metrics and tracking data, comprising:

[0089] The data acquisition and mapping module is used to acquire observation data from different sources, collect and parse the observation data, map the parsed observation data to a unified internal data model, establish correlation mappings between different types of data, and generate a unified observation data stream.

[0090] The columnar storage module is used to partition and block the data according to the unified observation data stream, analyze the data characteristics of each column and automatically select the encoding method and compression algorithm, create and maintain column-level metadata, and generate a columnar storage structure.

[0091] The hierarchical storage module is used to monitor and record data access according to the columnar storage structure, identify data access patterns, calculate data value scores, divide data into hot data, warm data and cold data according to the data value scores and store them in different storage media to generate hierarchical storage datasets.

[0092] The index optimization module is used to analyze user query behavior based on the hierarchical storage dataset, identify query pattern characteristics, formulate indexing strategies for different data layers, implement and maintain multiple types of index structures, optimize data layout, and generate optimized data storage structures.

[0093] The data augmentation module is used to perform cleaning and structured augmentation operations on the observation data based on the unified observation data stream, realize the conversion of logs to indicators, add contextual information to the observation data, and generate observation data with rich context.

[0094] The visualization analysis module is used to implement a unified query engine based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, to perform query optimization and resource management, provide a visualization interface, and generate a visualization analysis platform.

[0095] The observable method and system provided by this invention have the following beneficial effects:

[0096] 1. By using a unified data model, the problem of correlation and integration between different types of observation data was solved, data silos were eliminated, and the foundation for end-to-end analysis was laid;

[0097] 2. By employing columnar storage and adaptive compression technology, efficient data storage is achieved, significantly reducing storage costs and resource consumption;

[0098] 3. Through intelligent data lifecycle management, automatic tiered storage and retention strategy optimization of data are achieved, balancing performance and cost;

[0099] 4. By optimizing dynamic indexes and reordering data, query efficiency has been improved, enabling interactive-level responses for complex analytical queries;

[0100] 5. Through a unified query engine and visualization platform, it provides a consistent user experience and rich analytical capabilities, simplifying system operation and maintenance and problem diagnosis processes. Attached Figure Description

[0101] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1 A flowchart illustrating the method for making unified log metrics and tracking data observable in an embodiment of the present invention;

[0103] Figure 2 This is a schematic diagram of the structure of an observable system for unified log metrics and tracking data provided in an embodiment of the present invention. Detailed Implementation

[0104] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0105] Example 1:

[0106] like Figure 1 As shown, a unified log metric and tracking data observability method includes:

[0107] Step S1: Obtain observation data from different sources, collect and parse the observation data, map the parsed observation data to a unified internal data model and establish a correlation mapping between different types of data to generate a unified observation data stream;

[0108] Step S2: Based on the unified observation data stream, partition and columnar block the data, analyze the data characteristics of each column and automatically select the encoding method and compression algorithm, create and maintain column-level metadata, and generate a columnar storage structure;

[0109] Step S3: Based on the columnar storage structure, monitor and record data access, identify data access patterns, calculate data value scores, and divide the data into hot data, warm data, and cold data according to the data value scores and store them in different storage media to generate a hierarchical storage dataset.

[0110] Step S4: Based on the hierarchical storage dataset, analyze user query behavior, identify query pattern characteristics, formulate indexing strategies for different data layers, implement and maintain multiple types of index structures, optimize data layout, and generate an optimized data storage structure.

[0111] Step S5: Based on the unified observation data stream, perform cleaning and structure enhancement operations on the observation data to realize the conversion from logs to indicators, add context information to the observation data, and generate observation data with rich context.

[0112] Step S6: Based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, implement a unified query engine to optimize queries and manage resources, provide a visual interface, and generate a visual analysis platform.

[0113] Example 2:

[0114] Based on Example 1, the process of collecting and parsing the observation data, mapping the parsed observation data to a unified internal data model, establishing correlation mappings between different types of data, and generating a unified observation data stream includes:

[0115] Based on different types of observation data, raw observation data is received through multiple interfaces to form an initial multi-source data stream; wherein, the observation data includes log data, indicator data, and tracking data;

[0116] Based on the initial multi-source data stream, the data format is automatically identified and key fields and metadata are extracted. For semi-structured logs, heuristic algorithms are applied to perform automatic pattern recognition and field extraction to generate a structured intermediate dataset.

[0117] Based on the structured intermediate dataset, different types of observation data are mapped to a unified internal data model. The unified internal data model includes timestamp fields, data source identifiers, resource attributes, tag sets, indicator values, log content, and tracking information, generating observation data entities in a unified format.

[0118] Based on the observation data entities in the unified format, metadata is automatically enhanced and a correlation mapping is established between different types of data to generate the unified observation data stream.

[0119] Specifically, firstly, based on different types of observation data, various standardized data acquisition interfaces are implemented to receive raw observation data from diverse data sources. These interfaces may include the OpenTelemetry protocol interface, an open-source standardized observability data collection framework that supports multiple programming languages ​​and frameworks; the Prometheus remote write API, used to receive time-series metric data conforming to the Prometheus format; Fluentd or Logstash compatible interfaces, used to receive structured or semi-structured log data; and a REST API, providing a general HTTP interface for receiving observation data in various formats. The system receives three main categories of observation data through these interfaces: log data (text records documenting system events and application behavior), metric data (numerical time series describing system state and performance), and trace data (structured data recording the request flow path in the distributed system). This raw data forms the initial multi-source data stream, which serves as the input for subsequent processing.

[0120] The process begins with data format identification and parsing based on the received initial multi-source data stream. First, the system automatically identifies the data format type, such as JSON, XML, and plain text, and applies the corresponding parser to perform preliminary parsing, extracting key fields and metadata. For well-structured data (such as standard JSON formatted metrics), the system can directly parse field values. For semi-structured log data (such as free text logs interspersed with key-value pairs or structured information), the system applies heuristic algorithms for automatic pattern recognition and field extraction. This heuristic algorithm can identify patterns and regularities in the logs, converting unstructured text into a structured set of fields, greatly enhancing subsequent analysis capabilities. Through this step, the system transforms the raw data into a structured intermediate dataset, laying the foundation for subsequent unified model conversion.

[0121] Furthermore, based on the structured intermediate dataset obtained in the previous step, a unified data model transformation is performed. Different types of observation data are mapped to a predefined unified internal data model. This model is flexible enough to simultaneously represent various characteristics of log, metric, and trace data. Specifically, this unified model includes the following core fields: timestamp field (recording the precise time of data generation or collection), data source identifier (indicating the data producer, such as application name, hostname, etc.), resource attributes (describing the resources related to the data, such as container ID, process ID, etc.), tag set (a set of key-value pairs used for classifying and filtering data), metric value (for metric data, storing the actual measurement value), log content (for log data, storing the actual log message), and trace information (for trace data, storing trace-related fields such as span ID, trace ID, etc.). Through this mapping transformation, the system unifies observation data from different sources and formats into data entities with a consistent structure, facilitating subsequent storage and analysis.

[0122] Finally, based on the unified format observation data entities obtained in the previous step, metadata enhancement and association establishment are performed. Metadata information is automatically enhanced, including adding globally unique identifiers (to uniquely identify each data item within the system), calculating content hash values ​​(for quickly determining data duplication), extracting and standardizing time information (to ensure consistency in time representation), and adding data type tags (to clearly indicate the data type and source). More importantly, this step establishes association mappings between different types of data. For example, log data and trace data are associated using a common traceID, and metric data is associated with corresponding log and trace data using service names and time windows. This association allows users to seamlessly navigate from one type of observation data to other related types of data, greatly improving the efficiency of troubleshooting and performance analysis. Through these processes, the system ultimately outputs a unified observation data stream with rich and interconnected metadata, providing a unified data foundation for subsequent storage and analysis.

[0123] In the steps described above, a unified internal data model defines how to represent different types of observation data within the system. This model employs a flat structure, integrating all necessary information fields into a single, unified data entity. Crucially, the model must be flexible enough to accommodate the unique attributes of different data types, while also being standardized enough to support efficient storage and retrieval. Typically, the unified model uses standardized field naming and data type conventions to ensure data consistency. Furthermore, the model needs to support dynamic expansion, capable of handling new types of observation data or newly added data fields that may emerge in the future, without modifying the underlying storage structure. This unified representation enables the system to process and analyze different types of observation data in a consistent manner, forming the foundation for truly unified observability.

[0124] Example 3:

[0125] Based on Example 2, the step of applying heuristic algorithms to semi-structured logs for automatic pattern recognition and field extraction to generate a structured intermediate dataset includes:

[0126] Based on the original log data stream in the initial multi-source data stream, a multi-dimensional feature space is established and multiple sorting methods are predefined to form multiple sorting sets;

[0127] Based on the multiple sorting sets, a parallel binary search strategy is executed in multiple sorting directions. Each binary search query determines a log subset half-space that conforms to a specific pattern feature, generating multiple half-space query results.

[0128] Based on the query results of the multiple half-spaces, the intersection of the multiple half-spaces is calculated to form a log pattern description and generate a half-space intersection model.

[0129] Based on the semi-space intersection model, the subsequent query strategy is dynamically adjusted. When the confidence of pattern recognition reaches a preset threshold, the query is terminated, and a log pattern model is generated.

[0130] Based on the log pattern model, the original log data stream is parsed and fields are extracted to generate the structured intermediate dataset.

[0131] Specifically, based on the raw log data from the initial multi-source data stream, a multi-dimensional feature space for pattern recognition is constructed. This feature space is an abstract concept used to represent various structural features of the logs, with each dimension representing a possible structural characteristic, such as timestamp format, message body pattern, key-value pair distribution, etc. The system predefines multiple sorting methods within this feature space; for example, sorting by field frequency, character type distribution, or information entropy (information disorder). These different sorting methods form multiple sorting sets, providing multi-faceted exploration paths for subsequent parallel search strategies. Establishing such a feature space and sorting sets enables the system to simultaneously analyze log structure from multiple dimensions, improving the accuracy and efficiency of pattern recognition.

[0132] Next, based on the multiple sorted sets established in the previous step, a parallel binary search strategy is implemented for log pattern recognition. Unlike traditional sequential processing methods, binary queries are performed simultaneously in multiple sorting directions. Each query identifies a subset of logs that conforms to a specific pattern characteristic, i.e., a "half-space". For example, it might simultaneously ask questions such as "Does this log contain timestamps in ISO8601 format?", "Does this log contain JSON fragments?", and "Does this log contain a specific error code format?". The answer to each question divides the log space into two parts. This parallel binary search strategy greatly improves the efficiency of pattern recognition because each query eliminates a large number of possible patterns that do not meet the conditions, allowing the actual structural pattern of the log to be identified with fewer queries. Through this step, multiple half-space query results are generated, each representing the structural characteristics of the log in a specific dimension.

[0133] Then, based on the multiple half-space query results obtained in the previous step, the intersection of these half-spaces is calculated to form a more accurate log pattern description. Each half-space query result represents a feature constraint of the log in a certain dimension, and the intersection of multiple half-spaces represents a log pattern that simultaneously satisfies all these constraints. For example, if one half-space represents "containing timestamps" and another represents "containing user IDs," then their intersection represents a log pattern that "contains both timestamps and user IDs." By calculating the intersection of these half-spaces, the system can construct a more accurate and complete log pattern description, capturing the multi-dimensional structural features of the logs. This method does not require pre-enumerating all possible log patterns, nor does it require generating synthetic data for matching, thus enabling efficient handling of various complex log formats. Through this step, the system generates a half-space intersection model that accurately describes the structural pattern of the logs.

[0134] Next, based on the semi-space intersection model, the subsequent query strategy is dynamically adjusted to optimize the pattern recognition process. Existing query results and pattern estimates are analyzed, the expected information gain for each possible query is calculated, and the query direction that provides the maximum information gain is prioritized. Simultaneously, an early termination mechanism is implemented: when the confidence level of pattern recognition (the degree of certainty about the pattern estimate) reaches a preset threshold, the query process is terminated early, without executing all possible queries. This adaptive query strategy ensures that the system can obtain a sufficiently accurate log pattern description with a minimal number of queries, significantly improving the efficiency of pattern recognition. Especially for complex log formats, this optimization can reduce the number of queries from exponential to logarithmic levels, enabling efficient processing of large numbers of logs with different formats in real-world environments. Through this step, the system generates the final log pattern model, accurately describing the structural characteristics of the logs.

[0135] Finally, based on the generated log pattern model, the original log data stream is parsed and its fields are extracted. The system applies the log pattern model to each log record, identifying the structured parts and individual fields, transforming unstructured text into a structured set of fields. For example, for a log record containing a timestamp, service name, level, and message, the system can accurately extract these fields and assign them appropriate names. The system can also handle minor changes in log format and anomalies, ensuring the stability and accuracy of the parsing results. Through this structuring process, text logs that were originally difficult to query and analyze are transformed into structured data that is easy to index and query, greatly improving the value and usability of log data. Through this step, the system ultimately generates a structured intermediate dataset, providing a solid foundation for subsequent unified data model transformation.

[0136] In the above steps, within the context of log pattern recognition, the binary search strategy is a method for efficiently identifying log structures by progressively narrowing down the space of possible patterns. Unlike traditional exhaustive matching, binary search divides the space of possible patterns into two parts each time, quickly narrowing the search range by determining which part the target pattern belongs to. For example, the system might first determine if the log contains a JSON structure, and then, based on that determination, further determine if it contains specific field names. The key to this strategy is that each query eliminates approximately half of the possibilities, making the number of queries proportional to the logarithm of the number of possible patterns, rather than a linear relationship, significantly improving the efficiency of pattern recognition. In practical applications, the system typically performs binary search in parallel across multiple dimensions, combining the results from different dimensions to quickly locate complex log patterns.

[0137] Half-space is a term closely related to binary search strategies. In a multidimensional feature space, half-space refers to a spatial region partitioned by a hyperplane. In log pattern recognition, each binary query defines a hyperplane (e.g., whether a specific pattern is contained) that divides the log feature space into two half-spaces: those that satisfy the conditions and those that do not. Through multiple such partitions, the system can progressively narrow down the spatial region containing the target log pattern. The intersection calculation of the half-spaces is a crucial step in identifying accurate patterns; it combines constraints from multiple dimensions to form a complete description of the log pattern. This half-space-based pattern recognition method is particularly suitable for handling pattern recognition problems in high-dimensional feature spaces and can efficiently handle complex log format variations.

[0138] Example 4:

[0139] Based on Example 2, a columnar storage structure is generated based on the unified observation data stream, including:

[0140] Based on the unified observation data stream, intelligent partitioning is performed according to the timestamp field and data source identifier in the unified internal data model. The data is divided into logical partitions and blocks by column to form a set of columnar data blocks.

[0141] Based on the set of columnar data blocks, the data characteristics of each column are analyzed in real time and the encoding method is automatically selected to generate the encoded column dataset;

[0142] Based on the encoded column dataset, compression algorithms are dynamically selected and applied according to data with different characteristics and system resource conditions to generate compressed columnar data blocks.

[0143] Based on the compressed columnar data blocks, column-level metadata is created and maintained. The column-level metadata includes data type, encoding method, compression algorithm and data block location, thereby generating the columnar storage structure.

[0144] In this embodiment, an intelligent partitioning strategy is implemented based on a unified observation data stream, dividing the data into logical partitions and column-based blocks. Initial partitioning is performed based on the timestamp field and data source identifier in the unified internal data model. Time partitioning is one of the most commonly used strategies; the system typically divides data into multiple time windows according to time units such as hours, days, or weeks, with each window serving as a logical partition. Furthermore, the system considers data source identifiers, such as service names and data centers, to further subdivide the data. After determining the logical partitions, column-based block processing is performed on the data within each partition, storing data from the same column together instead of the traditional row-based storage. This storage method significantly improves query efficiency for specific columns, and data from the same column often has similar characteristics, facilitating efficient encoding and compression. The block size is dynamically adjusted based on data volume and query characteristics to balance query performance and management overhead. Through these processes, the system forms an initial set of column-based data blocks, laying the foundation for subsequent encoding and compression.

[0145] Furthermore, the system performs in-depth data characteristic analysis on each column in the previously formed columnar data block set and automatically selects the most suitable encoding method based on these characteristics. The analyzed data characteristics include data type (e.g., integers, floating-point numbers, strings), numerical distribution (e.g., mean, variance, skewness), cardinality (number of unique values), and repetition patterns. For columns with different characteristics, the system selects different encoding methods: for low-cardinality columns (columns with fewer unique values, such as status codes and log levels), dictionary encoding is typically used, replacing the original values ​​with integer IDs; for increasing or decreasing sequences (e.g., timestamps, consecutive IDs), incremental or Delta encoding is used, storing only the differences between adjacent values; for columns with a large number of repeated values, run-length encoding (RLE) is used, recording the value and its consecutive occurrences; for timestamp columns, specialized timestamp encoding may be used, leveraging the specific structural characteristics of time data. The system also considers the correlation between columns, and for highly correlated columns, a joint encoding strategy may be applied to further improve compression efficiency. Through this intelligent encoding selection, the system generates the optimal encoded representation for each column, forming the encoded column dataset.

[0146] Then, based on the encoded column dataset from the previous step, an adaptive compression algorithm is further applied to improve storage efficiency. The system dynamically selects the most suitable compression algorithm based on the data's characteristics and the current system resource status. For frequently accessed "hot" data, fast decompression algorithms such as LZ4 or Snappy are prioritized. While these algorithms may not have the highest compression ratio, they offer extremely fast decompression performance, suitable for query scenarios requiring rapid response. For less frequently accessed "cold" data, algorithms with higher compression ratios, such as Zstd or LZMA with high compression levels, may be chosen, sacrificing some decompression speed for better storage efficiency. The system also dynamically adjusts the compression level based on the current CPU load, using a higher compression level when the system load is low and reducing the compression intensity to free up computing resources when the load is high. Furthermore, specific compression strategies may be implemented for specific data types, such as using a specific floating-point compression algorithm for floating-point data. Through this adaptive compression strategy, storage costs are minimized while ensuring query performance, resulting in highly compressed columnar data blocks.

[0147] Finally, based on the compressed columnar data blocks generated in the previous step, detailed column-level metadata is created and maintained, completing the construction of the columnar storage structure. Column-level metadata is structured information describing the characteristics and storage status of each column's data, including data type (integer, floating-point, string, etc.), encoding method (e.g., dictionary encoding, incremental encoding, etc.), compression algorithm (e.g., LZ4, Zstd, etc.), numerical statistics (maximum value, minimum value, percentage of null values, etc.), and the physical location information of the data block. This metadata is crucial for query optimization; the query engine can use this information to decide whether to decompress the entire data block or skip irrelevant data blocks directly based on statistical information. This metadata is updated periodically to reflect changes in data characteristics. In addition to maintaining metadata, columns with similar characteristics are grouped for storage, I / O patterns are optimized, and column data pre-fetching and caching strategies are implemented to further improve query performance. Through these comprehensive processes, a complete columnar storage structure is ultimately generated, providing a solid foundation for efficient data querying and analysis.

[0148] Example 5:

[0149] Based on Example 4, the intelligent partitioning based on the timestamp field and data source identifier in the unified internal data model, dividing the data into logical partitions and column-based blocks to form a set of columnar data blocks, includes:

[0150] Based on the unified observation data stream, the correlation between observation data in the unified internal data model is modeled as a graph structure, realizing a graph sampling algorithm with sublinear complexity and generating graph sampling results;

[0151] Based on the graph sampling results, a sublinear time edge counting algorithm and a community detection algorithm are applied to generate graph structure feature estimates.

[0152] Based on the graph structure feature estimation, a topology-aware data partitioning strategy is generated to ensure that highly correlated data identified in the graph structure feature estimation are assigned to the same or adjacent partitions, thereby generating a topology-aware partitioning scheme.

[0153] Based on the topology-aware partitioning scheme, the optimal size of each partition is adaptively determined and a progressive partitioning adjustment mechanism is implemented to generate the columnar data block set.

[0154] Specifically, firstly, based on a unified observation data stream, the relationships between the observation data are modeled as a graph structure, and a sublinear complexity graph sampling algorithm is implemented. In this graph model, nodes represent different observation data entities (such as log entries, indicator points, and tracking segments), and edges represent the relationships between them (such as time overlap, shared source, and referencing relationships). Since the scale of observation data is usually very large, constructing a complete relationship graph is impractical; therefore, the system implements a sublinear complexity graph sampling algorithm. These algorithms obtain representative samples of the graph without traversing or storing the entire graph, using techniques such as random walks and local probing. For example, the system might use an edge-based sampling strategy, randomly selecting a subset of edges and their connected nodes; or it might use node-centric expansion sampling, starting with a small number of seed nodes and gradually exploring their neighbors. The key to these sampling algorithms is that they can obtain a good approximation of the global characteristics of the graph with very low computational cost. Through this step, the system generates graph sampling results, providing foundational data for subsequent graph structure analysis.

[0155] Next, based on the graph sampling results from the previous step, sublinear time edge counting and community detection algorithms are applied to analyze the structural features of the graph. Edge counting algorithms aim to estimate the number of edges and the degree distribution of nodes in the graph (the number of edges connected to each node), reflecting the tightness of connections between data points. Traditional edge counting requires traversing the entire graph, with complexity proportional to the graph size; while sublinear algorithms, through carefully designed random queries and statistical inference, provide accurate estimates in time much smaller than the graph size. Similarly, community detection algorithms aim to identify tightly connected subgroups (communities) in the graph, which typically represent highly correlated datasets. Sublinear community detection algorithms do not require constructing a complete adjacency matrix; instead, they quickly identify the main community structures through local probing and clustering techniques. The key to these algorithms is that their computation time is not linear with the number of nodes and edges in the graph, but rather sublinear (e.g., logarithmic or square root relationships), enabling them to efficiently process massive amounts of observational data. Through this analysis, the system generates graph structural feature estimates, including key information such as data connectivity and community distribution.

[0156] Then, based on the graph structure feature estimation from the previous step, a topology-aware data partitioning strategy is generated. The system designs a data partitioning scheme based on the identified community structure and connection patterns, ensuring that highly correlated data is assigned to the same or adjacent partitions. For example, if the system finds a strong correlation between log data, metric data, and tracking data for a service, it will try to place these data in the same partition to reduce the need for cross-partition queries. Similarly, for time-related data (such as multiple observations within the same time window), the system will also try to place them in nearby partitions. This topology-aware partitioning strategy differs from simple time or service partitioning; it considers the actual relationships between data and can better support complex correlation analysis queries. The system implements a sublinear approximation version of the graph partitioning algorithm to find a near-optimal partitioning scheme with limited computational resources, balancing data locality (the degree of clustering of related data) and partition balance (the balance of data volume in each partition). Through this step, the system generates a topology-aware partitioning scheme, providing an optimized blueprint for the physical organization of data.

[0157] Finally, based on the topology-aware partitioning scheme from the previous step, the system adaptively determines partition sizes and implements a progressive partition adjustment mechanism. Combining graph feature estimation and query pattern analysis, the system determines the optimal size for each partition. For densely connected data regions (such as data with high correlation and frequent access), the system creates smaller partitions to improve access efficiency; for sparsely connected data regions (such as data with low correlation and infrequent access), the system creates larger partitions to improve storage efficiency. This adaptive partitioning size strategy allows the system to achieve a good balance between performance and storage cost. Furthermore, the system implements a progressive partition adjustment mechanism that dynamically adjusts the partitioning strategy based on changes in data characteristics and access patterns. This adjustment does not require repartitioning all data but is achieved through local splitting, merging, and migration operations, significantly reducing adjustment costs. The system uses a sublinear algorithm to continuously monitor partition quality and triggers adjustment operations when necessary, ensuring that the partitioning strategy remains optimal. Through this series of processes, the system ultimately generates a columnar data block set, laying an optimized foundation for subsequent encoding and compression processing.

[0158] In the steps described above, within the context of observational data management, a graph structure is a mathematical model used to represent the relationships between observational data. In this model, nodes (vertices) represent different observational data entities, such as log entries, indicator data points, or tracking segments; edges (connections) represent various relationships between them, such as temporal overlap, shared origin, causal relationships, etc. The advantage of graph structures lies in their ability to naturally represent complex many-to-many relationships, without being limited by hierarchical or tabular structures. In observational data analysis, graph structures enable systems to capture and utilize the inherent connections between data, supporting more complex correlation analysis and root cause diagnosis. Graph structures can be directed (edges have direction, such as causal relationships) or undirected (edges have no direction, such as correlation), and can be weighted (edges have weights, representing the strength of the relationship) or unweighted, with the appropriate representation chosen based on specific application requirements.

[0159] Community detection refers to the process of identifying tightly connected subsets of nodes (called communities or modules) in a graph. In the context of observational data, communities typically represent highly correlated groups of data, such as different types of observations from the same service, or multiple log records describing the same event. The goal of community detection algorithms is to find a way to divide the nodes in a graph into groups such that nodes within the same group are tightly connected, while connections between different groups are relatively sparse. Common community detection methods include modularity-based optimizations (such as the Louvain algorithm), spectral clustering, random walks, and label propagation. In observational data management, community detection helps discover inherent relational structures between data, guides data partitioning and storage optimization, and supports more efficient data access and analysis. Sublinear community detection algorithms can quickly identify major community structures without processing all graph data, providing effective guidance for organizing large-scale observational data.

[0160] Example 6:

[0161] Based on Example 4, the step of performing real-time analysis of the data characteristics of each column and automatically selecting the encoding method to generate the encoded column dataset includes:

[0162] Based on the column data in the columnar data block set, a multidimensional description space of data characteristics is constructed, and a characteristic space structure is generated.

[0163] Based on the feature space structure, an active learning strategy is implemented and a query sequence is designed to perform binary search in parallel across multiple feature dimensions, generating parallel search results.

[0164] Based on the parallel search results, the search results in different sorting directions are merged into a unified data characteristic description model to generate a comprehensive characteristic model;

[0165] Based on the comprehensive characteristic model, the optimal encoding scheme is automatically selected for each column of data in the columnar data block set to generate the encoded columnar dataset.

[0166] Based on the column data in the columnar data block set, a multidimensional description space for data characteristics is constructed. This characteristic space is a conceptual structure used to represent various statistical and behavioral features of the data. For each column of data, the system calculates multiple characteristic indicators, including but not limited to: data type characteristics (such as integer, floating-point, string types and their subtypes), statistical distribution characteristics (such as mean, variance, skewness, kurtosis and other statistical measures describing the distribution of numerical values), cardinality characteristics (the number of unique values ​​and their proportion), sequence characteristics (such as whether it monotonically increases / decreases, whether it changes periodically), and text characteristics (such as average length, character distribution, common patterns). These characteristics together constitute a multidimensional space, in which each column of data corresponds to a point, the position of which is determined by its characteristic values ​​in each dimension. The system also defines multiple sorting methods, such as sorting by cardinality, sorting by coefficient of variation, and sorting by sequence correlation, providing multiple perspectives for subsequent characteristic analysis. Through this step, the system generates the characteristic space structure, laying the foundation for automatically selecting the encoding method.

[0167] Next, based on the constructed feature space structure, an active learning strategy is implemented and an optimized query sequence is designed. Active learning is a machine learning paradigm whose core idea is that the algorithm can actively select the most informative samples to learn from, rather than passively accepting all samples. In the context of data feature analysis, the system designs a query sequence that maximizes information gain. Each query divides the data feature space into two parts, helping the system quickly locate the feature positions of specific columns. For example, it might ask questions such as "Does the cardinality of this column exceed 10% of the total number of rows?", "Does this column exhibit obvious periodicity?", and "Are the values ​​of this column mainly concentrated in a specific range?" The answer to each question provides important information about the data features. Unlike traditional exhaustive analysis, this query strategy can obtain the most critical feature information with minimal analysis cost, making it particularly suitable for processing large-scale column data. The query order is dynamically adjusted, prioritizing queries that provide the maximum information gain to ensure sufficient feature information is obtained with the fewest number of queries. Through this step, the system determines the optimal query sequence, preparing for subsequent parallel searches.

[0168] This method performs a parallel binary search across multiple dimensions of the feature space to quickly locate the boundary conditions of data features. Simultaneously, queries are executed across multiple feature dimensions, such as cardinality (high or low cardinality), distribution (uniform or skewed distribution), and sequence (monotony or periodicity). Each dimension's query divides the feature space into two parts, and the parallel queries across multiple dimensions quickly narrow down the region containing the target feature. Unlike traditional methods that examine every possible encoding method one by one, this parallel search can simultaneously eliminate multiple unsuitable encoding options, significantly improving the efficiency of feature identification. For example, if the system determines that a column is low cardinality, non-uniformly distributed, and lacks obvious sequence characteristics, it can immediately eliminate unsuitable schemes such as incremental encoding and floating-point specialized encoding, focusing attention on more likely optimal schemes such as dictionary encoding. This method is particularly effective in high-dimensional feature spaces, locating the optimal encoding scheme with logarithmic query complexity, far superior to traditional linear exhaustive methods. Through this step, the system generates parallel search results, obtaining the feature location of the data across multiple dimensions.

[0169] Search results from different sorting directions are merged into a unified data characteristic description model. Search results in each dimension provide characteristic information about the data in that dimension, and the system needs to integrate this scattered information into a consistent, comprehensive characteristic model. This fusion considers the correlation and complementarity between different characteristics; for example, low cardinality and high repetition often coexist, while monotonicity and periodicity are usually mutually exclusive. The system uses a weighted fusion method, assigning weights according to the degree of influence of different characteristics on encoding selection, ensuring that the most critical characteristics are fully reflected in the final model. Furthermore, the interaction effects between characteristics are considered; for example, some encoding methods perform particularly well when processing data that is both low cardinality and sequential, and this combined characteristic needs special attention in the fusion model. Through this intelligent fusion of multi-dimensional characteristics, the system generates a comprehensive and accurate characteristic description model that can capture the core features and subtle differences of the data, providing a reliable basis for the selection of encoding methods.

[0170] Finally, based on the comprehensive characteristic model, the optimal encoding scheme is automatically selected for each column of data in the columnar data block set. The system has an encoding method library containing various encoding algorithms suitable for different data characteristics, such as dictionary encoding, incremental encoding, bitmap encoding, and run-length encoding. Based on the comprehensive characteristic model generated in the previous step, the optimal encoding method is selected for each column of data. This selection not only considers the static characteristics of the data but also the impact of expected query patterns on encoding efficiency. For example, if a column is frequently used for point queries (exact matching), hash encoding may be preferred; if it is frequently used for range queries, ordered encoding may be preferred. The system can predict the performance of different encoding schemes in terms of storage efficiency and query performance and make trade-off decisions based on the current optimization goals (such as prioritizing storage efficiency or prioritizing query performance). In addition, the impact of the hardware environment is considered, such as CPU instruction set support and memory bandwidth, to select the encoding implementation most suitable for the current environment. Through this comprehensive consideration, the optimal encoding scheme is selected for each column of data, generating the encoded column dataset, providing ideal input for subsequent compression processing.

[0171] In the steps described above, parallel binary search is a method that performs binary search simultaneously across multiple dimensions or directions, enabling faster target location than single-dimensional search. Traditional binary search searches for a specific value within an ordered sequence, halving the search space with each comparison; parallel binary search extends this idea to multi-dimensional space, performing binary search operations simultaneously across multiple dimensions. In data characteristic analysis, each dimension represents a data characteristic (such as cardinality, distribution, sequence properties, etc.), and performing binary search in parallel across multiple dimensions can quickly narrow down the possible range of characteristics. For example, one might simultaneously query "Is this column high-cardinality or low-cardinality?" and "Is this column uniformly distributed or skewedly distributed?" The answers to both questions together reduce the characteristic space to one-quarter of its original size. The advantage of this method is that the query complexity is logarithmically related to the number of dimensions, maintaining high efficiency even in high-dimensional characteristic spaces, allowing the system to accurately locate the characteristic position of the data with minimal query operations.

[0172] An encoding method library refers to a set of data encoding algorithms maintained by the system, each algorithm being suitable for specific types or characteristics of data. Common encoding methods include: dictionary encoding (replacing original values ​​with integer IDs, suitable for low-cardinality data), incremental encoding (storing the difference between adjacent values, suitable for monotonic sequences), bitmap encoding (using bits to represent the presence or absence of values, suitable for Boolean or enumeration types), run-length encoding (recording values ​​and their consecutive occurrences, suitable for data with recurring patterns), and floating-point specialized encoding (encoding specifically designed for the characteristics of floating-point numbers), etc. The encoding method library not only contains the implementations of these algorithms but also descriptions of the applicable conditions, performance characteristics, and resource requirements of each algorithm, enabling the system to select the most suitable encoding method based on data characteristics and environmental conditions. A comprehensive encoding method library is the foundation for automatically selecting the optimal encoding scheme, enabling the system to provide specially optimized processing methods for data with various characteristics, maximizing storage efficiency and query performance.

[0173] Example 7:

[0174] Based on Example 4, the process of monitoring and recording data access, identifying data access patterns, calculating data value scores, and dividing data into hot, warm, and cold data according to the data value scores and storing them in different storage media to generate a hierarchical storage dataset includes:

[0175] Based on the columnar storage structure, the column-level metadata is used to continuously monitor and record data access, and data access patterns and trends are identified through time series analysis and machine learning algorithms to generate a data access feature model.

[0176] Based on the data access feature model, a data value score is calculated for each data partition and data block in the columnar storage structure. Based on the data value score, the data is divided into hot data, warm data, and cold data, and a data hierarchical configuration scheme is generated.

[0177] According to the data tiering configuration scheme, a multi-level storage architecture is implemented, which stores hot data, warm data, and cold data in different storage media. Data blocks in the columnar storage structure are automatically moved between different storage media according to changes in data tiers to generate the tiered storage dataset.

[0178] Based on the hierarchical storage dataset and the data access feature model, an intelligent retention strategy is implemented to automatically perform downsampling, aggregation, or cleaning operations on the cold data, generating an optimized dataset that has undergone intelligent lifecycle management.

[0179] Based on the existing columnar storage structure, column-level metadata is used to continuously monitor and record data access, and time-series analysis and machine learning algorithms are used to identify data access patterns and trends. A comprehensive access monitoring framework is implemented, recording access details for each data partition and data block, including access time, frequency, duration, access type (read, scan, aggregation, etc.), and access source (query ID, user ID, etc.). These raw access logs are organized into a time-series data structure, with each data block having a corresponding access activity timeline. Multiple analysis techniques are applied to these time series: first, pattern recognition identifies periodic access patterns (such as regular access for daily and weekly reports), burst patterns (such as intensive access during troubleshooting), and long-term trends (such as a gradual decrease in access to certain data over time); second, correlation analysis discovers associated access patterns between different data blocks, such as certain data blocks often being queried together; and finally, predictive analysis predicts future access probabilities and trends in popularity based on historical access patterns. Multiple machine learning algorithms are employed to enhance these analyses, such as using time series models (ARIMA, Prophet, etc.) to predict access trends, using clustering algorithms to identify similar access patterns, and using anomaly detection algorithms to identify unusual access behaviors. Through these comprehensive analyses, a data access characteristic model is generated, accurately describing the access characteristics and future trends of each data block.

[0180] Next, based on the data access characteristic model generated in the previous step, a data value score is calculated for each data partition and data block in the columnar storage structure. Based on these scores, the data is then divided into different storage tiers. The data value score is a comprehensive indicator reflecting the importance and value of data to users and the system, determined by multiple factors: access frequency (the number of times data is queried), access recentity (the time since the last access), access pattern (periodic, bursty, or stable), data age (the time since data was generated), business importance (a predefined importance level based on data source or content), and data correlation (the degree of correlation with other high-value data). A weighted calculation method is implemented, dynamically adjusting the weights of each factor according to current business needs and system status. For example, in scenarios emphasizing real-time analysis, recentity may receive a higher weight; while in scenarios emphasizing historical trend analysis, business importance may be more important. Based on the calculated value score, the data is divided into hot data (high value score, requiring fast access), warm data (medium value score, requiring a balance between access speed and storage cost), and cold data (low value score, infrequent access and acceptable higher latency). A dynamic splitting point determination strategy is adopted, automatically adjusting the splitting thresholds based on the distribution of data value scores and system resource status to ensure that the data volume at each level remains within a reasonable range. This step generates a data tiering configuration scheme, providing guidance for subsequent physical storage optimization.

[0181] Then, based on the data tiering configuration scheme generated in the previous step, a multi-level storage architecture is implemented, storing data from different tiers on storage media with different performance characteristics. A multi-tiered storage architecture is constructed, typically including three main tiers: a hot data tier, using high-performance storage media such as memory, Optane persistent memory, or high-speed SSDs, providing extremely low access latency; a warm data tier, using conventional SSDs or high-performance HDDs, providing moderate access speeds and good cost-effectiveness; and a cold data tier, using large-capacity HDDs or object storage (such as Amazon S3 or Azure Blob Storage), providing large capacity but relatively slower access speeds. Under this tiered storage architecture, an automatic data block migration mechanism is implemented, capable of moving data between different storage tiers based on changes in data value scores. For example, when the access frequency of a warm data block suddenly increases, and its value score exceeds the hot data threshold, the system automatically promotes it from warm data storage to hot data storage; conversely, when the access frequency of hot data decreases, and its value score decreases, the system degrades it to warm data storage. This migration is typically performed when the system load is low and uses an incremental approach to minimize the impact on system performance. Furthermore, a data preheating mechanism was implemented, which can proactively move data that will be needed soon from cold storage to hot storage based on predicted access patterns, avoiding high latency during queries. Through this dynamic storage hierarchy management, a tiered storage dataset was generated, achieving efficient utilization of storage resources.

[0182] Finally, based on the hierarchical storage dataset and data access characteristic model, an intelligent data lifecycle management strategy is implemented to automatically perform downsampling, aggregation, or cleaning operations on cold data. For cold data that has not been accessed for a long time, not only is it moved to low-cost storage, but also various data compression and optimization strategies are implemented: First, time downsampling: for frequently collected time-series data, the system gradually reduces its resolution over time, for example, aggregating second-level data into minute-level, hour-level, and finally retaining daily or weekly summary data; second, dimensional compression: for multidimensional data, infrequently used dimensions or labels are removed, retaining the core analytical dimensions; third, a value-driven retention strategy: the retention period for different types of data is determined based on the business importance of the data and regulatory requirements, and automatic cleaning is performed after the expiration. An automated decision-making mechanism implementing these strategies selects the most suitable processing method for each cold data block based on data characteristics, access history, and business rules. In addition, the system also provides a data recovery mechanism, allowing the recovery of historical data that has been downsampled or compressed in special circumstances (such as audits or investigations), although this may require a long processing time. These intelligent lifecycle management strategies significantly reduce storage usage while preserving valuable information, generating optimized datasets that have undergone intelligent lifecycle management.

[0183] In the steps described above, the data access characteristic model is a structured representation describing data access behavior and patterns, used to predict future access patterns and guide storage optimization decisions. This model typically includes several components: a time pattern component, describing the periodic characteristics of data access (e.g., weekday / weekend patterns, day / night patterns, monthly / quarterly reporting cycles, etc.); a hotspot identification component, marking subsets of data with particularly high access frequency; a relational access component, representing the relationships between data items that are frequently accessed together; and a trend prediction component, predicting the trend of data popularity over time. Data access characteristic models are usually represented using probability distributions and time series models, capable of predicting future access probabilities based on historical observations. Such models are crucial for optimizing storage systems, enabling the "anticipation" of future access demands and allowing for corresponding storage layout adjustments in advance, maximizing access efficiency and minimizing storage costs.

[0184] A data value score is a quantitative metric used to measure the relative importance of data to the system and users. Unlike traditional storage systems that simply categorize data based on access frequency or data age, a data value score comprehensively considers multiple factors, including access patterns (frequency, relevance, regularity), data characteristics (age, volume, complexity), business value (key metrics, service level requirements), and correlation value (degree of association with other high-value data). Value scores are typically normalized to a range of 0-100 or 0-1 for easy comparison and stratification. The purpose of calculating this score is to support more granular storage optimization decisions, rather than simply categorizing data as "hot" or "cold," but rather identifying multiple value ranges to support more complex multi-tiered storage architectures and resource allocation strategies. High-precision data value assessment enables the system to make smarter storage decisions, allocating limited high-performance storage resources to truly important data.

[0185] Tiered storage architecture is a storage system design methodology that optimizes overall storage performance and cost by combining storage media with different performance and cost characteristics. The core idea of ​​this architecture is to allocate data to the most appropriate storage tier based on its access characteristics and value. Typical tiered storage architectures include: a hot storage tier (such as memory, Optane, and high-performance SSDs), offering the lowest latency and highest throughput, but at the highest cost, suitable for the most frequently accessed data; a warm storage tier (such as standard SSDs and high-performance HDDs), offering moderate performance and good cost-effectiveness, suitable for data accessed in a medium frequency; and a cold storage tier (such as large-capacity HDDs, object storage, and tape), offering large capacity and low cost, but with higher access latency, suitable for archived data that is accessed infrequently. The key to tiered storage architecture lies in the automatic data migration mechanism between tiers, enabling data to move between different tiers according to its value and access patterns, ensuring that system resources are always used most efficiently. This architecture can provide near-all-flash system performance within limited budget constraints while supporting large-scale data storage needs.

[0186] Example 8:

[0187] Based on Example 7, the step of identifying data access patterns and trends through time series analysis and machine learning algorithms, and generating a data access feature model, includes:

[0188] Based on the recorded data access information, a network topology describing the inter-block access relationships in the columnar storage structure is constructed, and a data access network topology is generated.

[0189] Based on the data access network topology, the data access behavior is parameterized into the Ising model, and an energy function is defined so that common access patterns in the data access network topology correspond to low-energy states, thereby generating a parameterized representation of the Ising model.

[0190] Based on the parameterized representation of the Ising model, an efficient learning algorithm based on very few samples is implemented to learn model parameters from a limited number of access logs and generate the learned model parameters.

[0191] Based on the learned model parameters, the access pattern prediction is optimized under hard constraints, and the future access probability prediction of data blocks in the columnar storage structure is output to generate the data access feature model.

[0192] Specifically, based on recorded data access patterns, a network topology describing the inter-block access relationships within a columnar storage structure is constructed. In this network, nodes represent data blocks (or partitions) in the columnar storage structure, and edges represent the inter-block access relationships between them, i.e., accesses by the same query or consecutive accesses within a short time window. These inter-block access patterns are identified by analyzing query logs and access records. For each pair of data blocks, the system calculates their co-access frequency (the number of times they are accessed by the same query) and temporal relationship (whether access to one data block frequently follows that of another), and determines the edge weights based on these metrics. Furthermore, contextual information about accesses, such as query type, user role, and time factors, is considered to further refine the calculation of association strength. Constructing such a network topology presents two main challenges: first, the network size can be very large, containing millions of nodes and billions of edges; second, the network structure dynamically changes over time, reflecting the evolution of access patterns. This large-scale dynamic network is efficiently maintained by implementing incremental construction and update mechanisms. Additionally, edge sampling and node aggregation techniques are used to reduce network complexity while maintaining key topological structures. This step generates the data access network topology, providing a structured representation for subsequent access pattern analysis.

[0193] Next, based on the data access network topology constructed in the previous step, the data access behavior is parameterized into the Ising model, a mathematical model derived from statistical physics that can effectively represent systems with complex interactions. In this model, the access state of each data block is represented as a binary variable (accessed or not accessed), and the associated access between data blocks is represented as interaction energy. A global energy function is defined such that common access patterns correspond to low-energy states, and rare access patterns correspond to high-energy states. Specifically, if two data blocks are frequently accessed together, their simultaneous access state should have a lower energy, while the state where one is accessed and the other is not should have a higher energy. Various interaction types are considered, including pairwise interactions (associations between two data blocks), three-body interactions (associations between three data blocks), and external field influences (such as the influence of external conditions like time factors and query type on access probabilities). This parameterization method based on a physics model has several advantages: first, it can naturally represent complex multi-entity interactions; second, it has a sound theoretical foundation and a large number of readily available algorithms; and finally, it can effectively handle noise and incomplete data and has good generalization ability. This parameterization process transforms complex data access networks into a structured Ising model, providing a mathematical foundation for subsequent learning and prediction.

[0194] Then, based on the parameterized representation of the Ising model obtained in the previous step, an efficient learning algorithm based on very few samples is implemented to learn model parameters from a limited number of access logs. Traditional machine learning methods typically require a large amount of training data to accurately estimate model parameters, but in practical applications, access records for some data blocks may be sparse, especially for newly added data or rare access patterns. This problem is addressed by implementing several innovative learning techniques: first, transfer learning, which transfers patterns learned from data blocks with abundant access records to data blocks with sparse access records; second, Bayesian inference, which uses prior knowledge (such as access patterns of similar data blocks) to guide parameter estimation; third, active learning, which proactively selects the most valuable samples for analysis to maximize information gain; and finally, gradient enhancement and regularization techniques to improve the robustness and generalization ability of the learning algorithm. The combination of these techniques enables the system to learn meaningful access patterns from very few samples and achieves high data efficiency in parameter estimation. For example, periodic patterns or associations can be identified from data blocks with only a few dozen access records, which typically requires hundreds or thousands of records in traditional methods. Through this efficient learning process, the learned model parameters are obtained, which accurately describe the access relationships and patterns between data blocks.

[0195] Finally, based on the learned model parameters, the access pattern prediction is optimized under hard constraints, outputting a prediction of the future access probability of data blocks in the columnar storage structure. The learned Ising model parameters are used to infer the probability of data blocks being accessed under different conditions. This prediction considers various contextual factors, such as time (time period, date, season, etc.), observed access patterns (e.g., the conditional probability of other data blocks being accessed given that some data blocks have been accessed), and external events (e.g., system upgrades, business activities, etc.). The system pays particular attention to satisfying hard constraints, which may stem from business rules (e.g., certain data must remain available), system limitations (e.g., storage capacity, access bandwidth), or performance requirements (e.g., response time targets). The system implements an efficient inference algorithm based on the Monte Carlo method to find the most likely access pattern under constraints. Furthermore, multi-step prediction capabilities are implemented, which can predict the trend of access probability changes over a future period (e.g., hours, days, weeks), providing guidance for long-term storage optimization. The output prediction results include the access probability of each data block at different time points and under different conditions, as well as the access probability of association between data blocks, which together constitute a complete data access feature model, providing strong support for subsequent storage optimization decisions.

[0196] In the steps described above, the data access network topology is a structure that uses graph theory to represent the access relationships between data blocks. In this network, nodes represent data blocks or partitions in the storage system, and edges represent access relationships between them, i.e., accesses that are accessed by the same query or are closely related in time. The weight of the edge typically represents the strength of the association and can be calculated based on factors such as common access frequency, temporal proximity, or access pattern similarity. This network representation has several key advantages: First, it can visually display the access relationships between data and identify highly correlated data clusters; second, it supports the application of mature graph analysis algorithms, such as community detection, centrality analysis, and path analysis; and third, it provides a natural framework for modeling and predicting complex access patterns, including indirect associations and higher-order dependencies. In storage optimization, the data access network topology can guide data placement decisions, for example, placing highly correlated data blocks on the same storage node or in close proximity to reduce query data access latency and network traffic.

[0197] The Ising model is used to describe large-scale systems with local interactions. In its classical form, the Ising model describes a system composed of many interacting binary units (spins), each spin can be in two states (e.g., up / down or +1 / -1), and there are interactions between adjacent spins. The total energy of the system is determined by the states of all spins and the strength of their interactions. In the context of data access pattern modeling, each data block corresponds to a spin, its access state (accessed or unaccessed) corresponds to the spin state, and the access associations between data blocks correspond to the interactions between spins. The system tends to enter lower-energy states, meaning that combinations of data blocks conforming to common access patterns are more likely to be accessed together. The advantage of the Ising model is that it can express complex multi-entity interactions, and there are a large number of mature statistical physics methods available for analyzing and simulating the behavior of such systems. In practice, the Ising model or its variants (such as Boltzmann machines, Markov random fields, etc.) can be used to effectively model the association patterns of data access and support predictions and optimizations based on these patterns.

[0198] Example 9:

[0199] Based on Example 7, the step of calculating a data value score for each data partition and data block in the columnar storage structure, dividing the data into hot data, warm data, and cold data according to the data value score, and generating a data tiered configuration scheme includes:

[0200] Based on the columnar storage structure, the interrelationships between the observation data recorded in the column-level metadata are modeled as a network structure, and a triangle counting algorithm with sublinear complexity is implemented to generate triangle structure detection results.

[0201] Based on the triangular structure detection results, the local density and centrality of the data in the columnar storage structure are calculated, and the data with high density and high centrality in the triangular structure detection results are initially ranked in terms of value to generate a topological value assessment.

[0202] Based on the topological value assessment, a sublinear time feature extraction algorithm is designed to extract key features from the observation data in the columnar storage structure. The key features and the topological value assessment are combined to construct a comprehensive data value model and generate a data value score.

[0203] Based on the distribution of the data value scores, the optimal threshold for data stratification is dynamically determined through sublinear statistical estimation. The capacity allocation of each layer is optimized by comprehensively considering storage costs, access patterns in the data access feature model, and the distribution of the data value scores, thereby generating the data stratification configuration scheme.

[0204] In this embodiment, based on a columnar storage structure, the relationships between observed data recorded in column-level metadata are modeled as a network structure to implement a triangular counting algorithm with sublinear complexity. In this network, nodes represent data entities (such as columns, data blocks, or records) in the columnar storage structure, and edges represent relationships between them (such as co-occurrence, reference, derivation, etc.). Constructing a complete network can be computationally expensive, especially for large-scale observed data containing billions of records. Therefore, a sublinear complexity network analysis method, particularly an efficient triangular counting algorithm, is implemented. A triangle is a structure formed by three interconnected nodes in a network and is a fundamental indicator of network clustering and tight connections. Traditional triangular counting methods require checking every possible combination of three nodes, with a complexity of O(n³), which is infeasible for large-scale networks. The sublinear algorithm implemented in this system is based on sampling and statistical inference, and can accurately estimate the number of triangles by checking only a small portion of the network. Specifically, an edge sampling method is used, randomly selecting a portion of edges and checking whether they form triangles, and then obtaining a global estimate through statistical amplification. In addition, graph partitioning and parallel computing techniques are utilized to further improve computational efficiency. This efficient network analysis enables the detection of triangular structures without constructing a complete network, providing a foundation for subsequent topology characteristic analysis.

[0205] Next, based on the triangle structure detection results from the previous step, the local density and centrality of the data in the columnar storage structure are calculated, and a preliminary value ranking is performed on the high-density and high-centrality data. Local density refers to the tightness of the network structure around a node, which can be measured by the number of triangles the node participates in or other local clustering indicators; centrality refers to the importance of a node in the network, which can be measured by degree centrality (number of connections), betweenness centrality (number of shortest paths through the node), or eigenvector centrality (a recursive definition considering the importance of neighbors). These network topology characteristics are important indicators of data value: high-density areas usually represent highly correlated datasets and may contain important patterns or relationships; high-centrality nodes are usually key hubs in the network, connecting different data groups or providing extensive context. First, these topology indicators are calculated for each data entity, and then the data is ranked according to the indicator values ​​to generate a preliminary value sequence. This ranking based on network topology differs from traditional ranking based on access frequency; it can capture the structural relationships between data and identify "bridge data" that may not be frequently accessed directly but plays a key role in connecting other important data. Efficient approximation algorithms, such as random walk approximate centrality, were employed in the computation to keep the complexity at a sublinear level. This analysis generated a topological value assessment, providing network structure-level input for a comprehensive data value model.

[0206] In the next step, based on the topology value assessment from the previous step, a sublinear time feature extraction algorithm is designed to extract key features from the observation data in the columnar storage structure. These features are then combined with the topology value assessment to construct a comprehensive data value model. The system considers various data features, including content features (such as information entropy, anomaly, and rarity), time features (such as freshness and historical length), usage features (such as access frequency and query complexity), and business features (such as service level requirements and data source importance). Extracting these features requires processing a large amount of data; traditional methods might require a complete scan of all data, resulting in high computational costs. A sublinear feature extraction algorithm is implemented, using techniques such as sampling, incremental computation, and distributed processing to accurately estimate these features without processing all the data. For example, for calculating information entropy, the system uses a sampling-based approximation algorithm, requiring only the analysis of a small portion of the data to obtain an effective estimate; for anomaly detection, algorithms such as isolated forests are used to detect outliers in sublinear time. These features are then combined with the previous topology assessment, and a comprehensive data value model is constructed using machine learning models (such as gradient boosting trees, random forests, or neural networks). This trained model can predict the actual value of data based on multidimensional features and output a normalized data value score, typically ranging from 0 to 100. This score comprehensively reflects the importance and value of the data in the system, providing a reliable basis for storage hierarchy partitioning.

[0207] Finally, based on the distribution of data value scores generated in the previous step, the optimal threshold for data stratification is dynamically determined through sublinear statistical estimation. The system comprehensively considers storage costs, access patterns, and value score distribution to optimize capacity allocation for each stratum, generating the final data stratification configuration scheme. Instead of using fixed thresholds to divide data into hot, warm, and cold strata, the system dynamically determines the optimal partitioning point based on the current distribution of data value scores. The system implements a sublinear complexity statistical estimation method, using sampling and quantile estimation techniques to accurately find suitable boundary points without sorting the value scores of all data. When determining the threshold, the system considers several factors: first, storage resource constraints, requiring the proportion of hot data to be determined based on the available high-performance storage (such as SSD) capacity; second, value distribution characteristics, such as the central tendency and dispersion of scores, with the system setting boundary points at locations of significant value differences; third, performance requirements, ensuring a sufficient proportion of high-value data is stored in fast storage to meet performance requirements; and finally, cost-benefit balance, analyzing the cost-benefit ratio of different partitioning schemes and selecting the optimal configuration. In addition to determining thresholds, the capacity allocation and configuration parameters of each storage tier are optimized, such as caching strategies, prefetch settings, and compression levels, to further improve overall performance. This dynamic, data-driven tiered configuration approach can adapt to changes in data characteristics and access patterns, always maintaining optimal storage resource utilization efficiency, significantly reducing total cost of ownership (TCO) while ensuring fast access to critical data.

[0208] In the steps described above, local density and centrality are two core concepts in network analysis, used to assess the importance and position of nodes within a network. Local density (or clustering coefficient) measures the tightness of the network surrounding a node, typically defined as the ratio of actual connections between a node's neighbors to the theoretically maximum possible number of connections. High local density means the node is in a tightly connected community, which usually represents highly correlated datasets. Centrality measures the importance of a node within the overall network and can be defined in several ways: degree centrality simply calculates the number of connections a node has; betweenness centrality calculates the number of shortest paths through a node, reflecting its importance as a "bridge" in the network; eigenvector centrality considers the degree to which a node connects to other important nodes, assigning higher scores to nodes connected to important nodes. In data value assessment, these metrics provide a structured perspective: high-density nodes are usually part of strongly correlated datasets and should be prioritized for retention; high-centrality nodes are key hubs in the network, playing a crucial role in connecting different datasets, and removing these nodes could lead to network fragmentation, therefore they should also be prioritized for retention. By comprehensively considering these network structure metrics, the system can identify important data that may be overlooked under traditional access frequency metrics but play a key role in the data association structure.

[0209] Sublinear statistical estimation refers to a class of statistical computation methods with lower complexity than linear (O(n)), such as logarithmic or square root complexity. When dealing with large-scale datasets, traditional statistical methods require processing all data points, resulting in high costs; while sublinear methods, through carefully designed sampling and estimation strategies, can obtain high-quality statistical results by processing only a subset of the data. These methods are typically based on random sampling, probabilistic data structures, and approximation algorithms, achieving a good balance between accuracy and computational cost; sublinear quantile estimation algorithms can provide approximate quantiles in O(log n) or even constant time by maintaining a much smaller data summary (such as a quantile sketch). In data stratification configurations, sublinear statistical estimation enables systems to efficiently handle the distribution of numerical scores and quickly find appropriate stratification thresholds without needing to sort or fully scan the value scores of all data. This high computational efficiency is particularly important for systems that require frequent adjustments to stratification configurations, allowing them to respond quickly to changes in data value and access patterns.

[0210] Example 10:

[0211] Building upon Example 7, the analysis of user query behavior, identification of query pattern characteristics, formulation of indexing strategies for different data layers, implementation and maintenance of multiple types of index structures, optimization of data layout, and generation of optimized data storage structures include:

[0212] Based on the hierarchical storage dataset and historical query logs, in-depth analysis of user query behavior is performed to identify query pattern features. Clustering algorithms are used to group similar queries and generate a query pattern feature library.

[0213] Based on the query pattern feature library and the hierarchical storage dataset, differentiated indexing strategies are formulated for different data layers, and dynamic indexing strategy schemes are generated.

[0214] Based on the dynamic indexing strategy, multiple types of index structures are implemented and maintained, the performance of each index structure is automatically monitored and the index parameters are dynamically adjusted to generate optimized and diversified index structures.

[0215] Based on the query pattern feature library and the optimized diversified index structure, a co-occurrence matrix of columns and fields in the query is constructed, and the co-occurrence frequency and relevance score between columns and fields are calculated. When the relevance score exceeds a preset association threshold, it is identified as a highly relevant data pair. The columns and fields in the highly relevant data pair are arranged in consecutive storage locations, the layout of time-series data in the hierarchical storage dataset is optimized, and the optimized data storage structure is generated.

[0216] In Example 10, it should be noted that, based on the hierarchical storage dataset and historical query logs, in-depth analysis of user query behavior is performed to identify query pattern characteristics. A clustering algorithm is used to group similar queries, generating a query pattern feature library. User query logs are collected and analyzed. These logs contain the complete text of the query, execution time, amount of returned data, fields and tables accessed, filtering conditions used, and contextual information of the query (such as the user, application, and time of the query). Natural language processing techniques are used to parse the query text, extracting its semantic structure and operational intent. Each query is represented as a multi-dimensional feature vector, including access pattern dimensions (such as point queries, range queries, or aggregation queries), data selection dimensions (the set of fields accessed), filtering condition dimensions (the type and complexity of the conditions used), and time dimensions (the execution frequency and time distribution of the query). A hierarchical clustering algorithm is applied to group queries with similar characteristics, forming query clusters. Hierarchical clustering is chosen instead of algorithms such as K-means because it does not require pre-specifying the number of clusters and can adaptively discover natural query groups. For each query cluster, representative features are extracted, such as commonly used fields, typical filtering conditions, average complexity, and execution frequency, forming a feature description for that type of query. The temporal distribution patterns of queries are also analyzed, such as daily report queries, work-hour monitoring queries, or periodic batch processing queries. These temporal patterns are invaluable for predicting future query load and optimizing caching strategies. All these query features and patterns are organized into a structured knowledge base, namely the query pattern feature library, providing data-driven guidance for subsequent index design and data layout optimization.

[0217] Next, based on the query pattern feature library and hierarchical storage dataset, differentiated indexing strategies are developed for different data layers, generating dynamic indexing strategy schemes. First, the resource requirements and performance expectations for each query pattern are assessed, and matching indexing strategies are formulated based on the characteristics of different data layers (e.g., fast access for hot data layers and low cost for cold data layers). For hot data layers, a comprehensive indexing strategy is adopted, creating multiple types of indexes for frequently queried fields, including B+ tree indexes (supporting range queries and sorting operations), inverted indexes (supporting text search and tag queries), and bitmap indexes (supporting fast filtering of low-cardinality numeric segments). For warm data layers, a balanced strategy is adopted, creating key indexes only for the most frequently used query patterns, using composite indexes to cover multiple related fields, reducing the number of indexes while maintaining good query performance. For cold data layers, a minimalist strategy is adopted, retaining only necessary primary key indexes and time-partitioned indexes, relying more on data partitioning and compression optimization. The relationships between queries are also considered, creating composite indexes for frequently queried field groups to reduce index merging operations during query execution. Special attention is paid to the creation and maintenance costs of indexes. A cost model is used to evaluate the storage overhead, update costs, and expected performance improvements of each index, and an index is only created when the performance benefits outweigh the costs. To cope with changing query patterns, a dynamic index adjustment mechanism is designed, which can automatically adjust index configurations based on observed changes in query frequency and performance metrics, such as adding new indexes, deleting idle indexes, or modifying the coverage of existing indexes. Through this differentiated and dynamic indexing strategy, the system minimizes the storage and maintenance overhead of indexes while ensuring query performance.

[0218] Then, based on the dynamic indexing strategy, various types of index structures are implemented and maintained. The performance of each index structure is automatically monitored, and index parameters are dynamically adjusted to generate optimized and diversified index structures. The system implements an index management framework capable of creating, updating, and deleting different types of indexes, including traditional relational indexes (such as B+ trees), specialized time-series data indexes (such as time-sharded indexes), and full-text search indexes (such as inverted indexes). For B+ tree indexes, various optimizations are implemented, such as prefix compression (reducing index size), incremental updates (efficiently handling new data), and page split control (reducing maintenance overhead). For inverted indexes, techniques such as term compression, skip list optimization, and partial index updates are implemented to improve retrieval efficiency and reduce update costs. For bitmap indexes, compressed bitmap techniques and a lazy merging strategy are used to balance storage efficiency and query performance. A comprehensive index monitoring mechanism is deployed to collect usage statistics and performance metrics for each index, such as usage frequency, query hit rate, range selectivity, and update overhead. These metrics are used to evaluate the actual utility of the indexes, identify poorly performing or underutilized indexes, and automatically adjust their parameters or recommend deletion. For example, if the page utilization of a B+ tree index is observed to be too low, its fill factor may be adjusted; if the selectivity of a bitmap index is found to be too low, resulting in high subsequent filtering costs, it may be recommended to use other types of indexes instead. Index segmentation and merging strategies are also implemented, which are particularly suitable for handling time-series data with large write volumes: new data is first written to small index segments, and then gradually merged into larger segments in the background, balancing write performance and query efficiency. Through this dynamic index management, the system can maintain the optimal index structure combination and adapt to changing data characteristics and query patterns.

[0219] Finally, based on the query pattern feature library and the optimized diversified index structure, a co-occurrence matrix of columns and fields in the query is constructed. The co-occurrence frequency and relevance score between columns and fields are calculated. When the relevance score exceeds a preset association threshold, it is identified as a highly relevant data pair. The columns and fields in the highly relevant data pair are arranged in contiguous storage locations, optimizing the layout of time-series data in the hierarchical storage dataset and generating an optimized data storage structure. Historical query logs are analyzed to construct an NxN co-occurrence matrix (N is the number of columns in the system). Each cell (i,j) in the matrix represents the frequency with which column i and column j appear together in the query. Multiple relevance measures are used to calculate the association strength between columns, such as the Jaccard coefficient (based on the proportion of queries with co-occurrence), point mutual information (measuring the statistical relevance between two columns), and adjusted cosine similarity (weighted similarity considering query frequency). Combining these indicators, a normalized relevance score is calculated for each pair of columns, ranging from 0 to 1. When the relevance score exceeds the system-set threshold (usually between 0.7 and 0.8, which can be adjusted according to system performance), the pair of columns is marked as a "highly relevant data pair". Using a greedy algorithm and graph partitioning techniques, highly correlated columns are organized into column families, ensuring that highly correlated columns are grouped together as much as possible while balancing the size of each column family. At the physical storage level, a column family storage strategy is implemented, storing data from the same column family in contiguous physical locations to reduce random I / O and addressing overhead when accessing correlated columns. Particular attention is paid to optimizing time-series data, implementing time partitioning and a multi-level storage structure: the latest hot data uses a columnar layout to optimize analytical query performance; historical data uses a hybrid layout, dynamically adjusting the physical organization of columns based on access patterns. A progressive data reorganization mechanism is also implemented, automatically reorganizing the data layout to adapt to the latest observed query patterns when the system load is low. Through this query pattern-based data layout optimization, the I / O volume and data access latency of common queries are significantly reduced, while maintaining the compression efficiency and analytical performance advantages of columnar storage.

[0220] In the steps described above, the query pattern feature library is a knowledge repository that extracts and structures the characteristics of user query behavior by analyzing a large amount of historical query logs. The query pattern feature library not only records the surface features of queries (such as which tables and fields are accessed, what filtering conditions are used, etc.), but also captures the semantic features (the purpose and intent of the query), temporal features (execution frequency and time distribution), and business features (associations with specific user roles or business processes). The construction of this feature library typically involves several steps: first, query log parsing, converting the raw query text into a structured representation; second, feature extraction, extracting key features such as access patterns, field sets, and condition types from the structured query; third, pattern mining, identifying recurring query patterns and field combinations; and finally, knowledge representation, storing the identified patterns in a structured form for easy system use.

[0221] Example 11:

[0222] Based on Example 10, the step of performing in-depth analysis of user query behavior, identifying query pattern features, and using a clustering algorithm to group similar queries to generate a query pattern feature library includes:

[0223] Based on historical query logs, a relational network describing the relationships between query operations is constructed, the temporal pattern of queries is identified, and the sequential dependencies between query operations are captured to generate a relational network of query operations.

[0224] Based on the query operation association network, the query pattern is represented as an Ising model. An energy function reflecting the probability of the query pattern is defined so that common query patterns in the query operation association network correspond to low-energy state configurations, thereby generating the Ising model representation of the query pattern.

[0225] Based on the Ising model representation of the query pattern, an efficient learning algorithm based on minimum sample size is implemented to extract key pattern features from a limited number of query logs and generate a learned query pattern model.

[0226] Based on the learned query pattern model, a representative set of query patterns is generated under the premise of satisfying the constraints, realizing deep inference of query intent, predicting future query needs, optimizing the corresponding index and data layout strategies in the hierarchical storage dataset, and outputting the query pattern feature library containing intent annotations.

[0227] Specifically, firstly, based on historical query logs, an association network describing the relationships between query operations is constructed to identify temporal patterns and capture the sequential dependencies between query operations, generating a query operation association network. The collected query logs are analyzed in depth, focusing not only on individual queries but also on the relationships and sequence patterns between them. Detailed query information is extracted from the logs, including query text, execution time, user ID, session ID, and contextual information (such as application name, client IP, etc.). Natural language processing and query parsing techniques are used to convert the query text into a structured representation, identifying the types of query operations (such as selection, filtering, aggregation, sorting, etc.) and the data entities involved (such as tables, fields). Temporal relationships between queries are analyzed to identify consecutively executed query sequences within the same session or workflow, discovering temporal dependency patterns between queries, such as "query A is usually followed by query B." These relationships are modeled as a directed graph, i.e., a query operation association network, where nodes represent query operations or query types, and edges represent relationships between them, such as temporal following, data dependency, or semantic relevance. The weights of the edges represent the strength of the relationship, typically based on the frequency and statistical significance of co-occurrence. The system applies network analysis algorithms, such as centrality analysis and community detection, to identify key query operations (central nodes in the graph) and query operation clusters (tightly connected subsets of nodes). It also extracts periodicity and triggering patterns from time-series data, such as specific queries frequently executing at a certain time of day, or queries being frequently triggered after specific system events. Through this complex association analysis, a rich query operation association network is generated, capturing the dynamic structure and patterns of user query behavior, providing a foundation for subsequent query pattern modeling.

[0228] Next, based on the query operation association network, the query pattern is represented as an Ising model. An energy function reflecting the probability of the query pattern is defined so that common query patterns in the query operation association network correspond to low-energy state configurations, generating the Ising model representation of the query pattern. The Ising model originates from statistical physics and is used to describe the interactions in particle systems. It consists of a series of variables (called "spins") that can take values ​​of +1 or -1, and these variables are connected by interaction energy terms. In query pattern modeling, each query operation is mapped to a variable in the Ising model, and the state of the variable (+1 or -1) indicates whether the operation is executed. A global energy function is defined such that common query patterns (query combinations frequently observed in the query operation association network) correspond to lower energy states, while rare query combinations correspond to higher energy states. This energy function contains various interaction terms: first-order terms reflect the base probability of a single query operation (e.g., popular queries have strong first-order terms); second-order terms represent the strength of the association between pairs of query operations (e.g., queries that are frequently executed together have strong positive second-order terms); and higher-order terms capture complex dependencies involving multiple query operations (e.g., a query is usually executed only after two other specific queries have been executed). Using methods such as maximum likelihood estimation or gradient descent, the strength parameters of these interaction terms are learned from historical query data, enabling the model to accurately reflect the observed distribution of query patterns. A significant advantage of the Ising model is its ability to naturally express complex conditional probability relationships, such as "the probability of query C being executed given that queries A and B have been executed," which is crucial for understanding and predicting complex query sequences. This physically-based model representation encodes complex patterns of query behavior into a mathematical structure, providing a theoretical foundation for subsequent pattern learning and prediction.

[0229] Then, based on the Ising model representation of query patterns, an efficient learning algorithm based on minimum sample size is implemented to extract key pattern features from limited query logs and generate a learned query pattern model. Traditional machine learning methods typically require a large amount of training data to accurately estimate model parameters, but in query pattern analysis, some query patterns may be relatively rare, failing to provide sufficient samples. This challenge is addressed by implementing several innovative learning techniques: first, transfer learning, applying knowledge learned from common query patterns to similar but less common patterns; second, Bayesian learning, integrating prior knowledge (such as query relationship assumptions provided by domain experts) into parameter estimation, reducing reliance on data; third, contrastive learning, learning feature representations by comparing similar and dissimilar queries without requiring full probability distribution estimation; and finally, few-shot learning algorithms, such as prototype networks or matching networks, capable of generalizing patterns from a small number of examples. Parameter sharing and regularization techniques are particularly emphasized, significantly reducing the number of independent parameters that need to be learned by assuming that similar queries share certain parameters. An active learning strategy is also implemented, prioritizing the labeling and analysis of the most informative query samples to maximize the learning effect of limited data. To handle the uncertainty of sparse samples, confidence intervals for the model parameters are maintained. This involves not only estimating the most probable parameter values ​​but also tracking the uncertainty of the estimates for subsequent robust decision-making. Through this combination of efficient learning techniques, key pattern features are extracted from limited query logs, constructing a query pattern model that can capture common patterns and generalize to rare scenarios.

[0230] Finally, based on the learned query pattern model, a representative set of query patterns is generated under the premise of satisfying constraints. This enables deep inference of query intent, predicts future query demands, and optimizes the corresponding indexing and data layout strategies in the hierarchical storage dataset, outputting a query pattern feature library containing intent annotations. Using the learned Ising model parameters, a representative set of query patterns is generated through methods such as Monte Carlo sampling or variational inference. These generated patterns are not simple copies of historical queries, but new patterns generated based on the learned probability distribution, incorporating both the statistical characteristics of historical data and covering potential unseen patterns. The system considers various constraints during the generation process, such as system resource limitations (queries cannot exceed a certain complexity), data access permissions (patterns must respect user access restrictions), and business rules (certain query operations must be executed in a specific order). For each generated query pattern, the system performs intent inference, attempting to understand the user's purpose and business scenario behind the query. This intent inference is based on various information, such as the query's structural characteristics (e.g., aggregation and filtering types), temporal characteristics (execution time and frequency), and contextual information (source application, user role, etc.). The inferred intents are categorized into various types, such as monitoring and inspection (regularly checking system status), problem diagnosis (investigating specific anomalies), trend analysis (understanding long-term changes), or report generation (summarizing business data). These intent annotations greatly enhance the semantic understanding of query patterns, enabling more accurate prediction of future query needs.

[0231] In the steps described above, the query operation association network is a structured representation of the relationships between query operations using a graph theory model, capturing the dynamic characteristics and patterns of user query behavior. In this network, nodes typically represent query operations (such as specific types of queries or query components), and edges represent the relationships between them, such as temporal relationships (operation A usually executes before operation B), data dependencies (operation B uses the result of operation A), or semantic associations (two operations access related data entities). These relationships are usually weighted, representing the strength or statistical significance of the relationship. The construction of the query operation association network is based on in-depth analysis of historical query logs, considering not only the characteristics of the queries themselves but also paying particular attention to the interactions and sequence patterns between queries. This network structure has several key applications: it can identify key query operations (central nodes or highly connected nodes in the network); discover query clusters (tightly connected subsets of nodes representing related groups of queries); predict query sequences (predicting possible next queries based on observed queries); and understand the business context of queries (by analyzing the position of queries within a larger workflow). In system optimization, the query operation association network provides valuable guidance for resource allocation, caching strategies, and query scheduling, enabling optimization based on the overall query behavior pattern, rather than just individual queries.

[0232] Query intent inference refers to the process of understanding a user's true purpose and needs from the surface form and context of a query. Unlike traditional query processing, which focuses solely on "what the query says," query intent inference attempts to understand "what the user wants." This deeper understanding is crucial for system optimization and improved user experience. Query intent inference typically analyzes multiple information sources: the query's structural features (such as aggregation type, filtering conditions, selected fields, etc.); temporal context (query execution time, frequency, and periodicity); user context (user role, department, past query history); and system context (current system state, recent events, etc.). Based on this information, it attempts to categorize queries into different intent types, such as real-time monitoring (checking the current status), fault diagnosis (investigating anomalies), performance optimization (analyzing efficiency bottlenecks), trend analysis (understanding long-term changes), or report generation (summarizing business data). Query intent inference can be implemented using various techniques, from rule-based methods to complex machine learning models such as classifiers, sequence models, or deep learning networks.

[0233] Example 12:

[0234] Building upon Example 2, the process of performing cleaning and structure enhancement operations on the observation data based on the unified observation data stream, converting logs into metrics, adding contextual information to the observation data, and generating observation data with rich context includes:

[0235] Based on the unified observation data stream, a configurable data processing pipeline framework is designed, and a configurable data pipeline definition is generated.

[0236] Based on the configurable data pipeline definition, cleaning and structuring enhancement operations are performed on the observation data in the unified observation data stream to generate a cleaned structured data stream.

[0237] Based on the cleaned structured data stream, the real-time conversion from logs to metrics is realized. Rules are defined to extract numerical metrics from the logs in the cleaned structured data stream and generate a log-derived metric dataset.

[0238] Based on the cleaned structured data stream and the indicator dataset derived from the logs, data association and context enrichment are achieved, service topology information and metadata are added to the observation data, and the observation data with rich context is generated.

[0239] Specifically, firstly, based on a unified observation data stream, a configurable data processing pipeline framework is designed, generating configurable data pipeline definitions. A highly flexible and scalable data processing pipeline framework is designed, allowing users to define complex data processing flows through configuration rather than programming. This framework adopts a component-based design principle, breaking down data processing into multiple independent processing steps or stages. Each stage performs specific data transformation or enhancement operations, such as parsing, filtering, transformation, and enrichment. The system provides multiple configurable implementations for each processing step, such as multiple log parsers (regular expressions, JSON, XML, etc.), multiple filters (based on conditions, statistical characteristics, etc.), and multiple converters (field mapping, type conversion, formatting, etc.). Users can define the combination and order of these components through declarative configuration languages ​​such as YAML or JSON to form a complete data processing pipeline. A configuration validation mechanism is implemented to ensure the syntactic and semantic correctness of the defined pipelines, such as checking whether the input and output of components match, whether data types are compatible, and whether dependencies are satisfied. Templates and a predefined pipeline library are also provided, allowing users to quickly build pipelines for common data processing scenarios, such as web server log processing, application log analysis, and system metric normalization. The pipeline framework supports dynamic reconfiguration, allowing pipeline definitions to be updated without halting data stream processing, enabling smooth configuration changes. This dynamism is crucial for adapting to constantly changing data formats and requirements. It also implements pipeline performance monitoring and self-tuning mechanisms, collecting metrics such as processing latency, throughput, and error rate for each component, automatically identifying bottlenecks and suggesting optimization measures. Through this configurable pipeline framework, highly customized processing of a unified observation data stream is achieved, meeting the data processing needs of different environments and scenarios while maintaining the clarity and maintainability of the processing logic.

[0240] Next, based on the configurable data pipeline definition, cleaning and structuring enhancement operations are performed on the observation data in the unified observation data stream to generate a cleaned structured data stream. First, the raw observation data is parsed and standardized, converting data from different sources and formats (such as log text, JSON events, binary metrics, etc.) into a unified internal data structure. For text logs, multiple parsing strategies are implemented: regular expression pattern matching for handling log formats with fixed structures; natural language processing techniques for extracting key information from unstructured text; and machine learning models for identifying new or changing log formats. After parsing, data cleaning operations are performed to remove outliers, erroneous records, and redundant information. The system implements multiple data validation rules to detect and correct common data problems, such as timestamp errors, missing fields, and abnormal numerical ranges. For missing or incomplete data, an intelligent completion strategy is applied, using contextual information or historical patterns to infer possible values. Special attention is paid to data consistency, ensuring that the same entities (such as service names and host identifiers) are represented consistently across different data sources, which is crucial for subsequent correlation analysis. After basic cleaning, structuring enhancement is performed to extract more structured fields and relationships from the data. Entity recognition technology was implemented to extract key entities, such as user IDs, transaction IDs, and error codes, from log text; pattern detection algorithms were used to identify recurring event sequences or behavioral patterns; and context association was employed to establish causal or temporal relationships between events. Data standardization transformations were also performed, such as unit unification, time zone adjustment, and naming conventions, to ensure data consistency in subsequent processing. All these operations were executed within a unified pipeline framework, following the processing flow defined in the previous step. The output of each stage served as the input for the next, forming a coherent data transformation chain. Through this series of processes, the raw observation data was transformed into a high-quality structured data stream, laying the foundation for subsequent analysis and enhancement.

[0241] Then, based on the cleaned structured data stream, real-time transformation from logs to metrics is achieved. Rules are defined to extract numerical metrics from the logs in the cleaned structured data stream, generating a log-derived metric dataset. A flexible transformation engine is designed, using declarative rules to convert unstructured or semi-structured log data into structured time-series metrics. This transformation enables log data, which was originally only used for text search, to be used for numerical analysis and monitoring, greatly expanding the application scope of log data. Multiple metric extraction modes are supported: count metrics, which count the number of log events that meet specific conditions, such as error log counts or authentication failures; numerical extraction metrics, which extract explicit numerical values ​​from logs, such as response time, resource usage, or transaction volume; and distribution metrics, which capture the distribution characteristics of numerical values, such as percentiles of response time or histograms of request size. A powerful rule definition language is implemented, allowing users to specify various aspects of the metric, including metric name, extraction method, aggregation function, tag set, and update frequency. Rules can extract values ​​from structured logs using regular expressions, JSON paths, or XPath, and can also use more complex expressions for calculation and transformation. It supports multi-level aggregation, allowing values ​​to be extracted at the event level first, and then aggregate functions (such as count, sum, average, maximum / minimum, percentile, etc.) to be applied within a time window. It also supports dynamic label extraction, extracting dimensional information (such as service name, hostname, client type, etc.) from logs as labels for metrics, resulting in richly dimensioned metrics for multi-faceted analysis. Incremental computation and streaming processing technologies are implemented, enabling immediate metric updates upon log event arrival, achieving near real-time metric generation. Metric verification and monitoring mechanisms are also provided to detect abnormal metric behavior (such as sudden value changes or metric disappearance), and debugging tools are offered to help diagnose problems in extraction rules. This transformation engine generates a rich dataset of log-derived metrics, seamlessly integrating log analysis and metric monitoring, providing users with more comprehensive and in-depth system observation capabilities.

[0242] Finally, based on the cleaned structured data stream and log-derived indicator dataset, data association and context enrichment are achieved, adding service topology information and metadata to the observation data to generate observation data with rich context. The system designs a multi-dimensional context enrichment framework, associating various relevant information with the observation data, greatly enhancing the analytical value of the data. First, service topology association is implemented, linking the observation data with the system's service dependency graph. By analyzing distributed tracing data, service registration information, and network traffic, a service topology graph is automatically constructed and maintained, recording the call relationships, dependency structures, and interaction patterns between services. When processing observation data, based on the source service identifier, it is associated with the corresponding node in the topology, adding the upstream dependencies and downstream service information of that service. This topology context is particularly valuable for troubleshooting, helping users quickly understand the scope of the problem and its potential sources. Second, infrastructure and deployment context are added, associating the observation data with its operating environment. The system obtains environmental information from various sources, such as resource management systems (Kubernetes, VM management platforms, etc.), configuration management databases (CMDB), and deployment pipelines, acquiring metadata such as container information, virtual machine configurations, hardware specifications, network settings, and deployment versions. This information helps users understand the physical and logical context of the observed data, such as whether an error occurred on a specific version of the service or whether a performance issue is related to a specific hardware configuration. It also associates business context, such as Service Level Objectives (SLOs), business importance, and responsible teams, enabling operations personnel to prioritize issues from a business impact perspective. Furthermore, historical event association is implemented, linking current observed data with similar historical events, known issues, and solutions. A knowledge base is maintained to record past failure cases, anomaly patterns, and resolution steps; then, similarity algorithms are used to match the current situation with historical records, providing relevant problem-solving suggestions. Entity association technology is employed to link related entities scattered across different observed data (such as request IDs, user sessions, and transactions), building a complete interactive view. Through this multi-dimensional context enrichment, raw observed data is transformed into intelligent data with comprehensive context, not only showing "what happened" but also providing insights into "why it happened" and "how to solve it," greatly enhancing the practical value and decision support capabilities of observable data.

[0243] In the steps described above, a data processing pipeline is a streaming data processing architecture that organizes data processing tasks into a series of consecutive stages or steps. Each stage receives the output of the previous stage as input, performs a specific transformation or processing, and then passes the result to the next stage. The core idea of ​​this pipeline architecture is "divide and conquer," breaking down complex data processing tasks into smaller, more focused components, each responsible for a specific function. A typical data processing pipeline includes multiple stages: a data acquisition stage, which obtains raw data from various source systems; a data parsing stage, which transforms the raw data into a structured format that the system can process; a data cleaning stage, which removes invalid, duplicate, or abnormal data; a data transformation stage, which applies various business rules and calculations; a data enrichment stage, which adds additional contextual information; and a data storage stage, which persists the processed data.

[0244] Log-to-metric transformation refers to the process of converting text-based log data into numerical time-series metrics. Logs and metrics are two fundamental dimensions of observability: logs provide detailed event records with rich contextual information, suitable for in-depth analysis and troubleshooting; metrics are periodically collected numerical data points, suitable for trend monitoring and alerting. Log-to-metric transformation creates a bridge between these two dimensions, enabling log data that was previously only suitable for text search and pattern matching analysis to be used for numerical analysis, trend monitoring, and threshold alerting. This transformation typically involves several typical patterns: event counting, which counts the frequency of specific types of events, such as the number of error logs, authentication failure rate, or API call frequency; value extraction, which extracts explicit numerical values ​​from logs, such as response time, memory usage, or the number of records processed; state tracking, which monitors changes in system state inferred from logs, such as service startup / stop or entering / exiting specific modes; and complex event detection, which identifies event sequences that conform to specific patterns and converts their frequency or duration into metrics. The implementation of log-to-metric transformation is usually based on a rule engine, with user-defined extraction rules (such as regular expressions or structured queries) and aggregation methods (such as counting, summing, averaging, etc.).

[0245] Example 13:

[0246] Building upon Example 12, the implementation of data association and context enrichment, by adding service topology information and metadata to the observation data, generates the observation data with rich context, including:

[0247] Construct a network structure that describes the potential correlation between different types of observation data in the cleaned structured data stream and the log-derived indicator dataset. Initialize the correlation strength using the timestamp field and data source identifier in the unified observation data stream to generate a heterogeneous observation data correlation network.

[0248] Based on the heterogeneous observation data association network, the association relationship between heterogeneous data is parameterized into an Ising model, prior constraints are introduced and modeled as hard constraints, and an Ising model representation of the association relationship is generated.

[0249] Based on the Ising model representation of the aforementioned association relationship, a small-sample association pattern learning algorithm is implemented to learn association patterns from a small number of known association instances and generate association pattern learning results.

[0250] Based on the learning results of the association patterns, association inference is performed under hard constraints. The probability that there is a real association between each pair of data entities in the heterogeneous observation data association network is calculated, and a consistent association network is generated.

[0251] Based on the consistent correlation network, a causal graph of heterogeneous observation data is constructed. The causal graph is then used to automatically add contextual information to the observation data in the cleaned structured data stream and the log-derived indicator dataset, thereby generating the observation data with rich context.

[0252] Specifically, firstly, a network structure is constructed to describe the potential correlations between different types of observation data in the cleaned structured data stream and log-derived indicator dataset. The correlation strength is initialized using timestamp fields and data source identifiers in the unified observation data stream, generating a heterogeneous observation data correlation network. The core challenge is to correlate various types of observation data (such as logs, indicators, traces, and events) to form a unified view of system behavior. First, a heterogeneous network structure is constructed, where nodes represent different observation data entities (such as log events, indicator time series, trace segments, etc.), and edges represent potential correlations between them. The system initializes these correlations using two basic clues: time correlation and source correlation. For time correlation, the timestamp distribution of the observation data is analyzed, connecting events that overlap or are close in time; the correlation strength weakens as time distance increases. Multi-scale time window analysis is implemented, considering both millisecond-level direct causal relationships and long-term impact patterns from minutes to hours. For source correlation, initial connections are established using data source identifiers (such as service names, hostnames, container IDs, etc.), assuming that different types of data from the same source are more likely to be correlated. The study also considered hierarchical source relationships, such as different services within the same container or different containers on the same host, assigning different initial weights to associations at different levels. Network analysis algorithms were applied to process these initial associations, such as a variant of PageRank to identify key nodes in the network, community detection algorithms to discover closely related data clusters, and path analysis algorithms to discover indirect chains of association between data. Particular attention was paid to identifying anomalous patterns, such as sudden changes in association strength or new associations that do not conform to historical patterns; these typically indicate significant changes in system behavior. Through these analyses, a comprehensive heterogeneous observation data association network was generated, capturing the potential association structures between different types of observation data, laying the foundation for subsequent precise association analysis.

[0253] Next, based on the heterogeneous observation data association network, the association relationships between heterogeneous data are parameterized into an Ising model. Prior constraints are introduced and modeled as hard constraints to generate the Ising model representation of the association relationships. The Ising model is a model derived from statistical physics, suitable for describing systems with complex interactions. In the observation data association analysis, the association state between each pair of data entities is represented as a binary variable: association exists or no association exists. A global energy function is defined such that association configurations that conform to actual system behavior correspond to low energy states (high probability), while association configurations that do not conform to actual behavior correspond to high energy states (low probability). This energy function contains various interaction terms: first-order terms reflect the prior probability of a single association, based on the association strength in the initial network; second-order terms represent the mutual influence between associations, such as mutual reinforcement or inhibition when two associations share the same node; higher-order terms capture more complex association patterns, such as the relationship between multiple associations forming a closed loop. Prior constraints are introduced as hard constraints to ensure that the generated association relationships conform to the basic rules and physical constraints of the system. These constraints include: causal consistency constraints, ensuring that associations follow a chronological order and avoiding circular dependencies that violate causal relationships; structural consistency constraints, ensuring that associations conform to the basic structure of the system architecture and service topology, such as the impossibility of direct communication between certain services; and business rule constraints, reflecting domain-specific knowledge, such as the inevitable consequences of certain errors leading to specific system behaviors. A probabilistic graphical model is used to encode these constraints as high-weight terms in the energy function, forcing the model to search for optimal association configurations within the solution space that satisfies the constraints. A constraint prioritization mechanism is also implemented, allowing for the relaxation of certain soft constraints when necessary while strictly adhering to hard constraints to handle noise and incompleteness in the observed data. This parameterized representation transforms the association analysis problem into a structured mathematical optimization problem, which can be efficiently solved using mature inference algorithms.

[0254] Then, based on the Ising model representation of association relationships, a few-shot association pattern learning algorithm is implemented to learn association patterns from a small number of known association instances and generate association pattern learning results. In practical applications, known definitive association instances are usually very limited, thus requiring effective learning of association patterns from a small amount of labeled data. A series of few-shot learning techniques are implemented to fully utilize limited association knowledge. First, a transfer learning method is used, leveraging association pattern knowledge learned in a general domain or similar systems and transferring it to the current specific system to accelerate the learning process. Domain adaptation technology is employed to adjust the general model to adapt to the specific features and behavioral patterns of the current system. Second, a meta-learning strategy is used, where the system "learns how to learn" association patterns by training a meta-model on multiple related tasks, enabling the system to quickly adapt to new association recognition tasks. In the specific implementation, the system uses a model-independent meta-learning algorithm, which can be seamlessly integrated with the Ising model inference framework. Active learning technology is also applied, intelligently selecting the most informative instances for labeling to maximize the learning effect of limited labeling resources. A combined strategy of uncertainty sampling and representative sampling is used, paying attention to associations that are uncertain in the model while ensuring that sampling covers different types of association patterns. Special attention is paid to zero-shot and few-shot generalization capabilities. By combining prior knowledge, structural information, and semantic features, the model can infer association types that have never been directly observed. An incremental learning mechanism is implemented, enabling the model to be continuously updated and improved as new labeled data becomes available without requiring complete retraining. To verify the reliability of the learning results, the system employs statistical techniques such as cross-validation and bootstrap to evaluate the uncertainty of model parameters and the confidence intervals of predictions. Through these few-shot learning techniques, rich association pattern knowledge is extracted from a limited number of known associations, laying the foundation for subsequent comprehensive association inference.

[0255] Next, based on the association pattern learning results, association inference is performed under hard constraints. The probability of a true association between each pair of data entities in the heterogeneous observation data association network is calculated, generating a consistent association network. Using the learned Ising model parameters, probabilistic inference is performed on all possible associations in the observation data. This inference process faces two main challenges: high computational complexity, requiring execution on large-scale datasets; and the strict satisfaction of hard constraints such as causal consistency. A combination of several efficient inference algorithms is implemented: for relatively independent subgraphs in the network, precise inference methods, such as those combining tree message passing algorithms, are used to obtain accurate association probabilities; for large-scale complex subgraphs, approximate inference methods, such as Markov chain Monte Carlo (MCMC) sampling or variational inference, are employed to obtain good approximate results within acceptable computational costs. Inference under constraints is specifically optimized, implementing a constraint satisfaction algorithm based on truncated sampling, sampling only in the state space that satisfies all hard constraints, ensuring the consistency of the inference results. Distributed computing technology is also applied, decomposing the large-scale inference task into multiple parallel subtasks, significantly improving the ability to process large amounts of observation data. For each pair of data entities, the posterior probability of a genuine association between them is calculated, taking into account prior knowledge, observational evidence, and learned association patterns. A probability threshold is set (usually dynamically adjusted based on the precision and recall requirements of a specific application scenario), and associations exceeding the threshold are established as valid associations, constructing the final association network. Special attention is paid to the transitive closure of associations to ensure that the inferred associations are consistent overall, without loops or conflicts that violate causal logic. Through this rigorous inference process, a highly reliable association network is generated, accurately reflecting the intrinsic connections between different types of observational data.

[0256] Finally, based on a consistent correlation network, a causal graph of heterogeneous observation data is constructed. This causal graph is used to automatically add contextual information to the observation data in the cleaned structured data stream and log-derived indicator datasets, generating observation data with rich context. The correlation network obtained in the previous step is transformed into a directed graph structure, i.e., the causal graph, explicitly representing the causal relationships and dependency structures between observation data. Causal discovery algorithms are applied, including Granger causal analysis based on time series and PC algorithms based on conditional independence tests, to infer the direction of correlations and distinguish between causes and effects within causal relationships. The system also integrates domain knowledge, such as service call relationships, system component dependencies, and known fault propagation patterns, to assist and verify the automatically discovered causal relationships. After constructing the complete causal graph, it serves as a framework for adding contextual information. For each observation data point, its upstream causes and downstream influences are traced along the causal graph to enrich its contextual information. Specifically, multiple contexts are added to each event or metric: causal context, including possible causal events and affected downstream events; temporal context, including the absolute and relative time of the event (e.g., time since system startup or service deployment) and its position in the relevant event sequence; spatial context, including physical or logical location information such as the service, node, and geographic location where the event occurred; state context, including the overall state of the system at the time of the event, the health status of relevant components, and load levels; and business context, including the potential impact of the event on business metrics, relevant Service Level Objectives (SLOs), and business processes. A hierarchical organization of contextual information is implemented, allowing users to expand or collapse different levels of detail as needed, avoiding information overload. Context-related recommendations are also provided, such as handling methods for similar historical events, potentially useful diagnostic commands, or potential solution steps. This comprehensive enrichment of context transforms raw observational data into high-value data with rich context, greatly enhancing the usability and insight value of observable data, enabling users to understand system behavior and root causes of problems more quickly and deeply.

[0257] In the steps described above, the heterogeneous observation data association network is a key technical term that requires detailed explanation. It is a special network structure used to represent potential associations between different types of observation data (such as logs, metrics, tracks, and events). In this network, nodes represent various observation data entities, such as single log records, metric time series, track fragments, or system events; edges represent potential associations between nodes, typically with weights indicating the strength or reliability of the association. The heterogeneity of this network is reflected in several aspects: heterogeneity of node types—different types of observation data have different structures, semantics, and characteristics; heterogeneity of association types—associations may be based on different criteria such as time correlation, causality, shared source, or content relevance; and heterogeneity of time scales—associations may exist at different time ranges, such as direct causal relationships at the millisecond level and long-term impact patterns at the hour level.

[0258] Hard constraints refer to conditions or rules that must be strictly satisfied during modeling and optimization, with no violations or relaxations permitted. In the context of observational data correlation analysis, hard constraints reflect fundamental physical or logical limitations on system behavior, ensuring that inferred relationships conform to basic rules of the real world. Typical hard constraints include: causal consistency constraints, ensuring relationships respect temporal order, i.e., causes must precede effects, preventing circular dependencies that violate causality; structural consistency constraints, ensuring relationships are consistent with the system's infrastructure and topology, such as no direct association between components without physical connections; and domain-specific rule constraints, such as specific errors necessarily leading to certain subsequent behaviors, or component failures necessarily causing specific service unavailability. In model implementation, hard constraints are typically enforced in several ways: assigning extremely high weights to the energy function, so that states violating constraints correspond to extremely high energy (extremely low probability); restricting the state space, performing inference only on a subset of configurations that satisfy all hard constraints; or using special inference algorithms, such as constraint satisfaction propagation or truncation sampling, to ensure that generated samples always satisfy all hard constraints.

[0259] A causal graph is a directed graph structure that explicitly represents the causal relationships and dependencies between events, states, or entities in a system. Unlike ordinary relational networks, causal graphs emphasize the directionality of relationships, distinguishing between the fundamentally different relationships of "A causes B" and "B causes A." In observability analysis, causal graphs typically contain multiple node types, such as event nodes (representing specific events occurring in the system), state nodes (representing the state of the system or components), and entity nodes (representing components or resources in the system); edges represent causal relationships and usually have attributes such as strength, delay time, and triggering conditions. The construction of causal graphs often combines multiple methods: data-driven causal discovery, using statistical methods and machine learning algorithms to infer causal relationships from observational data; knowledge-driven structure guidance, utilizing domain knowledge, design documents, and expert experience to define basic causal structures; and hybrid methods, using known partial causal relationships as priors to guide the data-driven discovery process.

[0260] Example 14:

[0261] like Figure 1 As shown, based on Example 10, a visualization analysis platform is generated based on the columnar storage structure, the optimized data storage structure, and the high-value observation data with rich context, including:

[0262] Based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, a unified query engine supporting multiple query languages ​​is realized. The optimal execution path is selected according to the data characteristics in the column-level metadata and the optimized data storage structure to generate query execution capabilities.

[0263] Based on the query execution capability, multi-level query optimization is achieved. The query plan is automatically rewritten based on the query conditions and the data statistics in the column-level metadata, realizing vectorized execution and parallel processing. The delayed materialization technology of the columnar storage structure is adopted to realize the intelligent caching mechanism to cache hot data in the hierarchical storage dataset, thereby generating a high-performance query processing system.

[0264] Based on the high-performance query processing system, an adaptive query resource management mechanism is implemented, which dynamically allocates computing resources according to query complexity, the amount of data in the columnar storage structure, and system load, supports query truncation and progressive result return, and obtains a stable query service.

[0265] Based on the stable query service, an integrated front-end visualization interface is provided, enabling data exploration tools to allow users to interactively analyze the observation data with rich context. An alarm configuration interface and an investigation and analysis toolchain are provided to generate the visualization analysis platform.

[0266] Specifically, firstly, based on a columnar storage structure, optimized data storage structure, and observational data with rich context, a unified query engine supporting multiple query languages ​​is implemented. The optimal execution path is selected based on the data characteristics in column-level metadata and the optimized data storage structure to generate query execution capabilities. A flexible multi-language query processing architecture is designed, capable of uniformly handling various observability query languages, including SQL (for structured data queries), PromQL (Prometheus query language for time series data analysis), LogQL, and TraceQL. The system implements a modular query parsing framework, containing dedicated parsers for each query language, capable of converting queries from different languages ​​into a unified internal representation (IR). This representation captures the core semantics of the query, such as data source, time range, filtering conditions, and aggregation operations, independent of the syntactic details of specific query languages. Query semantic validation is implemented to ensure that queries are logically valid and to resolve semantic compatibility issues in cross-data type queries, such as the semantics of join operations between logs and metrics. Based on the unified internal representation, intelligent query routing and execution path selection are implemented. The system analyzes query characteristics (such as data type, time range, selectivity, complexity, etc.) and data characteristics (understanding data distribution, correlation, access patterns, etc. through column-level metadata) to select the optimal execution strategy for each query. It supports multiple execution paths: for simple, high-selectivity queries, it directly routes to index lookup; for time-series aggregation, it uses a specially optimized time-series processing engine; and for complex cross-data type queries, it employs a distributed execution framework. Special attention is paid to optimizing queries on a hierarchical storage architecture, adjusting execution strategies based on the storage layer (hot, warm, or cold) where the data accessed by the query resides. For example, it prioritizes in-memory computation for hot data and applies pre-fetching and batch processing techniques for cold data. The system also implements a query optimizer that can rewrite and optimize query plans based on statistical information and cost models, ensuring efficient execution even for complex queries. This unified query processing architecture enables consistent access to various observational data, allowing users to access any type of observational data using the most suitable query language while maintaining optimal query performance.

[0267] Next, based on query execution capabilities, multi-level query optimization is implemented. The query plan is automatically rewritten based on query conditions and data statistics in column-level metadata, achieving vectorized execution and parallel processing. Delayed materialization technology using a columnar storage structure is employed, and an intelligent caching mechanism is implemented to cache hot data in the hierarchical storage dataset, generating a high-performance query processing system. A multi-level query optimization framework is implemented, applying different types of optimization techniques at different stages. In the logical optimization stage, a series of rule-based query rewriting techniques are implemented, such as predicate pushdown (applying filtering conditions as early as possible to reduce the amount of data processed later), column pruning (reading only the columns required by the query to avoid unnecessary I / O), constant folding (pre-computing constant expressions in the query), and subquery flattening (converting nested subqueries into join operations). Time-series data queries are particularly optimized, implementing dedicated techniques such as time window merging, downsampling optimization, and pre-computed aggregation. In the physical optimization stage, the most suitable algorithm and execution strategy are selected based on data characteristics and query characteristics, such as choosing the best index access method for selection operators and nested loops, hash joins, or sort-merge algorithms for join operators. A cost-model-driven optimizer was implemented, taking into account I / O costs, CPU costs, and memory usage to generate optimal execution plans for complex queries. At the execution engine level, a vectorized execution model was implemented, processing batches of data instead of rows at a time, fully utilizing modern CPU caches and SIMD instructions to significantly improve computational efficiency. Multi-level parallel processing was also implemented: data-level parallelism, processing different data partitions simultaneously; operation-level parallelism, executing different parts of complex queries in parallel; and pipelined parallelism, starting the next operation as soon as the previous operation produces its first batch of results, reducing waiting time. Delayed materialization, a technique unique to columnar storage, was employed, processing only necessary column and row IDs in columnar storage for as long as possible during computation, and materializing only in the final stage when a complete record is needed, significantly reducing memory usage and data movement. An intelligent multi-level caching mechanism was also implemented: query result caching, storing complete query results to handle frequently repeated queries; intermediate result caching, storing intermediate outputs of expensive computations for optimizing queries with common substructures; and data block caching, caching frequently accessed data blocks to reduce I / O operations. The caching system employs a data access pattern-aware replacement strategy, prioritizing the retention of frequently accessed data and implementing a proactive prefetching mechanism to load potentially needed data in advance based on query patterns. Through these multi-layered optimization techniques, a highly efficient query processing system is built, capable of rapidly responding to observation data queries of varying complexity.

[0268] Then, based on the high-performance query processing system, an adaptive query resource management mechanism is implemented. This mechanism dynamically allocates computing resources according to query complexity, the amount of data in the columnar storage structure, and system load, supporting query truncation and incremental result return to achieve stable query services. A comprehensive resource management framework is designed to ensure system stability and responsiveness under various load conditions. First, a query complexity assessment mechanism is implemented, capable of analyzing resource requirements before query execution. This assessment is based on multiple factors, such as the query operation type (scanning, filtering, aggregation, etc.), the amount of data involved (estimated by accessing column-level metadata), the selectivity of filtering conditions, and the estimated result set size. Resource quotas are allocated to each query, including CPU time, memory usage, and I / O bandwidth, dynamically adjusted based on query priority, user role, and estimated complexity. The system implements a multi-level resource protection mechanism: admission control, which delays or rejects low-priority queries when system load is high; query queuing, which queues queries according to priority and resource requirements to ensure fair resource allocation; and execution control, which monitors query resource usage and takes measures for queries exceeding allocated quotas. Special attention is paid to complex, long-running queries. A query truncation technique is implemented, allowing query execution to terminate when preset resource limits or timeout thresholds are reached, but returning partially processed results. This mechanism ensures responsiveness even with overly complex queries, while providing users with valuable preliminary results. Progressive result return is also implemented, allowing queries to gradually return computed partial results during processing, rather than waiting for the entire query to complete. This is particularly useful for exploratory analysis of large-scale data, allowing users to see preliminary results and decide whether to wait for the full results or adjust their query strategy. The system implements query monitoring and adaptive adjustment mechanisms, tracking query execution status, resource usage, and progress in real time. When performance issues or resource pressure are detected, execution strategies are dynamically adjusted, such as reducing parallelism, simplifying execution plans, or proactively prompting users to optimize queries. Through these adaptive resource management techniques, high query performance is maintained while ensuring overall service stability and reliability, providing consistent service quality even under complex queries and high load conditions.

[0269] Finally, based on a stable query service, an integrated front-end visualization interface is provided, enabling data exploration tools that allow users to interactively analyze observational data with rich context. An alarm configuration interface and a survey analysis toolchain are also provided, generating a visualization analysis platform. The system features a comprehensive front-end interface that presents complex data analysis capabilities to users in an intuitive and easy-to-use manner. The interface adopts a modular design, containing multiple functional areas: a data browsing area, displaying available data sources and data structures, supporting filtering and searching based on tags, time ranges, and data types; a query building area, providing an intuitive query building interface, supporting natural language query input, a visual query designer, and a traditional query editor; a visualization display area, flexibly displaying query results, supporting various chart types and interaction methods; and a dashboard area, allowing users to organize and save analysis views and build customized monitoring dashboards. It features a rich library of visualization components to adapt to different types of observational data: for time series metrics, it provides line charts, area charts, heatmaps, scatter plots, etc., supporting multi-series comparisons, trend analysis, and anomaly highlighting; for log data, it offers advanced text analysis views, supporting pattern highlighting, field extraction, and time distribution display; for tracing data, it provides flame charts, timeline views, and service topology diagrams to intuitively display request execution paths and performance bottlenecks; for events and alarms, it provides timeline views and state transition diagrams to clearly show system state changes. It pays special attention to the data exploration experience, implementing various interactive features: drill-down analysis, allowing users to delve deeper into detailed information from overview data; correlation analysis, establishing and displaying relationships between different data sources; time synchronization, maintaining consistency of time context across different views; and context-aware recommendations, automatically suggesting relevant data views or analysis paths based on the current analysis content. It also provides complete alarm management functions, including expression-based alarm rule configuration, automatic alarm threshold suggestion, alarm notification channel management, and alarm history review. To support in-depth problem investigation, the system implements a toolchain for investigation and analysis, including a root cause analysis wizard, performance profiling tools, anomaly detection modules, and problem diagnosis workflow templates. It supports the sharing and collaboration of analysis results, allowing users to save and share queries, charts, dashboards, and investigation findings, promoting team collaboration and knowledge transfer. Through this comprehensive front-end interface, the underlying data processing and analysis capabilities are transformed into intuitive and practical visual analysis tools, enabling users of all skill levels to effectively utilize observational data for system monitoring, problem diagnosis, and performance optimization.

[0270] In the steps described above, the unified query engine, an integrated query processing system capable of handling multiple query languages ​​and data types, plays a central role in the observability platform. Traditional observability tools typically provide independent query interfaces and languages ​​for different types of data (such as logs, metrics, and traces), requiring users to switch between different tools and making cross-data type correlation analysis difficult. The unified query engine solves this problem by providing a single query entry point that understands and processes multiple query languages, such as SQL, PromQL, and LogQL, while also handling data with different structures, from structured time-series metrics to semi-structured log and trace data.

[0271] Example 15:

[0272] like Figure 1 As shown, based on Example 14, the implementation of multi-level query optimization, automatically rewriting the query plan according to the query conditions and the data statistics information in the column-level metadata, realizing vectorized execution and parallel processing, adopting the delayed materialization technology of the columnar storage structure, and implementing an intelligent caching mechanism to cache hot data in the hierarchical storage dataset, generates a high-performance query processing system, including:

[0273] Before executing the full query, a sublinear algorithm is applied to estimate the size of the query results. An approximate value of the result set size is obtained by performing a random query on the optimized diversified index structure. The query execution strategy is optimized based on the estimated result set size, and a query strategy optimization scheme is generated.

[0274] Based on the query strategy optimization scheme, for analytical queries, an approximate query processing based on sublinear sampling is implemented. The sampling strategy is designed according to the query conditions and the statistical characteristics in the column-level metadata. When processing a very small proportion of data in the columnar storage structure, an approximate result is provided, and an approximate query result is generated.

[0275] Based on the query strategy optimization scheme, an optimization algorithm for sub-linear complexity is implemented for aggregation operations. The data statistical characteristics in the column-level metadata and the optimized diversified index structure are used to avoid full table scans and generate aggregation optimization results.

[0276] According to the query strategy optimization scheme, for complex queries involving the association of multiple observation data in the observation data with rich context, the sublinear graph algorithm is applied to accelerate the association query, realize the rapid identification of key paths and abnormal patterns in the causal graph, and generate accelerated association query results.

[0277] Based on the query strategy optimization scheme, the approximate query results, the aggregation optimization results, and the correlation query acceleration results, an adaptive query execution strategy is implemented, and a progressive processing method is adopted for complex queries to generate the high-performance query processing system.

[0278] Specifically, firstly, before executing the full query, a sublinear algorithm is applied to estimate the size of the query results. An approximate value for the result set size is obtained by executing random queries on an optimized, diversified index structure. Based on the estimated result set size, the query execution strategy is optimized, generating an optimized query strategy scheme. A series of sublinear time complexity algorithms are implemented, enabling rapid estimation of the query result size without executing the full query. The system uses probabilistic data structures and sampling techniques, performing lightweight exploratory analysis before query execution. For range queries and equality queries, histograms and Bloom filters on column-level indexes are used for basic estimation, providing preliminary selectivity information. For more complex query conditions, a multi-stage sampling technique is implemented: first, stratified random sampling, drawing samples from different data partitions and time ranges to ensure representativeness; then, conditional sampling, applying query conditions to the samples and calculating the proportion satisfying the conditions; finally, statistical amplification, amplifying the sample results to the full dataset level based on the sampling proportion and data distribution characteristics. Special attention is paid to potential estimation errors, implementing confidence interval calculations to provide not only point estimates but also the range of possible estimation errors. For complex multi-condition queries, relevance analysis and Bayesian inference are used to consider the mutual influence between different conditions, improving the accuracy of the joint selection rate estimation. Historical query statistics are also maintained, and current estimates are adjusted and improved using the actual result set sizes of similar queries. Based on these estimation results, query execution strategies are optimized, making key decisions such as whether to use index scans or full table scans, whether to apply parallel processing, how much memory and computing resources to allocate, and whether to apply approximate query processing techniques. For queries with very large estimated result sets, users may be advised to refine query conditions, or constraints may be automatically applied to avoid resource exhaustion. For queries with smaller estimated result sets, resources may be prioritized to ensure a fast response. The system also determines whether to pre-allocate a result buffer or use streaming processing to return results based on the estimated result set size, optimizing memory usage. Through this query optimization based on sublinear estimation, the system can "anticipate" the resource requirements and execution characteristics of queries before execution, making more informed optimization decisions and significantly improving the efficiency and reliability of query processing.

[0279] Next, based on the query strategy optimization scheme, an approximate query processing based on sublinear sampling is implemented for analytical queries. A sampling strategy is designed according to the query conditions and statistical characteristics in column-level metadata to provide approximate results when processing a very small proportion of data in the columnar storage structure, generating approximate query results. A comprehensive approximate query processing framework is implemented specifically for large-scale analytical queries, such as aggregation, statistical, and trend analysis. These queries typically do not require results accurate to the last digit; users are more concerned with overall trends and patterns and can accept approximate values ​​within a certain error range in exchange for significant performance improvements. First, the most suitable sampling strategy is determined based on query characteristics and data distribution. For uniformly distributed data, simple random sampling is used, drawing samples from the entire dataset with equal probability. For skewed distributed data, stratified sampling and weighted sampling techniques are implemented to ensure that rare but important data subsets are adequately represented. Special attention is paid to the sampling of time-series data, implementing a time-aware sampling strategy to ensure uniform sampling across the entire time range and avoid bias caused by time clustering effects. For data with multidimensional attributes, multidimensional stratified sampling is used to maintain the representativeness of the data distribution across each dimension. The sampling ratio is dynamically determined based on the query's precision requirements, data distribution characteristics, and available computing resources. For queries requiring high precision, a sampling rate of 10-20% may be used; while for rapid exploratory analysis, a sampling rate of 1% or lower may be used. After sampling, the system executes the standard query processing flow, but only processes the sampled data, significantly reducing computational load and memory usage. For aggregated results, statistical adjustments are applied to compensate for sampling biases and improve the accuracy of approximate results. The system also calculates and returns the confidence interval or error range of the results, allowing users to understand the reliability of the approximations. For time series data, an adaptive sampling technique is specifically implemented, increasing the sampling rate during periods of rapid data change and decreasing it during stable periods, optimizing the balance between precision and performance. Multi-level approximations are supported, allowing users to first view the quickly generated coarse results and then gradually refine the analysis by increasing precision as needed. This approximate query processing framework enables high-quality analytical results with a minimal data processing ratio, making large-scale data analysis more interactive and responsive, greatly improving user experience and analytical efficiency.

[0280] Then, based on the query strategy optimization scheme, an optimization algorithm with sub-linear complexity was implemented for aggregation operations. This leverages the statistical characteristics of data in column-level metadata and optimized, diversified index structures to avoid full table scans and generate optimized aggregation results. A series of specialized algorithms were designed to enable common aggregation operations (such as COUNT, SUM, AVG, MIN, MAX, etc.) to be executed with sub-linear time complexity, avoiding the processing of all data. First, partition-level metadata optimization was applied, using pre-computed partition statistics to quickly eliminate irrelevant data partitions. For example, for MIN / MAX queries, the statistical metadata of each partition (such as minimum value, maximum value, null value ratio, etc.) was checked to directly exclude partitions that could not possibly contain results; for COUNT queries, the row count of the partition might be used directly, without scanning the actual data. Grouped aggregation optimization was implemented, directly calculating some aggregation results using the statistical information within the partition. For queries grouped by time or other dimensions, basic aggregations might have already been pre-computed at the partition level; only these pre-computed results needed to be integrated, without processing the original data. Time-series data aggregation was particularly optimized, implementing multi-level downsampling and pre-aggregation techniques. Aggregate values ​​at different time granularities (e.g., minutes, hours, days) are pre-calculated during data writing. Queries directly use the pre-calculated results closest to the requested granularity, significantly reducing processing load. For complex conditional aggregations, index access and statistical inference techniques are combined. The index is used to find the set of record IDs that meet the conditions, and then bitmap operations are used to efficiently calculate the aggregation results, avoiding access to actual data values. The system also implements approximate aggregation algorithms, such as HyperLogLog (for cardinality estimation), Count-Min Sketch (for frequency estimation), and t-digest (for percentile calculation). These algorithms use a fixed amount of memory and provide high-quality approximate aggregation results with constant time complexity, regardless of the data size. Outlier handling is optimized, and an outlier-aware aggregation strategy is implemented to identify and appropriately handle extreme values, avoiding their disproportionate impact on the aggregation results. Through these sub-linear complexity aggregation optimization algorithms, computational load and I / O burden are significantly reduced when processing large-scale datasets, lowering the response time of aggregation queries from minutes to seconds or milliseconds, greatly improving the interactivity and efficiency of the analysis.

[0281] Next, based on the query strategy optimization scheme, for complex queries involving multiple observation data associations within rich contextual observation data, sublinear graph algorithms are applied to accelerate association queries, enabling rapid identification of key paths and anomaly patterns in causal graphs and generating accelerated association query results. The core challenge is efficiently processing complex queries involving large-scale network structures (such as causal graphs, service topology graphs, or event association networks), which traditionally require traversing the entire graph structure, resulting in high computational complexity. Several sublinear graph algorithms were implemented, significantly reducing the complexity of these queries. For path queries (such as "finding all possible paths from service A to service B" or "identifying the event chain leading to a specific error"), a bidirectional search algorithm was implemented, searching simultaneously from both the starting and ending points, meeting in the middle, greatly reducing the search space. A... Algorithms and heuristic search leverage the topological properties and node attributes of the graph to guide the search direction, prioritizing the exploration of regions more likely to contain the target path. For connectivity and reachability analysis, jump pointer and tree-cover techniques are implemented, pre-computing and maintaining the hierarchical structure information of the graph, enabling connectivity queries to be completed in near constant time, rather than linear traversal. Centrality and influence analysis are optimized by implementing sampling-based approximation algorithms, such as random walk approximate centrality, which can accurately estimate the centrality and influence range of each node while handling a small subset of the graph's nodes. For community detection and pattern mining, locality-sensitive hashing and label propagation algorithms are applied, enabling the identification of community structures and anomalous patterns in the graph in linear or sublinear time. Special attention is paid to temporal graph analysis, implementing incremental computation and time window techniques to efficiently handle dynamically changing graph structures and track the evolution of causal relationships and dependency structures. Visualization-related graph queries are also optimized, such as selecting the most representative subgraph for interactive displays, implementing graph sampling algorithms based on influence and representativeness, which can significantly reduce the number of nodes that need to be processed and displayed while preserving key structural features. It integrates GPU acceleration technology, leveraging the parallel computing capabilities of graphics processors to accelerate large-scale graph analysis algorithms, such as shortest path calculation, centrality analysis, and graph matching. Through these sublinear graph algorithms, it can efficiently process complex analysis queries based on causal graphs, reducing graph analysis operations that previously took minutes or even hours to seconds or milliseconds. This makes complex association analysis and root cause diagnosis interactive and feasible in real time, greatly improving the efficiency and depth of problem investigation.

[0282] Finally, based on the query strategy optimization scheme, approximate query results, aggregation optimization results, and correlation query acceleration results, an adaptive query execution strategy is implemented. A progressive processing method is adopted for complex queries, generating a high-performance query processing system. An intelligent query execution scheduler is designed, capable of dynamically selecting and adjusting the most suitable execution strategy based on query characteristics, data characteristics, and system status. A query classification mechanism is implemented, categorizing queries according to multiple dimensions, such as accuracy requirements (exact or approximate queries), time sensitivity (real-time or batch processing queries), resource requirements (lightweight or heavyweight queries), and priority (high-priority monitoring queries or low-priority analysis queries). Different execution strategies are applied according to the query category: for exact and lightweight queries, traditional exact calculation methods are used to ensure complete accuracy of the results; for approximate but fast exploratory queries, sampling and approximation algorithms are applied, prioritizing response speed; for large-scale aggregation queries, sub-linear aggregation optimization and distributed computing are used; for complex graph structure queries, specialized graph algorithm acceleration techniques are applied. The execution of complex queries is particularly optimized, implementing a progressive processing framework that decomposes the query into multiple stages, each generating meaningful intermediate results. This approach allows users to see results early in the query execution process, without waiting for the full computation to complete. For example, for complex root cause analysis queries, the most obvious related events might be returned first, followed by progressively deeper analysis to uncover more indirectly related factors. Adaptive control of query execution is implemented, continuously monitoring query performance and resource usage, and dynamically adjusting the execution plan. If execution bottlenecks or resource pressure are detected, the system may adjust parallelism, switch algorithms, simplify the execution plan, or apply more aggressive approximations. The system also implements an intermediate result reuse mechanism, identifying shared computational parts between queries and reusing already computed results to reduce redundant work. Special attention is paid to user experience, providing query progress estimation and result preview functions, enabling users to understand the query execution status and make informed decisions to wait or cancel. A query suggestion mechanism is also implemented, automatically analyzing query patterns and identifying potential optimization opportunities, such as suggesting the creation of useful indexes, pre-aggregated views, or optimizing the query structure. Through this adaptive, multi-strategy query execution framework, optimal performance and user experience can be provided in various query scenarios. Whether it's a simple monitoring query or a complex deep analysis, it can be executed efficiently in the most suitable way, truly achieving a high-performance, flexible, and user-friendly query processing system.

[0283] In the above steps, sublinear complexity specifically refers to the case where the algorithm's time complexity is strictly lower than linear (O(n)), including logarithmic complexity, square root complexity, or constant complexity. In the context of data processing, sublinear algorithms can complete computational tasks without processing all n data items, which is crucial for large-scale data analysis. Sublinear complexity algorithms are typically implemented in several ways: leveraging the inherent structure and characteristics of the data, such as orderliness, sparsity, or locality; using pre-computed summaries or indexes to avoid accessing the original data; employing probabilistic and approximation methods to trade small probability errors for significant performance gains; and using hierarchical or recursive strategies to decompose the problem into smaller subproblems for efficient solution. Sublinear algorithms are particularly valuable in aggregation computations: for COUNT operations, pre-computed counts and indexes can be used; for SUM / AVG, hierarchical aggregation and sampling estimation can be used; for MIN / MAX, results can be directly derived using index boundaries; and for grouped aggregation, histograms and distribution statistics can be used to reduce processing volume. In big data environments, sublinear algorithms are key to enabling interactive analytics. They reduce queries that would otherwise take hours to complete to seconds, providing real-time support for data exploration and decision-making. Sublinear algorithms fundamentally differ from the traditional "process all data row by row" approach; they represent a more intelligent and efficient data processing paradigm, particularly well-suited for modern large-scale analytics applications.

[0284] Incremental processing is a query execution strategy that breaks down complex queries into multiple stages, each generating meaningful partial results, which are continuously refined and improved as processing progresses. Unlike the traditional "all or nothing" query model, incremental processing allows users to view and analyze preliminary results before the query is fully completed, and then the results gradually become more complete and accurate as more data is processed. The core concept of incremental processing is "fast first, then precise," prioritizing providing quick but potentially incomplete results, and then gradually improving accuracy and completeness. This approach is particularly suitable for exploratory analysis and interactive data exploration. Incremental processing is typically implemented through several techniques: incremental computation, processing a portion of the data to generate preliminary results and then gradually adding more data; hierarchical processing, providing a high-level overview first and then gradually delving into more detailed levels based on user interests; priority processing, processing the most important or most informative data first; and online aggregation, continuously updating aggregated results instead of recalculating.

[0285] Example 16:

[0286] like Figure 2 As shown, the present invention also provides an observable system for unified log metrics and tracking data, comprising:

[0287] The data acquisition and mapping module 10 is used to acquire observation data from different sources, collect and parse the observation data, map the parsed observation data to a unified internal data model and establish correlation mapping between different types of data to generate a unified observation data stream.

[0288] The columnar storage module 20 is used to partition and block the data according to the unified observation data stream, analyze the data characteristics of each column and automatically select the encoding method and compression algorithm, create and maintain column-level metadata, and generate a columnar storage structure.

[0289] The hierarchical storage module 30 is used to monitor and record data access according to the columnar storage structure, identify data access patterns, calculate data value scores, divide data into hot data, warm data and cold data according to the data value scores and store them in different storage media to generate a hierarchical storage dataset.

[0290] The index optimization module 40 is used to analyze user query behavior based on the hierarchical storage dataset, identify query pattern characteristics, formulate indexing strategies for different data layers, implement and maintain multiple types of index structures, optimize data layout, and generate optimized data storage structures.

[0291] The data augmentation module 50 is used to perform cleaning and structured augmentation operations on the observation data based on the unified observation data stream, realize the conversion from logs to indicators, add contextual information to the observation data, and generate observation data with rich context.

[0292] The visualization analysis module 60 is used to implement a unified query engine based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, to perform query optimization and resource management, provide a visualization interface, and generate a visualization analysis platform.

[0293] Through the detailed description of the above embodiments, those skilled in the art should understand that the present invention discloses a unified log metric and tracking data observability method and system. Through unified data model, columnar storage, intelligent hierarchical and dynamic optimization and other technical means, it solves the problems of data silos, storage redundancy and resource waste in the existing observation system, and realizes efficient and unified observation data management and analysis capabilities.

[0294] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for making log metrics and tracking data observable, characterized in that, include: Acquire observation data from different sources, collect and parse the observation data, map the parsed observation data to a unified internal data model, establish correlation mappings between different types of data, and generate a unified observation data stream, including: Based on different types of observation data, raw observation data is received through multiple interfaces to form an initial multi-source data stream. This observation data includes log data, indicator data, and tracking data. According to the initial multi-source data stream, the data format is automatically identified, and key fields and metadata are extracted. For semi-structured logs, heuristic algorithms are applied for automatic pattern recognition and field extraction to generate a structured intermediate dataset. Based on this structured intermediate dataset, different types of observation data are mapped to a unified internal data model. This unified internal data model includes timestamp fields, data source identifiers, resource attributes, tag sets, indicator values, log content, and tracking information, generating observation data entities in a unified format. Based on these unified format observation data entities, metadata is automatically enhanced, and association mappings between different types of data are established to generate the unified observation data stream. Based on the unified observation data stream, the data is partitioned and columnarly divided into blocks. The data characteristics of each column are analyzed, and the encoding method and compression algorithm are automatically selected. Column-level metadata is created and maintained, and a columnar storage structure is generated. This includes: intelligent partitioning based on the unified observation data stream, according to the timestamp field and data source identifier in the unified internal data model, dividing the data into logical partitions and column-based blocks to form a set of columnar data blocks; real-time analysis of the data characteristics of each column is performed based on the set of columnar data blocks, and the encoding method is automatically selected to generate an encoded columnar dataset; based on the encoded columnar dataset, compression algorithms are dynamically selected and applied for data with different characteristics and system resource conditions to generate compressed columnar data blocks; and column-level metadata is created and maintained based on the compressed columnar data blocks, including data type, encoding method, compression algorithm, and data block location, to generate the columnar storage structure. Based on the columnar storage structure, data access is monitored and recorded, data access patterns are identified, data value scores are calculated, and data is divided into hot data, warm data, and cold data according to the data value scores and stored in different storage media to generate a hierarchical storage dataset. Based on the hierarchical storage dataset, user query behavior is analyzed, query pattern characteristics are identified, indexing strategies are formulated for different data layers, multiple types of index structures are implemented and maintained, data layout is optimized, and an optimized data storage structure is generated. Based on the unified observation data stream, cleaning and structure enhancement operations are performed on the observation data to realize the conversion from logs to indicators, add contextual information to the observation data, and generate observation data with rich context. Based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, a unified query engine is implemented to optimize queries and manage resources, provide a visual interface, and generate a visual analysis platform.

2. The method according to claim 1, characterized in that, The method of applying heuristic algorithms to semi-structured logs for automatic pattern recognition and field extraction to generate structured intermediate datasets includes: Based on the original log data stream in the initial multi-source data stream, a multi-dimensional feature space is established and multiple sorting methods are predefined to form multiple sorting sets; Based on the multiple sorting sets, a parallel binary search strategy is executed in multiple sorting directions. Each binary search query determines a log subset half-space that conforms to a specific pattern feature, generating multiple half-space query results. Based on the query results of the multiple half-spaces, the intersection of the multiple half-spaces is calculated to form a log pattern description and generate a half-space intersection model. Based on the semi-space intersection model, the subsequent query strategy is dynamically adjusted. When the confidence of pattern recognition reaches a preset threshold, the query is terminated, and a log pattern model is generated. Based on the log pattern model, the original log data stream is parsed and fields are extracted to generate the structured intermediate dataset.

3. The method according to claim 1, characterized in that, The intelligent partitioning based on the timestamp field and data source identifier in the unified internal data model divides the data into logical partitions and columns, forming a set of columnar data blocks, including: Based on the unified observation data stream, the correlation between observation data in the unified internal data model is modeled as a graph structure, realizing a graph sampling algorithm with sublinear complexity and generating graph sampling results; Based on the graph sampling results, a sublinear time edge counting algorithm and a community detection algorithm are applied to generate graph structure feature estimates. Based on the graph structure feature estimation, a topology-aware data partitioning strategy is generated to ensure that highly correlated data identified in the graph structure feature estimation are assigned to the same or adjacent partitions, thereby generating a topology-aware partitioning scheme. Based on the topology-aware partitioning scheme, the optimal size of each partition is adaptively determined and a progressive partitioning adjustment mechanism is implemented to generate the columnar data block set.

4. The method according to claim 1, characterized in that, The process of real-time analysis of the data characteristics of each column and automatic selection of the encoding method to generate an encoded column dataset includes: Based on the column data in the columnar data block set, a multidimensional description space of data characteristics is constructed, and a characteristic space structure is generated. Based on the feature space structure, an active learning strategy is implemented and a query sequence is designed to perform binary search in parallel across multiple feature dimensions, generating parallel search results. Based on the parallel search results, the search results from different sorting directions are merged into a unified data characteristic description model to generate a comprehensive characteristic model; Based on the comprehensive characteristic model, the optimal encoding scheme is automatically selected for each column of data in the columnar data block set to generate the encoded columnar dataset.

5. The method according to claim 1, characterized in that, The process involves monitoring and recording data access, identifying data access patterns, calculating data value scores, and classifying data into hot, warm, and cold data based on these scores, storing them on different storage media to generate a tiered storage dataset, including: Based on the columnar storage structure, the column-level metadata is used to continuously monitor and record data access, and data access patterns and trends are identified through time series analysis and machine learning algorithms to generate a data access feature model. Based on the data access feature model, a data value score is calculated for each data partition and data block in the columnar storage structure. Based on the data value score, the data is divided into hot data, warm data, and cold data, and a data hierarchical configuration scheme is generated. According to the data tiering configuration scheme, a multi-level storage architecture is implemented, which stores hot data, warm data, and cold data in different storage media. Data blocks in the columnar storage structure are automatically moved between different storage media according to changes in data tiers to generate the tiered storage dataset. Based on the hierarchical storage dataset and the data access feature model, an intelligent retention strategy is implemented to automatically perform downsampling, aggregation, or cleaning operations on the cold data, generating an optimized dataset that has undergone intelligent lifecycle management.

6. The method according to claim 5, characterized in that, The process of identifying data access patterns and trends through time series analysis and machine learning algorithms, and generating a data access feature model, includes: Based on the recorded data access information, a network topology describing the inter-block association access in the columnar storage structure is constructed, and a data access network topology is generated. Based on the data access network topology, the data access behavior is parameterized into the Ising model, and an energy function is defined so that common access patterns in the data access network topology correspond to low-energy states, thereby generating a parameterized representation of the Ising model. Based on the parameterized representation of the Ising model, an efficient learning algorithm based on very few samples is implemented to learn model parameters from a limited number of access logs and generate the learned model parameters. Based on the learned model parameters, the access pattern prediction is optimized under hard constraints, and the future access probability prediction of data blocks in the columnar storage structure is output to generate the data access feature model.

7. The method according to claim 5, characterized in that, The process of calculating a data value score for each data partition and data block in the columnar storage structure, dividing the data into hot data, warm data, and cold data based on the data value score, and generating a data tiering configuration scheme includes: Based on the columnar storage structure, the interrelationships between the observation data recorded in the column-level metadata are modeled as a network structure, and a triangle counting algorithm with sublinear complexity is implemented to generate triangle structure detection results. Based on the triangular structure detection results, the local density and centrality of the data in the columnar storage structure are calculated, and the data with high density and high centrality in the triangular structure detection results are initially ranked in terms of value to generate a topological value assessment. Based on the topological value assessment, a sublinear time feature extraction algorithm is designed to extract key features from the observation data in the columnar storage structure. The key features and the topological value assessment are combined to construct a comprehensive data value model and generate a data value score. Based on the distribution of the data value scores, the optimal threshold for data stratification is dynamically determined through sublinear statistical estimation. The capacity allocation of each layer is optimized by comprehensively considering storage costs, access patterns in the data access feature model, and the distribution of the data value scores, thereby generating the data stratification configuration scheme.

8. The method according to claim 5, characterized in that, The process of analyzing user query behavior, identifying query pattern characteristics, developing indexing strategies for different data layers, implementing and maintaining various types of index structures, optimizing data layout, and generating optimized data storage structures includes: Based on the hierarchical storage dataset and historical query logs, in-depth analysis of user query behavior is performed to identify query pattern features. Clustering algorithms are used to group similar queries and generate a query pattern feature library. Based on the query pattern feature library and the hierarchical storage dataset, differentiated indexing strategies are formulated for different data layers, and dynamic indexing strategy schemes are generated. Based on the dynamic indexing strategy, multiple types of index structures are implemented and maintained, the performance of each index structure is automatically monitored and the index parameters are dynamically adjusted to generate optimized and diversified index structures. Based on the query pattern feature library and the optimized diversified index structure, a co-occurrence matrix of columns and fields in the query is constructed, and the co-occurrence frequency and relevance score between columns and fields are calculated. When the relevance score exceeds a preset association threshold, it is identified as a highly relevant data pair. The columns and fields in the highly relevant data pair are arranged in consecutive storage locations, the layout of time-series data in the hierarchical storage dataset is optimized, and the optimized data storage structure is generated.

9. The method according to claim 8, characterized in that, The process of performing in-depth analysis of user query behavior, identifying query pattern characteristics, and using clustering algorithms to group similar queries to generate a query pattern feature library includes: Based on historical query logs, a relational network describing the relationships between query operations is constructed, the temporal pattern of queries is identified, and the sequential dependencies between query operations are captured to generate a relational network of query operations. Based on the query operation association network, the query pattern is represented as an Ising model. An energy function reflecting the probability of the query pattern is defined so that common query patterns in the query operation association network correspond to low-energy state configurations, thereby generating the Ising model representation of the query pattern. Based on the Ising model representation of the query pattern, an efficient learning algorithm based on minimum sample size is implemented to extract key pattern features from a limited number of query logs and generate a learned query pattern model. Based on the learned query pattern model, a representative set of query patterns is generated under the premise of satisfying the constraints, realizing deep inference of query intent, predicting future query needs, optimizing the corresponding index and data layout strategies in the hierarchical storage dataset, and outputting the query pattern feature library containing intent annotations.

10. The method according to claim 1, characterized in that, Based on the unified observation data stream, the observation data undergoes cleaning and structure enhancement operations to transform logs into metrics, add contextual information to the observation data, and generate observation data with rich context, including: Based on the unified observation data stream, a configurable data processing pipeline framework is designed, and a configurable data pipeline definition is generated. Based on the configurable data pipeline definition, cleaning and structuring enhancement operations are performed on the observation data in the unified observation data stream to generate a cleaned structured data stream. Based on the cleaned structured data stream, the real-time conversion from logs to metrics is realized. Rules are defined to extract numerical metrics from the logs in the cleaned structured data stream and generate a log-derived metric dataset. Based on the cleaned structured data stream and the indicator dataset derived from the logs, data association and context enrichment are achieved, service topology information and metadata are added to the observation data, and the observation data with rich context is generated.

11. The method according to claim 10, characterized in that, The process of implementing data association and enriching context involves adding service topology information and metadata to the observation data to generate the observation data with rich context, including: Construct a network structure that describes the potential correlation between different types of observation data in the cleaned structured data stream and the log-derived indicator dataset. Initialize the correlation strength using the timestamp field and data source identifier in the unified observation data stream to generate a heterogeneous observation data correlation network. Based on the heterogeneous observation data association network, the association relationship between heterogeneous data is parameterized into an Ising model, prior constraints are introduced and modeled as hard constraints, and an Ising model representation of the association relationship is generated. Based on the Ising model representation of the aforementioned association relationship, a small-sample association pattern learning algorithm is implemented to learn association patterns from a small number of known association instances and generate association pattern learning results. Based on the learning results of the association patterns, association inference is performed under hard constraints. The probability that there is a real association between each pair of data entities in the heterogeneous observation data association network is calculated, and a consistent association network is generated. Based on the consistent correlation network, a causal graph of heterogeneous observation data is constructed. The causal graph is then used to automatically add contextual information to the observation data in the cleaned structured data stream and the log-derived indicator dataset, thereby generating the observation data with rich context.

12. The method according to claim 8, characterized in that, Based on the columnar storage structure, the optimized data storage structure, and the high-value observation data with rich context, a visualization and analysis platform is generated, including: Based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, a unified query engine supporting multiple query languages ​​is realized. The optimal execution path is selected according to the data characteristics in the column-level metadata and the optimized data storage structure to generate query execution capabilities. Based on the query execution capability, multi-level query optimization is achieved. The query plan is automatically rewritten based on the query conditions and the data statistics in the column-level metadata, realizing vectorized execution and parallel processing. The delayed materialization technology of the columnar storage structure is adopted to realize the intelligent caching mechanism to cache hot data in the hierarchical storage dataset, thereby generating a high-performance query processing system. Based on the high-performance query processing system, an adaptive query resource management mechanism is implemented, which dynamically allocates computing resources according to query complexity, the amount of data in the columnar storage structure, and system load, supports query truncation and progressive result return, and obtains a stable query service. Based on the stable query service, an integrated front-end visualization interface is provided, enabling data exploration tools to allow users to interactively analyze the observation data with rich context. An alarm configuration interface and an investigation and analysis toolchain are provided to generate the visualization analysis platform.

13. The method according to claim 12, characterized in that, The system achieves multi-level query optimization by automatically rewriting the query plan based on query conditions and data statistics in the column-level metadata, enabling vectorized execution and parallel processing. It employs the delayed materialization technique of the columnar storage structure to implement an intelligent caching mechanism for hot data in the hierarchical storage dataset, generating a high-performance query processing system, including: Before executing the full query, a sublinear algorithm is applied to estimate the size of the query results. By executing random queries on the optimized diversified index structure, an approximate value of the result set size is obtained. The query execution strategy is optimized based on the estimated result set size, and a query strategy optimization scheme is generated. Based on the query strategy optimization scheme, for analytical queries, an approximate query processing based on sublinear sampling is implemented. The sampling strategy is designed according to the query conditions and the statistical characteristics in the column-level metadata. When processing a very small proportion of data in the columnar storage structure, an approximate result is provided, and an approximate query result is generated. Based on the query strategy optimization scheme, an optimization algorithm for sub-linear complexity is implemented for aggregation operations. The data statistical characteristics in the column-level metadata and the optimized diversified index structure are used to avoid full table scans and generate aggregation optimization results. According to the query strategy optimization scheme, for complex queries involving the association of multiple observation data in the observation data with rich context, the sublinear graph algorithm is applied to accelerate the association query, realize the rapid identification of key paths and abnormal patterns in the causal graph, and generate accelerated association query results. Based on the query strategy optimization scheme, the approximate query results, the aggregation optimization results, and the correlation query acceleration results, an adaptive query execution strategy is implemented, and a progressive processing method is adopted for complex queries to generate the high-performance query processing system.

14. An observable system for unified log metrics and tracking data, characterized in that, include: The data acquisition and mapping module is used to acquire observation data from different sources, collect and parse the observation data, map the parsed observation data to a unified internal data model, establish correlation mappings between different types of data, and generate a unified observation data stream. This includes: receiving raw observation data through multiple interfaces based on different types of observation data to form an initial multi-source data stream; wherein the observation data includes log data, indicator data, and tracking data; automatically identifying data formats and extracting key fields and metadata based on the initial multi-source data stream; applying heuristic algorithms for automatic pattern recognition and field extraction of semi-structured logs to generate a structured intermediate dataset; mapping different types of observation data to a unified internal data model based on the structured intermediate dataset; the unified internal data model includes timestamp fields, data source identifiers, resource attributes, tag sets, indicator values, log content, and tracking information to generate observation data entities in a unified format; and automatically enhancing metadata and establishing correlation mappings between different types of data based on the unified format observation data entities to generate the unified observation data stream. The columnar storage module is used to partition and columnarly divide data according to the unified observation data stream, analyze the data characteristics of each column and automatically select the encoding method and compression algorithm, create and maintain column-level metadata, and generate a columnar storage structure. This includes: intelligently partitioning the data based on the unified observation data stream and the timestamp field and data source identifier in the unified internal data model, dividing the data into logical partitions and column-based blocks to form a columnar data block set; performing real-time analysis of the data characteristics of each column based on the columnar data block set and automatically selecting the encoding method to generate an encoded columnar dataset; dynamically selecting and applying compression algorithms based on the encoded columnar dataset for data with different characteristics and system resource conditions to generate compressed columnar data blocks; and creating and maintaining column-level metadata based on the compressed columnar data blocks, the column-level metadata including data type, encoding method, compression algorithm, and data block location, to generate the columnar storage structure. The hierarchical storage module is used to monitor and record data access according to the columnar storage structure, identify data access patterns, calculate data value scores, divide data into hot data, warm data and cold data according to the data value scores and store them in different storage media to generate hierarchical storage datasets. The index optimization module is used to analyze user query behavior based on the hierarchical storage dataset, identify query pattern characteristics, formulate indexing strategies for different data layers, implement and maintain multiple types of index structures, optimize data layout, and generate optimized data storage structures. The data augmentation module is used to perform cleaning and structured augmentation operations on the observation data based on the unified observation data stream, realize the conversion of logs to indicators, add contextual information to the observation data, and generate observation data with rich context. The visualization analysis module is used to implement a unified query engine based on the columnar storage structure, the optimized data storage structure, and the observation data with rich context, to perform query optimization and resource management, provide a visualization interface, and generate a visualization analysis platform.

Images