A semi-structured data processing method and system for GreptimeDB analytical observability database

Abstract

The application discloses a semi-structured data processing method and system for GreptimeDB analysis type observable database, comprising: a data access module for receiving semi-structured data records; a JSON analysis and intermediate representation module for analyzing the records to generate an intermediate representation form of data and identifying the current value type of each data; a path and type prompt module for maintaining explicit type declaration of the records by a user and taking the explicit type declaration as a path prompt; the explicit type declaration comprises path type constraints and null value constraints; a schema inference and evolution module for inferring a file level schema based on the intermediate representation form and performing merging and compatibility checking processing on the file level schema of multiple batches of data; and fundamentally reducing JSON read-time repeated analysis and run-time dynamic dispatch, greatly improving query processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to a semi-structured data processing method and system for the GreptimeDB analytical observable database. Background Technology

[0002] Observable data (such as logs, metrics, traces, events, etc.) typically has semi-structured characteristics: some fields are stably generated by the collector / SDK / agent program (e.g., service, instance, host, trace_id, etc.), while other fields come from user-defined attributes, dynamic tags, exception stacks, extended dimensions, etc., and have the characteristic that the structure and type change over time.

[0003] In analytical database scenarios, common queries include multidimensional filtering, grouping aggregation, drill-down analysis, TopN, and window calculations, which heavily rely on columnar storage and vectorized execution. Semi-structured processing that "completely preserves the original JSON string" incurs repetitive parsing costs during reads; while "complete splitting (shredding)" can easily lead to dimensionality explosion, metadata bloat, and excessive write and merge overhead. Summary of the Invention

[0004] The purpose of this invention is to provide a semi-structured data processing method and system for the GreptimeDB analytical observable database, in order to solve the problem of balancing performance and flexibility mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A semi-structured data processing system for the GreptimeDB analytical observable database includes: Data access module: used to receive semi-structured data records; JSON parsing and intermediate representation module: Parses the records to generate an intermediate representation of the data and identifies the current value type of each data item; Path and type hint module: used to maintain the explicit type declarations of the user for the record, and to use the explicit type declarations as path hints; the explicit type declarations include path type constraints and null value constraints; Schema inference and evolution module: Infers the file-level schema based on the intermediate representation and performs merging and compatibility verification on the file-level schemas of multiple batches of data; The classification and routing module is used to classify and determine the path suggestions, user-defined suggestions, historical metadata, and current value type, and to generate output category signals and additional processing signals through a classification and determination algorithm. Category processing module: used to perform differential processing based on the category signal, and finally generate standardized semi-structured data; the category processing module includes a columnar construction submodule, an evolution and inheritance submodule, a variant construction module and a dynamic key-value module; Immutable file generation / disk writing module: used to write the standardized semi-structured data into an immutable columnar file in column vector form, and simultaneously generate RowGroup level statistics and optional indexes; Meta-information management module: used to integrate the path hints, file-level schema, RowGroup-level statistics and optional indexes to build a unified schema query view; Query path rewriting module: used to rewrite the path type in the schema query view into a unified function format that the database engine can recognize; Query processing module: Used to receive user query requests and pass the rewritten path type information to the type inversion inference module; Type inversion inference module: Based on the rewritten path type and the path expression in the query statement, it infers the target reading type of the path required for the query and generates type hint information; Scanning, clipping, and materialization module: This module is used to perform clipping based on the target read type and type hint information, combined with the RowGroup level statistics and index, and only materializes the data required by the query. The results reconstruction / display module is used to reconstruct complex data types in the materialized data into JSON objects according to JSON semantics and output the final query results.

[0006] Preferably, the record contains JSON text, and the JSON text contains at least one JSON type field or JSON text field.

[0007] Preferably, the category signals include Class A, Class B, Class C, and Class D.

[0008] As a preferred embodiment, the differentiation processing steps of the category processing module are as follows: The A-class module is constructed by mapping the record fields to Struct sub-columns or atomic columns, and constructing column vectors and null value markers. The B category is defined by the evolved sub-module as a record file-level schema, field occurrence information, and supports union schema and missing completion during queries; The C class is represented by the variant building module using either Variant or Union for multiple current type values ​​of the same path.

[0009] Preferably, the category processing module further includes a D class, which is a Map created by the dynamic key-value module, and the long-tail dynamic key-values ​​are written into the Map key-value set column and the Key index digest.

[0010] Preferably, the processing system further includes a statistical feedback and consolidation module. The statistical feedback and consolidation module is configured with path promotion / demotion trigger indicators. The trigger indicators include path query popularity threshold, field missing rate threshold, type stability indicator, and path cardinality threshold. The module triggers dynamic conversion of paths between four categories (A / B / C / D) based on the indicators and achieves the conversion by generating a new file without modifying the original file and corresponding metadata.

[0011] Preferably, the dynamic conversion step is as follows: Promotion: When a certain path participates in filtering / aggregation frequently over a long period of time and its type tends to be stable, it can be promoted step by step from D→C→B / A to make it more columnar and user-friendly; Degradation: When a path changes from stable to long tail and its usage frequency decreases, it can be downgraded from A / B to C / D to reduce column and metadata overhead. The conversion is achieved by generating a new file through compaction, without modifying the old file.

[0012] A semi-structured data processing method for the GreptimeDB analytical observable database, applied to a semi-structured data processing system for the GreptimeDB analytical observable database, is characterized by the following steps: S101. Data Reception: Receive one or more semi-structured data records R, ensuring that the JSON text in each record contains at least one JSON type field or JSON text field, and complete the initial data access; S102. Reading Table Meta Information and Configuration: Read and confirm the JSON column set of the record, the user path type prompt including path type constraints and null value constraints, table options and runtime strategies, and retrieve historical meta information to provide basic configuration support for subsequent parsing and judgment; S103. Parse into intermediate representation: Parse the JSON text of each record to generate an intermediate representation containing the path and value, and identify the current value type of each data. S104. File-level schema inference and merging: Based on the intermediate representation generated in S103, infer the file-level schema for a single batch of data; for multiple batches of data, perform a merging operation on the file-level schemas of each batch and complete compatibility verification to ensure schema consistency. S105. Path Traversal and Classification: Combining the intermediate representation of the path and value generated in S103, the user path type prompt read in S102, historical metadata, and the current value type identified in S103, path traversal and classification are performed through a classification algorithm, and the output classification signal is divided into class A, class B, class C, class D and additional processing signal; S106. Construct columnar representations by category: Perform differential processing based on the category signals output in S105, and construct corresponding columnar representations: Category A path, construct a Struct sub-column with a defined data type; Category B path, record file-level schema information and the frequency and location of occurrence of corresponding fields; Category C path, construct a Variant type column; Category D path, construct a Map key-value set and Key index summary; S107. Generate an immutable file and write it to disk: Write the columnar data processed in S106 into an immutable columnar file in column vector form, synchronously generate RowGroup level statistics, and optionally generate indexes as needed; S108. Update Manifest / Meta metadata: Based on the immutable columnar physical file generated in S107, extract and integrate its file identifier, file-level schema, RowGroup-level statistical information, optional index summary, and extended metadata. The extended metadata includes Variant type summary, field occurrence information, and Key index summary to form a unified file representation record and persist it. Construct a unified schema query view to provide metadata support for subsequent queries. S109. Query Planning and Execution: After receiving the user's query statement, the query path is rewritten and transformed into a unified function format that the database engine can recognize; combined with the path expression context, type inversion inference is performed to generate the read schema, loading strategy and predicate pruning strategy; data pruning is performed according to the above strategies to materialize only the data required by the query; finally, the materialized data is processed through vectorization operators to output the query results. S110. Compaction Phase: Based on the statistical information generated in S107 and the query feedback in S109, promotion / demotion operations are performed on various paths in the compaction phase, and new immutable columnar files are reprocessed and generated; the Manifest / Meta metadata is updated synchronously.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for processing semi-structured data in an analytical observable database.

[0014] An electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for processing semi-structured data in the GreptimeDB analytical observable database. The electronic device includes one or more of a server, a cloud server, and a distributed computing cluster, and the memory is a non-volatile computer-readable storage medium.

[0015] The beneficial effects of this invention are: 1. In step S105, this invention completes the path classification and discrimination of A / B / C / D classes. In step S106, it constructs a structured columnar representation for A / B classes, a Variant type column for C classes, and generates a Key index digest for D classes. Then, through type inversion inference, scan pruning, and materialization in step S109, it extracts only the data required for the query. Combining RowGroup-level statistical information and indexes, it achieves A / B columnar statistical pruning, C class Variant + type inversion, and D class Key pruning, fundamentally reducing repeated parsing during JSON reads and dynamic allocation at runtime, and significantly improving query processing efficiency.

[0016] 2. In this invention, S105 routes highly dynamic long-tail key-value pairs to class D, and S106 uses the class D dynamic key-value module to converge them to the Map key-value set column and Key index digest. At the same time, class C uses Variant to uniformly represent multiple types of values ​​for the same path, and the statistical feedback and merging module limits the expansion of Variant / path through threshold indicators, avoiding the surge in the number of columns and schema metadata caused by full splitting, and effectively controlling the scale of metadata.

[0017] 3. In this invention, S104 supports multi-batch file-level schema merging and compatibility verification, allowing for inconsistencies in schemas across different files; in S106, the B-type evolution submodule implements union schema and missing data completion during queries, and then completes schema consolidation and merging with controllable overhead through the compaction stage in S110, without requiring mandatory schema unification during the data access stage, achieving schema evolution friendliness and significantly reducing the schema management cost of multi-batch dynamic data.

[0018] 4. In S106 of this invention, the A-type defined columns and the C-type Variant sub-representations are stored in a structured columnar format, which can be adapted to efficient encoding such as dictionary, RLE, and delta and combined with compression; the D-type dynamic key-value pairs are encapsulated into a small number of columns, reducing the overhead of column metadata maintenance and index construction, and optimizing compression and storage costs from both the storage structure and encoding method dimensions.

[0019] 5. In step S107 of this invention, standardized semi-structured data is written into an immutable columnar file in the form of column vectors. It is preferred to write the data to disk based on the standard type and extensibility of Parquet, without relying on non-standard technology for extension. At the same time, a unified schema query view is built through the metadata management module, and the query path rewriting module converts the path type into a unified function format that the database engine can recognize, ensuring seamless integration with various tools and engines in the data lake ecosystem, and significantly enhancing interoperability and system scalability. Attached Figure Description

[0020] Figure 1 This is a diagram showing the overall module structure and connection relationships of an embodiment of the present invention; Figure 2 This is a flowchart illustrating the overall process of an embodiment of the present invention. Figure 3 This is a schematic diagram of the physical layout of a Class A Struct column according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the evolution of the B-class file-level schema and the read-time union schema according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the physical layout of a Class C Variant according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a D-class Map representation and RowGroup Key index according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the promotion / demotion state machine and triggering indicators according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the query execution pipeline in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1 Please see Figure 1 Figure 8 This invention provides a semi-structured data processing system for the GreptimeDB analytical observable database, comprising the following functional modules (each module can be deployed in the same process or different services): (1) Data access module: Receives observable data records. Records can be entered into the system through SQLINSERT, batch import (such as COPY / NDJSON), message queue or file disk.

[0023] (2) JSON parsing and intermediate representation module: Parses JSON text into an intermediate representation (e.g., tree structure of objects / arrays / scalars), and preferably adopts the method of "parsing once and reusing the parsing result" to simultaneously support schema inference and column vector construction.

[0024] (3) Path and type hint module: Maintain the explicit type declaration of the JSON path by the user (such as the path type and null value constraint given when CREATETABLE), and use the hint as the basis for the identification and writing of deterministic fields.

[0025] (4) Schema inference and evolution module: Infer file-level schema based on parsed records, and perform schema merging and compatibility in multi-batch / multi-file scenarios.

[0026] (5) Classification and routing module: Based on path prompts, historical schema, current value type, etc., output category signals and route to the physical write links of A / B / C / D.

[0027] (6) Class A column construction module: Maps the defined fields to Struct sub-columns / atomic columns, and constructs column vectors and null value markers.

[0028] (7) Type B evolution and succession module: records file-level schema and field occurrence information, and supports union schema and missing completion during query.

[0029] (8) C-type variant building blocks: Variant / Union is used to represent multiple types of values ​​for the same path; in one implementation, the ParquetVariant extended type (which can be physically represented by metadata / value binary fields) or the equivalent implementation of "type_tag + multiple sub-columns" can be used.

[0030] (9) D-type dynamic key-value module (optional): Writes long-tail dynamic key-values ​​into Map(Path→Value) or equivalent key-value set columns, and maintains a local index digest of the Key to support path existence pruning.

[0031] (10) Immutable file generation / disk writing module: Writes the processing results to an immutable columnar file (preferably Apache Parquet) and generates RowGroup statistics (min / max / null_count, etc.) and optional indexes (e.g. Bloom Filter).

[0032] (11) Meta information management module (Manifest / Meta): Maintains table-level metadata (user path hints, table options, file list, file-level schema, etc.) and provides query views for merging schemas.

[0033] (12) Query path rewriting module: rewrites the access to JSON path in SQL (e.g., data.ab) into a call to a unified function (e.g., get_field_typed(data,'a.b',type_hint)), so that planning does not depend on whether the path exists in the schema beforehand.

[0034] (13) Type inversion inference module: Combining user path hints and expression context (comparison / arithmetic / aggregation / string matching, etc.), it automatically infers the target reading type of the path and generates type hints to drive the scanning schema and predicate pushdown.

[0035] (14) Scanning, pruning and materialization module: Based on the inference results, only the required Struct sub-column / Variant sub-column / Mapkey is loaded, and pruning is performed based on file-level / RowGroup-level statistics and indexes; during the loading phase, the results are materialized into column blocks of a certain structure and a certain type.

[0036] (15) Result reconstruction / display module: Reconstruct complex types such as Struct / Variant into JSON strings according to JSON semantics and output them (e.g. json_display).

[0037] (16) Statistical feedback and consolidation module (optional): Collects statistical information such as query popularity, missing rate, and type set, and realizes path promotion / demotion and generates new files during the compaction / consolidation stage.

[0038] The data and signal connection relationships between the modules are as follows: the data access module outputs record R → the parsing module outputs the intermediate representation of the path value → the discrimination and routing module outputs the category signal and distributes it to A / B / C / D to write to the link → the file generation module writes it to disk and updates the Manifest / Meta; during the query, the path rewriting and type inversion module generates and reads the schema and type hints → the scanning and pruning module selectively loads and prunes → the vectorized execution engine executes the query and outputs the results.

[0039] This invention provides a method for processing semi-structured data in an analytically observable database, comprising the following: S101: Data Reception. Receive one or more semi-structured data records R, each containing at least one JSON type field or JSON text field.

[0040] S102: Read table metadata and configuration. Determine the target table's JSON column set, user path type hints, table options (compression, RowGroup size, Bloom Filter, etc.), and runtime strategies (whether type conversion is allowed, whether multiple types are allowed, etc.).

[0041] S103: Parse into intermediate representation. Parse the JSON text of each record to obtain an iterable intermediate representation; preferably, reuse the same parsing result for subsequent schema inference and column vector construction to avoid repeated parsing.

[0042] S104: Path Traversal and Discrimination. For the path and value in the intermediate representation, perform classification and discrimination based on user prompts and historical metadata, and output class_signal(Path)∈{A,B,C,D} and additional signals (need_cast, type_tag, etc.).

[0043] S105: Construct columnar representations by category. Perform the following actions based on the category signals: Category A: Construct a Struct sub-column of a defined type; Category B: Record file schema and field occurrences; Category C: Construct a Variant; Category D (optional): Construct a Map key-value set and Key index summary.

[0044] S106: Generate an immutable file and write it to disk. Organize column vectors into RowGroups / blocks and write them to the columnar file; generate statistics (min / max / null_count) and optionally generate indexes such as Bloom filters during writing.

[0045] S107: Update Manifest / Meta. Record file identifiers, file-level schemas, and extended meta-information related to JSON columns (e.g., Variant type collection summary, field occurrence information, index summary, etc.).

[0046] S108: Query Planning and Execution. After receiving the query statement, the JSON path access is rewritten; type inversion inference is performed based on the expression context to generate the reading schema, selective loading strategy, and predicate pruning strategy; after pruning and materialization are completed in the scanning phase, vectorized operators are executed.

[0047] S109: Compacting and Merging (Optional). Based on statistics and query feedback, the compaction phase performs promotion / demotion of paths and generates new files, while updating metadata.

[0048] The classification algorithm based on the above embodiments is as follows (reinforcement: module input / output + rules + threshold / anomaly): (1) Input: a) Table metadata: JSON column identifiers, user-declared path type hints (including nullability), and optional partition / index configurations; b) Configuration items: Whether type conversion is allowed, conflict handling strategy, threshold parameters (stable window W, missing rate threshold T_missing, maximum number of variant types T_type_max, maximum number of enumerable paths T_path_max, etc., can all be configured); c) Current record parsing results: Path, Value, and its actual type (Type_actual); d) Historical metadata: schema from historical files, path occurrence rate, historical type set (Path), etc.

[0049] (2) Output: a) class_signal(Path): Path category A / B / C / D; b) reason_code: Determines the reason (e.g., "user declaration", "high structural missing rate", "type conflict", "path limit exceeded"). c) need_cast: Whether type conversion is required according to configuration; d) type_tag: Records the actual type tag (or equivalent Variant encoding information) when entering C / D.

[0050] (3) A preferred selection rule is as follows (which can be implemented as a flowchart or pseudocode): Rule 1: Paths explicitly declared by the user are processed first, according to the specified columns. If Path is declared in the user prompt: If Type_actual is consistent with Type_declared, it is classified as class A. Type_actual and Type_declared are inconsistent: If compatible conversions (such as int→double, string→timestamp, etc.) are allowed and the configuration allows it, then after the conversion, it will be judged as class A; otherwise, write will be rejected / error will be reported according to the configuration, or it will be downgraded to class C and an alarm will be recorded (configurable).

[0051] Rule 2: Undeclared paths are determined based on historical and current observations. If the path is not declared in the user prompt: If the path appears for the first time: in an implementation, it can be first considered as a candidate for class B and included in the file schema; or it can be directly classified as class D when the number of paths is high (configurable). If the frequency of Path in the most recent W file windows is lower than the threshold (high missing rate): it is classified as Class B (gradual structural evolution / sparse field). If the size of Types(Path) is found to be greater than 1, or the current type is inconsistent with the historical main type: it is judged as Class C (dynamic type), and the logical type of the Path is fixed to Variant / Union to avoid repeated schema changes; If the system detects that the total number of paths or the number of Variant paths exceeds the upper limit (T_path_max, T_C_max, etc.): subsequent newly added dynamic paths can enter class D to control the growth of metadata (configurable).

[0052] Rule 3: Conflict and Exception Handling. For cases of parsing failure, non-conversion, or exceeding constraints, possible strategies include: refusing to write, setting the value to null and issuing an alert, or downgrading to a more lenient category (C / D).

[0053] Among them, Category A: Columnar storage (Struct / column family) with fully defined parts (reinforcement: write steps + how queries filter / aggregate / sort / index) For deterministic parts of semi-structured data (such as service, instance, resource attributes, etc.), they are stored as nested structure columns (Struct columns) or equivalent column families, with their subfields physically corresponding to independent atomic columns.

[0054] Specifically, the detailed processing procedure for Category A is as follows: In one embodiment, the Class A processing module performs the following steps: A201: Determine column mapping. Based on user path suggestions or stability assessment results, map the A-class Path to the target Struct sub-column / atomic column.

[0055] A202: Type Validation and Conversion (Optional). Validate Type_actual based on Type_declared; if conversion is allowed by the configuration, perform the conversion; otherwise, refuse to write / report an error or set it to null according to the policy and log an alarm.

[0056] A203: Column vector buffer and null flag. Appended to the corresponding column vector in units of RowGroup / block, and nullbitmap / nesting definition level information is maintained.

[0057] A204: Encoding and Compression (Preferred). Atomic columns are encoded using dictionary encoding, RLE, delta+bitpacking, etc., and compressed using Snappy / ZSTD, etc. (Encoding / compression methods can be configured via table options).

[0058] A205: Statistics and Pruned Index Construction. Calculates statistics such as min / max / null_count for columns / RowGroups; optionally, it constructs index summaries such as Bloom filters for specified columns.

[0059] A206: Write to disk and update metadata. Write the struct sub-column to an immutable columnar file (Parquet preferred), and update the file-level schema and statistics / index summary to Manifest / Meta.

[0060] A diagram illustrating the physical layout of a Struct column (the Struct expands into multiple sub-columns and carries RowGroup-level statistical information).

[0061] The usage of type A in filtering / aggregation / sorting / partitioned indexes.

[0062] Since column A is a defined atomic column, it can directly enter the vectorized operator chain of the analytical execution engine, specifically including: Filtering: Preferred pruning is based on RowGroup statistics (zonemap); further, finer-grained pruning can be performed based on page-level statistics (PageIndex); after entering the execution engine, vectorized comparison operations are performed on the column vectors to generate selectionvector.

[0063] Aggregation: Perform hash aggregation or sort aggregation directly on groupkey of type A; determining the type can reduce dynamic allocation at runtime.

[0064] Sort / TopN: Perform vectorized sorting or TopN selection on class A sort keys; can be combined with partition / sorting information to reduce the sorting range.

[0065] Partition pruning: If column A is set as the partition key, the query predicate can be pruned at the file level to reduce the number of files scanned.

[0066] Index (optional): In addition to statistics, you can optionally build dictionary indexes, Bloom filters, bitmap indexes, etc., to accelerate equality / range filtering.

[0067] Specifically, the detailed processing procedure and explanation for Category B are as follows: Category B: Compatibility Inference and Reorganization for Dynamic Structures (File-Level Schema Inconsistency) (Reinforcement: File Schema, Missing Field Records) On the write side, immutable data files are typically used: once the file is created, it is not modified. Structural changes (adding fields, missing fields) occur gradually over time, resulting in different files having different physical schemas.

[0068] Type B write side: File-level schema version and field occurrence records In one embodiment, when the B-class processing module writes to each immutable file F_k: Generate the schema_k of this file (which contains the set of Struct subfields that actually appear in the file); Record field occurrence information (e.g., field list summary or bitmap, optional); Write the version number / hash of schema_k to the file metadata and update Manifest / Meta.

[0069] Type B query side: Read-time compatible inference and missing completion When the query scans multiple files F_i…F_j: The Planner generates a unified read schema_read = UNION(schema_i…schema_j); For missing fields: Generate a vector of all null columns / null markers in files that do not contain the field; For newly added fields: they are treated as missing in the old file, but read as actual columns in the new file; If a field with the same name but a type conflict occurs: it will be converted to a C-class Variant for reading according to the rules, or an explicit cast will be required (configurable).

[0070] A schematic diagram of the dynamic processing flow of the structure (different schemas for different files; when querying, use unionschema and fill in null).

[0071] Specifically, the detailed processing procedure and explanation for Category C are as follows: Class C: Type-based dynamic (Variant / Union) clustered storage and access (Supplement: Write steps + query upcast rules) When different atomic types appear in the same path / field (such as a mixture of numeric and string), this scheme uses variant columns (Variant / Union) for storage to avoid frequent schema changes or write-time recoding.

[0072] Class C write side: Variant Build Steps For a logical path Path_p, in one embodiment it includes: C401: Type identification. Identify the actual type t (int / float / string / bool / object / array...) for Value.

[0073] C402: Write Variant representation. Writes the value as a Variant logical type; in an optional implementation, an equivalent layout of "type_tag + multiple type sub-columns" can be used.

[0074] C403: Record type set. Types (Path_p) and conflict information are written to Manifest / Meta for subsequent queries and evolution decisions.

[0075] C404: Type number control (configurable). When the size of Types(Path_p) exceeds T_type_max, an alarm, write rejection, or overflow to D (configurable) can be triggered.

[0076] Variant physical layout diagram (can be an equivalent implementation of ParquetVariant or type_tag+sub-column).

[0077] C-type query side: Type inversion-driven selective loading and upcast The expression constraints on Path_p during the query planning phase determine the reading strategy: If the expression is a numeric comparison / aggregation (>, <, SUM, AVG, etc.): the numeric representation is read first, and upcast to a uniform numeric type (such as double) as needed. If the expression is a string match (LIKE, REGEXP, etc.): Read the relevant string representation; If the expression context is unclear: handle it according to the configurable strategy (convert to string uniformly, require explicit cast, set to null and issue an alarm if it cannot be converted, etc.).

[0078] This strategy allows type dynamics to be resolved during the loading phase, and then determined as a block of type columns after entering the execution engine.

[0079] Specifically, the detailed processing procedure and explanation for Category D are as follows: Class D: Path→ValueMap storage for the fully dynamic part and path predicate acceleration (enhanced by: Key / Value encoding + Key index construction / usage) For uncontrollable dynamic inputs (large number of paths, unstable structure), splitting them into a large number of columns would lead to an explosion of metadata. As a preferred embodiment, this solution represents this part as a Map column (or an equivalent key-value set column): Key: Path (preferably JSONPath or dotted path); can be dictionary-encoded as path_id within the RowGroup / file; Value: The corresponding value, which can be a Variant value or "binary / string representation + type_tag".

[0080] Note: In the PoC implementation, "Struct evolution + Variant" can also be used to handle dynamism; Map can be used as an optional enhancement to further control the scale of metadata.

[0081] Class D Write Side: Map Representation and Encoding In one embodiment: Organize the dynamic part of each record into [(path_id1,value1),(path_id2,value2),...]; The path_id is dictionary-encoded within the RowGroup / file, and a dictionary digest is written into the metadata. The value is represented by a Variant or a unified value with a type_tag; Write the collection into a single Map column (or two columns, keys and values), encapsulating a large number of dynamic keys in "one column / a few columns".

[0082] Type D query side: Path predicate acceleration (Key index) To accelerate predicates such as "Does path P exist?" and "Does the value of path P satisfy the condition?", in one embodiment, a RowGroup-level Key index is constructed: During writing: Build one of the following for the keys of each RowGroup: Bloom Filter / Bitmap / Dictionary digest; During the query: First, use the Key index to determine whether the RowGroup may contain Path=P; if not, skip it; if so, only decode the key-value pairs related to P and materialize them into columns of a specific type.

[0083] MapKey local index illustration (RowGroup first checks existence and then decodes values).

[0084] Dynamic transitions (promotion / demotion) between the four categories and query feedback-driven compaction (optional) Since the data files are immutable and generated on a rolling time basis, this solution supports adjusting the path-level storage strategy based on statistical and query feedback during the compaction / organization phase: Promotion: When a certain path participates in filtering / aggregation frequently over a long period and its type tends to be stable, it can be promoted by... The gradual promotion process transformed it into a more formal and friendly expression. Degradation: When a path changes from stable to long-tailed and its usage frequency decreases, it can be downgraded. Degradation reduces column and metadata overhead; The conversion is achieved by generating a new file through compaction, without modifying the old file.

[0085] Promotion / demotion state machine diagram ( ) and trigger indicators.

[0086] Computation layer: Schema metadata from the planning phase + type inversion inference to eliminate runtime dynamism (complete SQL examples and inference process). This scheme not only defines the persistent storage representation, but also the computation layer read strategy: During the query planning phase, a read schema and physical scan strategy are generated based on the Manifest / Meta (file schema, column statistics, Variant type collection summary, optional Mapkey index summary, etc.); Perform type inversion inference for each JSON path involved in the query: statically determine the read type and necessary conversions based on the projection expression, filtering predicate, and aggregation operator input requirements; The selective loading and materialization of Struct / Variant / Map are completed during the data loading phase, ensuring that the data blocks entering the execution engine have a defined structure and a defined type column.

[0087] (i) Extract the path reference set Paths(Q) from the SQL / expression; (ii) For each path p, read the candidate physical source (Struct sub-column, Variant, or Mapkey) from Meta. (iii) Determine the target reading type based on the operator requirements (e.g., comparison operations require numerical values, LIKE requires strings, and time functions require timestamps). (iv) Select the minimum read set and generate type hints: A / B directly read the corresponding atomic columns; C select only the representation that meets the requirements and upcast; D (optional) select only the relevant keys and materialize them into columns; (v) Generate scan plan: predicate pushdown, RowGroup / Page pruning, optional BloomFilter / Key index pruning; (vi) Enter the execution engine to execute the vectorization operator.

[0088] [Example: Implementing type-specific calculations by combining user path suggestions] CREATE TABLE typed (id INT, data JSONFUSION (age INT NOT NULL, activeBOOLEAN)); SELECT data.age + 1 AS age_plus_one, CASE WHEN data.active THEN 1ELSE 0 END AS active_flag FROM typed; Note: The system reads data.age as a specific integer column based on user prompts and completes the type determination during the planning / loading phase, thereby avoiding repeated JSON parsing and dynamic dispatching at runtime.

[0089] [Example: Type inversion triggered by the expression context when there is no explicit hint] SELECTCOUNT( )FROMlogsWHEREdata.http.status_code>=500; Note: The comparison operation ">=" requires numerical input. During the planning stage, the system can infer the reading type of the path http.status_code as integer / floating point and generate the corresponding type hint. During the scanning stage, RowGroup / Page is clipped based on Parquet statistics and then materialized into columns of a specific type for filtering.

[0090] Query execution pipeline (Planner → Type inversion → Trimming / materialization → Vectorized execution).

[0091] Interoperability: Based on the standard columnar file format (Parquet preferred), without any modifications outside the standard. This solution preferentially utilizes Apache Parquet's Struct / Map / (Variant or equivalent Union implementation) capabilities for disk persistence, and does not require non-standard extensions to the upstream standard. The generated files can be read in a usable manner on other systems that support the corresponding types (although additional metadata and some flexible read / write capabilities of this system may be lost). For example: A / B type fields can be directly read as nested Struct columns; D type Map columns can be read as key-value pairs; C type Variant, if not supported by the external system, can be degenerated into a composite method of a type_tag column + binary / string value column to simulate it.

[0092] The specific operation process based on the above embodiment one is as follows: The following is an example illustrating one implementation of the Proof-of-Concept (PoC) of this project. This example is for illustrative purposes only and is not intended to limit the scope of protection.

[0093] The first approach: On the write side, column validation and columnarization are determined based on path type hints. (1) Table creation example: CREATE TABLE typed (id INT, data JSONFUSION (age INT NOTNULL, active BOOLEAN)); (2) Example of writing: INSERT INTO typed VALUES (1, '{"age":21,"active":true}'); (3) Example of an exception: When the age in the record is a string (such as "age":"nope"), the system refuses to write and reports a type mismatch.

[0094] (4) Beneficial effect: The arithmetic expression for data.age (data.age + 1) can determine the input type during the planning stage and directly materialize it into an integer column during the loading stage, avoiding read-time parsing and dynamic dispatch.

[0095] The second approach: File-level schema evolution and read-time merging in dynamic structural scenarios. (1) Table creation example: CREATE TABLE merge_demo (id INT, data JSONFUSION); (2) Write to different fields respectively: INSERT INTO merge_demo VALUES (1, '{"a":1}'); INSERT INTO merge_demo VALUES (2, '{"b":"two"}'); (3) Query example: SELECT data.a AS a, data.b AS b FROM merge_demo ORDER BY id; (4) Explanation: The system merges file-level schemas in Manifest / Meta to generate a read schema; it fills in null for missing fields so that the same query can return a unified column view across files.

[0096] The third approach: Variant representation and JSON semantic access in dynamically typed scenarios. (1) Example of writing: The value of the same path has multiple types such as int / string / object / array; the system fixes the logical type of the path as Variant.

[0097] (2) Query example: SELECT id, data.value AS value FROM variant_demo ORDER BY id; The returned value maintains JSON semantics (e.g., 1, "two", {"nested":3}, [1,2,3]).

[0098] (3) Example of typed reading: SELECT get_field_typed(data, 'value', CAST(NULL ASDOUBLE)) FROM variant_demo WHERE get_field_typed(data, 'value', CAST(NULL ASDOUBLE))>10; (4) Explanation: The system infers that value must be a number based on the expression context. During the loading phase, the value is extracted from Variant and upcast to double; non-convertible values ​​are set to null or rejected according to the strategy.

[0099] Other technical information that helps in understanding this invention Configuration items and default policies (allow_cast, W, T_missing, T_type_max, T_path_max, etc.).

[0100] Examples of metadata (Manifest / Meta) field definitions (file list, file-level schema, path type set summary, field occurrence information, index summary, etc.).

[0101] 8. Product performance test data (testing methods that demonstrate the product's superior performance, such as enterprise standard number, national standard number, and test results, etc.) (It is recommended to provide: test methods, metrics, baseline comparison, and result attribution. An example is provided below; specific data can be substituted.) Test method (example): Select 1 million rows of observable event JSON data as the dataset, and write it using "only the original JSON string (comparative example)" and "the four-level mapping of the present invention (example)" respectively; compare storage usage, cold start query time and preheating query time.

[0102] Metrics (examples): Data file size after import; cold run and hot run latency for typical filtering / aggregation queries in Q1~Q5; CPU usage and number of bytes read.

[0103] Test results (example): Test environment and dataset (example): AWS EC2 c6a.4xlarge; dataset consists of 1,000,000 rows of observable events in JSON format; comparison systems are Elasticsearch (Comparison 1) and PostgreSQL (Comparison 2). Test items include storage usage after import, cold run query time, and hot run query time.

[0104] Table 1: Typical query latency comparison (unit: ms; sample data, can be replaced with actual test results) Note: Q1~Q5 are typical JSON path filtering / aggregation query sets (such as range filtering and grouping aggregation of stable paths, existence / equality filtering of dynamic keys, and comparison / aggregation of dynamic fields of type).

[0105] Table 2: Comparison of storage space after import (sample data, can be replaced with actual test results) Attribution of Results (Example): The advantages of the embodiments of the present invention over the comparative examples in terms of storage usage, cold start, and query performance after warm-up can be attributed to the following mechanisms (which may be selected and adopted according to specific implementations): (1) Write side "parse once and reuse parsing results": each line of JSON text is parsed into an intermediate representation and reused for schema inference and column vector construction, avoiding repeated parsing and allocation of the same JSON in the write and query stages, thereby reducing CPU and memory overhead.

[0106] (2) Structured disk replacement of original JSON string: Map stable / high frequency paths to Struct sub-columns / atomic columns and write them to standard column files so that filtering / aggregation / sorting can be performed directly on columns of a certain type, instead of repeatedly performing JSONPath / field access parsing on JSON strings.

[0107] (3) File-level schema evolution + read-time merging: Each immutable file carries its actual schema, and the metadata maintains the merged schema; during querying, it reads according to the union schema and fills in missing fields as a null column vector, avoiding expensive full rewrites on the write side to adapt to the structure evolution.

[0108] (4) Path type hints and constraints reduce dynamism: When users create tables, they are given hints on the type and nullability of the path. The write side can perform strong validation / write rejection or conversion according to the strategy, so that common paths can enter the determined type link more quickly and reduce the type assignment and fault tolerance branches during query runtime.

[0109] (5) Type inversion in the planning stage + type hint propagation: The system infers the target type of the path from the expression context (comparison / arithmetic / aggregation / string matching, etc.) and passes the type as a hint to the scanning / loading stage, so that the loading stage can complete the cast / upcast and fault tolerance strategy, and the data block entering the execution engine maintains "determined structure + determined type".

[0110] (6) Column-level / leaf-level projection reduces I / O: Deep path access can be normalized to leaf path and mapped to Parquet leaf columns, scanning only necessary columns / sub-columns, reducing the number of bytes read and the workload of decoding, especially in wide table / multi-nested scenarios where the benefits are obvious.

[0111] (7) RowGroup / page-level statistical pruning reduces scanning: For columns of a certain type, it is preferable to use ParquetRowGroup statistics (min / max / null_count) for zone map pruning; when supported, PageIndex is further used for page-level pruning, thereby reducing the number of scanned rows and cold start IO.

[0112] (8) Optional indexing and encoding / compression bring additional benefits: Bloom filter index summaries can be generated for high-frequency equal value filtering columns; at the same time, dictionary encoding, RLE, delta+bitpacking and other column encodings can be configured and combined with ZSTD / Snappy compression to improve the compression ratio and reduce the reading volume in scenarios with duplicate values / low cardinality.

[0113] (9) The source of performance improvement after warm-up: can be attributed to scan-side caching (e.g., caching materialized column blocks / batches by leaf projection and predicate dimension) and Parquet metadata caching (schema / page index / statistics), reducing the overhead of repeated metadata parsing and cold start initialization.

[0114] (10) Selective extraction of dynamic type paths: After using Variant / Union to represent type conflict paths, when the query context is clear, only the target type can be extracted and upcast (e.g., only the numerical representation is extracted for numerical comparison), avoiding conversion or JSON serialization of non-target types, thereby reducing computational overhead.

[0115] Example 2 The difference from the above embodiment one is that File format alternatives: In addition to Parquet, ORC / Arrow IPC / Lance / custom columnar segment files can be used, as long as nested columns, Map, or equivalent representations are supported.

[0116] Example 3 The difference from the above embodiment one is that Variant alternative implementation: If the underlying system does not support the Variant logical type, a combined multi-column implementation using "type_tag + multi-value column (by type)" can be used.

[0117] Example 4 The difference from the above embodiment one is that Map / Key index alternatives: any structure that can quickly determine the existence of a path, such as Bloom Filter, bitmap, inverted dictionary, or dictionary digest.

[0118] Example 5 The difference from the above embodiment one is that Promotion / demotion strategy alternatives: can be based on rule thresholds, cost models, or machine learning popularity predictions; or can be specified by the user.

[0119] Example 6 The difference from the above embodiment one is that Type inversion substitution: can be completed in the logical plan, physical plan, or scan operator initialization phase; it can be combined with column statistics for more aggressive pruning.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A semi-structured data processing system for the GreptimeDB analytical observable database, characterized in that: include: Data access module: used to receive semi-structured data records; JSON parsing and intermediate representation module: Parses the records to generate an intermediate representation of the data and identifies the current value type of each data item; Path and type hint module: used to maintain the explicit type declarations of the user for the record, and to use the explicit type declarations as path hints; the explicit type declarations include path type constraints and null value constraints; Schema inference and evolution module: Infers the file-level schema based on the intermediate representation and performs merging and compatibility verification on the file-level schemas of multiple batches of data; The classification and routing module is used to classify and determine the path suggestions, user-defined suggestions, historical metadata, and current value type, and to generate output category signals and additional processing signals through a classification and determination algorithm. Category processing module: used to perform differential processing based on the category signal, and finally generate standardized semi-structured data; the category processing module includes a columnar construction submodule, an evolution and inheritance submodule, a variant construction module and a dynamic key-value module; Immutable file generation / disk writing module: used to write the standardized semi-structured data into an immutable columnar file in column vector form, and simultaneously generate RowGroup level statistics and optional indexes; Meta-information management module: used to integrate the path hints, file-level schema, RowGroup-level statistics and optional indexes to build a unified schema query view; Query path rewriting module: used to rewrite the path type in the schema query view into a unified function format that the database engine can recognize; Query processing module: Used to receive user query requests and pass the rewritten path type information to the type inversion inference module; Type inversion inference module: Based on the rewritten path type and the path expression in the query statement, it infers the target reading type of the path required for the query and generates type hint information; Scanning, clipping, and materialization module: This module is used to perform clipping based on the target read type and type hint information, combined with the RowGroup level statistics and index, and only materializes the data required by the query. The results reconstruction / display module is used to reconstruct complex data types in the materialized data into JSON objects according to JSON semantics and output the final query results.

2. The semi-structured data processing system for the GreptimeDB analytical observable database according to claim 1, characterized in that: The record contains JSON text, and the JSON text contains at least one JSON type field or JSON text field.

3. The semi-structured data processing system for the GreptimeDB analytical observable database according to claim 1, characterized in that: The signal categories include Class A, Class B, Class C, and Class D.

4. A semi-structured data processing system for the GreptimeDB analytical observable database according to claim 1, characterized in that: The specific differentiation processing steps of the category processing module are as follows: The A-class module is constructed by mapping the record fields to Struct sub-columns or atomic columns, and constructing column vectors and null value markers. The B category is defined by the evolved sub-module as a record file-level schema, field occurrence information, and supports union schema and missing completion during queries; The C class is represented by the variant building module using either Variant or Union for multiple current type values ​​of the same path.

5. A semi-structured data processing system for the GreptimeDB analytical observable database according to claim 1, characterized in that: The category processing module also includes a D class, which is a Map created by the dynamic key-value module, and the long-tail dynamic key-values ​​are written into the Map key-value set column and the Key index digest.

6. A semi-structured data processing system for the GreptimeDB analytical observable database according to claim 1, characterized in that: The processing system also includes a statistical feedback and consolidation module. The statistical feedback and consolidation module is configured with path promotion / demotion trigger indicators. The trigger indicators include path query popularity threshold, field missing rate threshold, type stability indicator, and path cardinality threshold. The module triggers the dynamic conversion of the path between four types of signals (A / B / C / D) based on the indicators, and realizes the conversion by generating a new file without modifying the original file and corresponding metadata.

7. A semi-structured data processing system for the GreptimeDB analytical observable database according to claim 6, characterized in that: The dynamic conversion steps are as follows: Promotion: When a certain path participates in filtering / aggregation frequently over a long period of time and its type tends to be stable, it can be promoted step by step from D→C→B / A to make it more columnar and user-friendly; Degradation: When a path changes from stable to long tail and its usage frequency decreases, it can be downgraded from A / B to C / D to reduce column and metadata overhead. The conversion is achieved by generating a new file through compaction, without modifying the old file.

8. A semi-structured data processing method for the GreptimeDB analytical observable database, applied to the semi-structured data processing system for the GreptimeDB analytical observable database as described in any one of claims 1-7, characterized in that: Includes the following steps: S101. Data Reception: Receive one or more semi-structured data records R, ensuring that the JSON text in each record contains at least one JSON type field or JSON text field, and complete the initial data access; S102. Reading Table Meta Information and Configuration: Read and confirm the JSON column set of the record, the user path type prompt including path type constraints and null value constraints, table options and runtime strategies, and retrieve historical meta information to provide basic configuration support for subsequent parsing and judgment; S103. Parse into intermediate representation: Parse the JSON text of each record to generate an intermediate representation containing the path and value, and identify the current value type of each data. S104. File-level schema inference and merging: Based on the intermediate representation generated in S103, infer the file-level schema of a single batch of data; For multiple batches of data, perform a merge operation on the file-level schema of each batch and complete compatibility verification to ensure schema consistency; S105. Path Traversal and Classification: Combining the intermediate representation of the path and value generated in S103, the user path type prompt read in S102, historical metadata, and the current value type identified in S103, path traversal and classification are performed through a classification algorithm, and the output classification signal is divided into class A, class B, class C, class D and additional processing signal; S106. Construct columnar representation by category: Perform differential processing based on the category signal output in S105, and construct the corresponding columnar representation: For A-type paths, construct a Struct sub-column that determines the data type; for B-type paths, record file-level schema information and the frequency and location information of the corresponding fields. C-type paths construct Variant type columns; D-type paths construct a Map key-value set and a Key index summary. S107. Generate an immutable file and write it to disk: Write the columnar data processed in S106 into an immutable columnar file in column vector form, synchronously generate RowGroup level statistics, and optionally generate indexes as needed; S108. Update Manifest / Meta metadata: Based on the immutable columnar physical file generated in S107, extract and integrate its file identifier, file-level schema, RowGroup-level statistical information, optional index summary, and extended metadata. The extended metadata includes Variant type summary, field occurrence information, and Key index summary to form a unified file representation record and persist it. Construct a unified schema query view to provide metadata support for subsequent queries. S109. Query Planning and Execution: After receiving the user's query statement, the query path is rewritten and transformed into a unified function format that the database engine can recognize; combined with the path expression context, type inversion inference is performed to generate the read schema, loading strategy and predicate pruning strategy; data pruning is performed according to the above strategies to materialize only the data required by the query; finally, the materialized data is processed through vectorization operators to output the query results. S110. Compaction Phase: Based on the statistical information generated in S107 and the query feedback in S109, promotion / demotion operations are performed on various paths in the compaction phase, and new immutable columnar files are reprocessed and generated; the Manifest / Meta metadata is updated synchronously.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for processing semi-structured data in the GreptimeDB analytical observable database as described in claim 8.

10. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the method for processing semi-structured data in the GreptimeDB analytical observable database as described in claim 9. The electronic device includes one or more of a server, a cloud server, and a distributed computing cluster, and the memory is a non-volatile computer-readable storage medium.

Images