A multi-mode time series data processing method and system

Abstract

The application belongs to the technical field of databases, and specifically discloses a multi-mode time series data processing method and system. The method comprises the following steps: establishing a row storage table and a column storage table according to a table creation request, and establishing an associated mapping relationship between the row storage table and the column storage table; receiving time series data to be inserted and extracting label data and index data, determining a target data node based on label data hash calculation, writing the label data into the row storage table, and writing the index data into a target time series data block of the column storage table; performing partition pruning and scanning on the stored time series data based on a screening condition, and generating a query result by parallel execution of split query tasks and merging of results based on cross-mode query requirements; determining the affected time series data block based on an operation request, and updating the label data and the index data or deleting invalid data according to the type of the operation request. The application can realize low-redundancy storage, high-concurrency read-write and fast query analysis of time series data.

Description

Technical Field

[0001] This application belongs to the field of database technology, and more specifically, relates to a multi-modal time series data processing method and system. Background Technology

[0002] With the rapid development of technologies such as the Internet of Things (IoT) and the Industrial Internet, the scale of time-series data has exploded. Existing technologies, when processing such multi-modal time-series data, mainly face problems such as single mode, uneven data distribution, difficulty in cross-modal querying, rigid partition management, and poor data operation performance.

[0003] Therefore, there is an urgent need for a data processing solution that can take into account the multi-model nature of time series data, achieve intelligent distribution, support cross-model queries, and optimize the performance of various operations, so as to realize low-redundancy storage, high-concurrency read and write, and fast query and analysis of time series data. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of this application is to achieve low-redundancy storage, high-concurrency read and write, and fast query and analysis of time-series data.

[0005] To achieve the above objectives, in a first aspect, this application provides a multi-modal time series data processing method, comprising:

[0006] Receive a table creation request, create a row storage table and a column storage table according to the table creation request, and establish an association mapping relationship between the row storage table and the column storage table;

[0007] Receive the time series data to be inserted and extract the tag data and indicator data. Determine the target data node based on the hash calculation of the tag data. Write the tag data into the row storage table and write the indicator data into the target time series data block of the column storage table.

[0008] Receive a query request, extract filtering conditions and cross-model query requirements based on the query request, partition, prune and scan the stored time series data based on the filtering conditions, and generate query results by splitting query tasks for parallel execution and merging results based on the cross-model query requirements;

[0009] Receive an operation request, determine the affected time-series data block based on the operation request, update the tag data and indicator data according to the type of operation request, or delete invalid data.

[0010] Optionally, the row storage table stores the tag data in a row-oriented storage format to support transaction operations such as adding, deleting, modifying, and querying the tag data;

[0011] The column storage table stores indicator data in a columnar storage format and uses an adaptive compression algorithm to compress the data, in order to support sequential writing and batch querying of the indicator data.

[0012] The row storage table and the column storage table establish an association mapping relationship through entity identifiers, and an index is built based on the entity identifiers to realize the location from tag data to corresponding indicator data.

[0013] Optionally, the process of writing the time-series data specifically includes:

[0014] Parse the time series data to be inserted, and extract the label data, indicator data and corresponding timestamps;

[0015] Based on the tag data, a hash calculation is performed to determine the target data node for storing the time-series data;

[0016] In the target data node, the target time-series data block corresponding to the timestamp is determined by querying the cache;

[0017] If the target time-series data block is found in the cache, the tag data is written to the row storage table, and the indicator data is written to the target time-series data block in column storage format; if the target time-series data block is not found in the cache, a new time-series data block is created based on the timestamp, the new time-series data block is used as the target time-series data block, and the indicator data is written to the new time-series data block.

[0018] Optionally, the step of creating a new time-series data block based on the timestamp and writing the indicator data into the new time-series data block includes:

[0019] If the target time-series data block is not found in the cache, a time interval slice is calculated based on the timestamp, and the dimension metadata associated with the existing time interval slice is reused to generate a new time-series data block.

[0020] Optionally, the response process for the query request specifically includes:

[0021] In response to the query request, extract time filtering conditions, tag filtering conditions, and cross-data model association requirements;

[0022] Based on the time filtering conditions and tag filtering conditions, the time-series data block is partitioned and pruned to obtain a target data block set. The compressed data in the target data block set is then decompressed and scanned.

[0023] The query request is determined to be a cross-modal query. The query request is split into a first query task corresponding to the time series table and a second query task corresponding to the relation table, and executed in parallel using different data nodes.

[0024] It receives and merges query results from different data nodes, performs data association, aggregation, and sorting operations, and generates and returns the final query results.

[0025] Optionally, the response process for the operation request specifically includes:

[0026] In response to the operation request, the target table, filtering conditions, and updated content are extracted based on the operation request;

[0027] The affected time-series data blocks are located based on the filtering conditions, and the compressed time-series data blocks are decompressed.

[0028] Filter target data in the decompressed time-series data block according to the time filtering conditions and tag filtering conditions;

[0029] When the operation request is an update request, the corresponding tag data is updated in the row storage table, and the indicator data is updated in the corresponding column storage time series data block;

[0030] When the operation request is a deletion request, the target data is logically deleted in the corresponding column storage time-series data block;

[0031] The time-series data blocks after the operation are recompressed and stored, and the relevant metadata and index information are updated.

[0032] Commit data operation transactions to ensure the atomicity of operations and data consistency.

[0033] Optionally, the implementation process of the cross-modal query includes:

[0034] A syntax parsing extension mechanism is used to add time-series table-specific syntax support to the parser of data nodes, and to identify the association conditions between time-series data tables and relational data tables in the query request;

[0035] A distributed execution optimization mechanism is used to generate a distributed query execution plan in the optimizer of the data node based on the association conditions, route the first query task for the time-series data table to the corresponding data node for execution, and execute the second query task for the relational data table in parallel;

[0036] The result collaborative processing mechanism is used to receive time-series data sub-results and relational data sub-results from each data node at the computing node, and to perform correlation calculations on the time-series data sub-results and relational data sub-results through connection operators to generate the fused final query result.

[0037] Secondly, this application provides a multi-modal time-series data processing system, including:

[0038] The table creation module is used to receive a table creation request, create a row storage table and a column storage table according to the table creation request, and establish an association mapping relationship between the row storage table and the column storage table.

[0039] The writing module is used to receive the time series data to be inserted and extract the tag data and indicator data, determine the target data node based on the hash calculation of the tag data, write the tag data into the row storage table, and write the indicator data into the target time series data block of the column storage table.

[0040] The query module is used to receive query requests, extract filtering conditions and cross-model query requirements based on the query requests, partition, prune and scan the stored time-series data based on the filtering conditions, and generate query results by splitting query tasks for parallel execution and merging results based on the cross-model query requirements.

[0041] An operation module is used to receive operation requests, determine the affected time-series data blocks based on the operation requests, update the tag data and indicator data according to the type of operation request, or delete invalid data.

[0042] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0043] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0044] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0045] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0046] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0047] (1) This application significantly reduces storage redundancy by splitting time series data into tag data and indicator data according to their characteristics and adopting a hybrid storage mode of row storage and column storage respectively, combined with an adaptive compression algorithm; it achieves high-concurrency read and write by evenly distributing data through intelligent sharding based on tag hash and optimizing the write path by using caching, batch and asynchronous mechanisms; it achieves high-concurrency read and write by constructing a composite index based on time and tag dimensions, performing efficient partition pruning during querying, reducing invalid scans, and using the built-in cross-modal query fusion engine to achieve unified parallel processing and correlation analysis of time series and relational data, thereby achieving a comprehensive performance improvement of low-redundancy storage, high-concurrency read and write and fast query analysis.

[0048] (2) This application adopts a hybrid storage mode of row storage for tag data and column storage for indicator data to adapt to the differentiated characteristics of time-series data. Combined with an adaptive compression algorithm, it reduces storage overhead compared to the traditional single storage mode, while also reducing I / O overhead for tag data addition, deletion, modification, and query. Based on the tag data hash sharding strategy, it ensures that the data of the same device is stored centrally, and the data of different devices is evenly distributed on each DN node, reducing the cross-node access ratio and avoiding performance bottlenecks caused by data skew.

[0049] (3) This application improves the data insertion throughput, reduces response latency and improves operation efficiency in high-concurrency scenarios by optimizing insertion performance, query performance and update and delete performance;

[0050] (4) This application uses time-partitioned hot and cold data hierarchical management, supports automatic creation, merging and cleaning of data blocks, reduces manual maintenance costs, and ensures the locality of data access. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the multi-mode time-series data processing method provided in the embodiments of this application;

[0052] Figure 2 This is a schematic diagram of the layered architecture of an embodiment of this application;

[0053] Figure 3 This is a flowchart illustrating the timing table creation process according to an embodiment of this application;

[0054] Figure 4 This is a flowchart illustrating the cross-modal query execution process according to an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the structure of the multi-mode time-series data processing system provided in the embodiments of this application;

[0056] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0059] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0060] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0061] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0062] The embodiments of this application are described below with reference to the accompanying drawings.

[0063] Reference Figure 1 This application provides a multi-modal time series data processing method, including:

[0064] S101. Receive a table creation request, establish a row storage table and a column storage table according to the table creation request, and establish an association mapping relationship between the row storage table and the column storage table;

[0065] S102. Receive the time series data to be inserted and extract the tag data and indicator data, determine the target data node based on the hash calculation of the tag data, write the tag data into the row storage table, and write the indicator data into the target time series data block of the column storage table;

[0066] S103. Receive a query request, extract filtering conditions and cross-model query requirements based on the query request, partition, prune and scan the stored time-series data based on the filtering conditions, and generate query results by splitting query tasks for parallel execution and merging results based on the cross-model query requirements.

[0067] S104. Receive an operation request, determine the affected time-series data block based on the operation request, update the tag data and indicator data according to the type of operation request, or delete invalid data.

[0068] Specifically, this application provides a multi-modal time-series data processing method. Through an innovative hybrid storage architecture and intelligent data organization strategy, the core of the method is to separate and store time-series data according to its characteristics and optimize its distribution, query and maintenance processes, thereby achieving low redundancy, high concurrency and fast analysis.

[0069] This embodiment can be applied to database structure systems, namely, multi-mode time-series data processing systems built through distributed architecture design, hybrid storage mode, intelligent partitioning strategy and optimized execution engine.

[0070] It should be noted that the system consists of a two-layer architecture: compute nodes (CN) and data nodes (DN), which work together to achieve distributed processing of time-series data.

[0071] Compute Node (CN): Responsible for SQL parsing, optimizer expansion, cross-modal query coordination, and execution plan generation. It leverages the parsing, optimizer, and executor capabilities of existing distributed databases to extend time-series data-specific operation operators.

[0072] Data Nodes (DNs): Responsible for the physical storage and local computation of time-series data. Based on tag hash sharding and time partitioning strategies, they achieve uniform data distribution and lifecycle management.

[0073] First, via S101, the system receives a table creation request. Based on the column information defined in the request, it simultaneously creates two types of physical tables: a row-based storage table for storing metadata such as device tags, and a column-based storage table for storing time-series indicators such as temperature and pressure. Establishing the association and mapping relationship between the two is the foundation for all subsequent operations, typically achieved by assigning a unique entity identifier to each data entity, ensuring that tag data and indicator data can be mutually located.

[0074] Secondly, data insertion is implemented via S102. When time-series data to be inserted arrives, the system parses and separates the tag data and indicator data. Based on the tag data (such as device ID), a hash calculation is performed to determine which target data node the data should be routed to, thus achieving centralized storage of data from the same device. Subsequently, the tag data is written to a row-based storage table, while the indicator data is written in a column-based storage format to the target time-series data block corresponding to the data's timestamp in the column-based storage table.

[0075] Furthermore, data querying is implemented through S103. After receiving a query request, the system extracts filtering conditions such as time range and label conditions, as well as whether cross-data model association is required. The query engine first partitions and prunes the stored time-series data based on the filtering conditions, quickly filtering out irrelevant data partitions and significantly reducing the scanning scope. Then, it performs a scan on the target data. If the query is a cross-model query, the query task is automatically split, and the time-series data query and relational data query are executed in parallel. Finally, the results are merged at the computing node, and the results are returned after completing operations such as association and aggregation.

[0076] It should be noted that the time-series data in this application embodiment adopts an intelligent sharding and multi-dimensional partitioning strategy, as detailed below:

[0077] Spatial Sharding: Hash sharding is performed based on the user's unique tag column (such as device number) to route time-series data from the same device to the same DN node, avoiding cross-node access. Time Partitioning: Within each DN node, data with the same tag is divided into multiple data chunks based on timestamps. Each chunk corresponds to a fixed time interval (such as 1 day, 1 week), supporting tiered management of hot and cold data. Multidimensional Indexing: A composite index based on the time and tag dimensions is constructed. The chunk metadata records the time range and tag identifier, enabling rapid partition pruning.

[0078] Finally, data updates and deletions are implemented via S104. For update or deletion requests, the system first locates the affected time-series data blocks, typically based on the time range and tag conditions specified in the request. Depending on the operation type: update operations simultaneously modify the tag data in the row store table and the indicator data in the column store block; deletion operations use logical deletion markers, which are subsequently cleaned up by asynchronous tasks.

[0079] Optionally, the row storage table stores the tag data in a row-oriented storage format to support transaction operations such as adding, deleting, modifying, and querying the tag data;

[0080] The column storage table stores indicator data in a columnar storage format and uses an adaptive compression algorithm to compress the data, in order to support sequential writing and batch querying of the indicator data.

[0081] The row storage table and the column storage table establish an association mapping relationship through entity identifiers, and an index is built based on the entity identifiers to realize the location from tag data to corresponding indicator data.

[0082] Specifically, in this embodiment, the row storage table adopts a row-based storage format to efficiently support random addition, deletion, modification, and query operations on tag data. The column storage table adopts a column-based storage format and integrates an adaptive compression algorithm, perfectly adapting to the sequential writing and batch analysis query of indicator data, significantly reducing storage overhead. The two are associated through entity identifiers, and an index is built based on these identifiers to achieve sub-millisecond-level fast location from tags to massive indicator data.

[0083] It is understandable that in this embodiment, tag data and metric data are separated based on the characteristics of time-series data, and a differentiated storage strategy is adopted:

[0084] 1. Tag Data Storage: Tag data contains tag information such as device identifiers. It is used for adding, deleting, modifying and querying transactions. It is stored in row-based storage to ensure transaction consistency and operational flexibility.

[0085] 2. Metrics Data Storage: Metrics data consists of device metrics and only supports sequential writing and querying. It is stored in columnar format and combined with adaptive compression algorithms (such as snappy and lz4) to reduce storage overhead.

[0086] 3. Association mechanism: The mapping relationship between the Tags table and the Metrics data file is established through the entity identifier (entity_id), and the Hash index is used to achieve fast location with O(1) complexity.

[0087] Optionally, the process of writing the time-series data specifically includes:

[0088] Parse the time series data to be inserted, and extract the label data, indicator data and corresponding timestamps;

[0089] Based on the tag data, a hash calculation is performed to determine the target data node for storing the time-series data;

[0090] In the target data node, the target time-series data block corresponding to the timestamp is determined by querying the cache;

[0091] If the target time-series data block is found in the cache, the tag data is written to the row storage table, and the indicator data is written to the target time-series data block in column storage format; if the target time-series data block is not found in the cache, a new time-series data block is created based on the timestamp, the new time-series data block is used as the target time-series data block, and the indicator data is written to the new time-series data block.

[0092] Specifically, in this embodiment, the data is first parsed to extract the tag field, metric field, and timestamp. Next, the target data node is determined based on the hash value of the tag field. Within this node, a dedicated cache is queried to locate the target time-series data block corresponding to the timestamp. If the cache is hit, the metric data is directly written to that block; otherwise, a new time-series data block is created based on the timestamp and written as the target block. This process, through caching and intelligent creation mechanisms, ensures low latency and high throughput under high-concurrency writes.

[0093] Furthermore, if the target time-series data block is not found in the cache, a time interval slice is calculated based on the timestamp, and the dimension metadata associated with the existing same time interval slice is reused to generate a new time-series data block.

[0094] Understandably, when a cache miss occurs, the system does not simply allocate an empty block, but first calculates the time interval slice to which it belongs based on the timestamp. Crucially, the system reuses existing dimensional metadata associated with the same time interval slice to generate a new block. This embodiment avoids repeatedly creating and storing the same time period description information for each new data block, significantly reducing metadata overhead and improving storage management efficiency.

[0095] Optionally, the response process for the query request specifically includes:

[0096] In response to the query request, extract time filtering conditions, tag filtering conditions, and cross-data model association requirements;

[0097] Based on the time filtering conditions and tag filtering conditions, the time-series data block is partitioned and pruned to obtain a target data block set. The compressed data in the target data block set is then decompressed and scanned.

[0098] The query request is determined to be a cross-modal query. The query request is split into a first query task corresponding to the time series table and a second query task corresponding to the relation table, and executed in parallel using different data nodes.

[0099] It receives and merges query results from different data nodes, performs data association, aggregation, and sorting operations, and generates and returns the final query results.

[0100] Specifically, in this embodiment, when the system responds to a query, it first extracts the time filtering conditions, tag filtering conditions, and cross-modal association requirements. Using block metadata that records time ranges and tag identifiers, it performs efficient partitioning and pruning to obtain a very small set of target data blocks. The compressed data in this set is then decompressed and scanned. If a cross-modal query is determined, it is automatically split into a time-series query task and a relational query task, and distributed to different nodes for parallel execution. Finally, the results from both sides are merged at the computing node, and the association, aggregation, and sorting are completed through join operators, achieving unified and rapid analysis.

[0101] Optionally, the response process for the operation request specifically includes:

[0102] In response to the operation request, the target table, filtering conditions, and updated content are extracted based on the operation request;

[0103] The affected time-series data blocks are located based on the filtering conditions, and the compressed time-series data blocks are decompressed.

[0104] Filter target data in the decompressed time-series data block according to the time filtering conditions and tag filtering conditions;

[0105] When the operation request is an update request, the corresponding tag data is updated in the row storage table, and the indicator data is updated in the corresponding column storage time series data block;

[0106] When the operation request is a deletion request, the target data is logically deleted in the corresponding column storage time-series data block;

[0107] The time-series data blocks after the operation are recompressed and stored, and the relevant metadata and index information are updated.

[0108] Commit data operation transactions to ensure the atomicity of operations and data consistency.

[0109] This embodiment clarifies the complete transaction process for data updates and deletions. The system first parses the operation request to obtain the target table, filter conditions, and update content. Based on the filter conditions, it locates the specific time-series data block affected and decompresses it. In the decompressed data, it precisely locates the target row: for updates, it synchronously modifies the row storage table and column storage blocks; for deletions, it only marks them as logically deleted. After the operation is complete, the data block is recompressed, and the metadata and indexes are updated. Finally, the transaction is committed. This process, while ensuring data consistency, minimizes the impact of update and delete operations on system performance through precise location and logical deletion.

[0110] Optionally, the implementation process of the cross-modal query includes:

[0111] A syntax parsing extension mechanism is used to add time-series table-specific syntax support to the parser of data nodes, and to identify the association conditions between time-series data tables and relational data tables in the query request;

[0112] A distributed execution optimization mechanism is used to generate a distributed query execution plan in the optimizer of the data node based on the association conditions, route the first query task for the time-series data table to the corresponding data node for execution, and execute the second query task for the relational data table in parallel;

[0113] The result collaborative processing mechanism is used to receive time-series data sub-results and relational data sub-results from each data node at the computing node, and to perform correlation calculations on the time-series data sub-results and relational data sub-results through connection operators to generate the fused final query result.

[0114] Specifically, in this embodiment, the syntax parsing extension mechanism enables the system to natively recognize the association syntax between time-series tables and relational tables in SQL. The distributed execution optimization mechanism is responsible for generating a distributed plan based on the association conditions, routing time-series queries to the corresponding data nodes, and initiating them in parallel with relational queries. The result collaborative processing mechanism aggregates the intermediate results returned by each node at the computing node and performs the final association calculation and aggregation through join operators. These three mechanisms work together to make cross-modal queries transparent and efficient to the user.

[0115] Syntax parsing extension mechanism: Add support for time series table-specific syntax to the CN node parser to identify the association conditions between time series tables and relational tables in cross-modal query statements.

[0116] Execution plan optimization mechanism: The optimizer generates a distributed execution plan, routes time-series table query tasks to the corresponding DN nodes, and executes relation table queries in parallel.

[0117] The result collaborative processing mechanism is as follows: CN nodes receive time-series data and relational data returned by each DN node, and complete the association calculation through connection operators (NestLoop, MergeJoin, HashJoin), supporting aggregation and sorting operations such as GROUPBY and ORDERBY.

[0118] The following provides a detailed description of the time-series data table creation process, data insertion process, data query process, and data update and deletion process in the embodiments of this application:

[0119] (1) Time series data table creation process:

[0120] Step 1: Receive the creation request and collect parameters such as table name, column definition, partition key (time column), and chunk time interval.

[0121] Step 2: Parameter validation, verifying the validity of the partition key type, the uniqueness of the table name, and user permissions.

[0122] Step 3: Generate the physical table structure, create a Tag row table and a Metrics column table, and establish an entity_id mapping relationship.

[0123] Step 4: Record metadata, storing supertable (Ts_table) information, partitioning strategies, Chunk configurations, etc. in the system tables.

[0124] Step 5: Initialize the SubspaceStore cache and pre-allocate memory for Chunk location acceleration.

[0125] (2) Data insertion process:

[0126] Step 1: Parse the inserted data and extract the Tag, Metrics, and timestamp fields.

[0127] Step 2: Determine the target DN node based on Tag hash calculation and generate a multi-dimensional spatial positioning point.

[0128] Step 3: Query the target Chunk through the SubspaceStore cache. If no match is found, trigger Chunk creation.

[0129] Step 4: When creating a Chunk, calculate the time interval slice (DimensionSlice) to generate time series data blocks such as time series data cubes (Ts_cube), and reuse existing dimension slices to reduce metadata overhead.

[0130] Step 5: Write Tag data to the row storage table, and write Metrics data to the target Chunk in batches according to the column storage format. Asynchronous insertion and batch submission are supported.

[0131] (3) Data query process:

[0132] Step 1: Parse the query statement and extract time filtering conditions, tag filtering conditions, and cross-model association requirements.

[0133] Step 2: Partition trimming, filtering out invalid chunks based on time range and tag identifiers, keeping only the target partition.

[0134] Step 3: Chunk scan optimization, automatically decompress compressed chunks, and select the optimal scan method (index scan / sequential scan).

[0135] Step 4: Cross-modal query processing, splitting the time-series table and relational table query tasks, and executing them in parallel on the DN node respectively.

[0136] Step 5: Result merging. The association, aggregation, and sorting operations of time series data and relational data are completed at the CN node, and the final result is returned.

[0137] (4) Data update and deletion process:

[0138] Step 1: Parse the operation statement to extract the target table, filter conditions (including partition keys), and update content.

[0139] Step 2: Locate the affected chunk and decompress the compressed chunk.

[0140] Step 3: Filter target data based on Tag and time range. Update operations only modify the Tag row storage table and the corresponding Metrics column storage block. Delete operations mark invalid data (to be cleaned up asynchronously later).

[0141] Step 4: Re-compress the Chunk and update the metadata and index information.

[0142] Step 5: Commit the transaction to ensure the atomicity of the operation and the consistency of the data.

[0143] Reference Figure 2 , Figure 2 This is a schematic diagram of the layered architecture of an embodiment of this application, showing the layered architecture of compute nodes (CN) and data nodes (DN), including the parsing optimization layer of CN, the cross-mode coordination layer and the storage layer and partitioning management layer of DN, which clarifies the collaborative process of data sharding routing and cross-mode query.

[0144] The diagram illustrates the hybrid storage architecture of Tags and Metrics. It shows the relationship between the Tag row storage table, the Metrics column storage block, the entity_id mapping, and the Hash index, demonstrating the complete link from Tag filtering to data block location.

[0145] Reference Figure 3 , Figure 3 This is a flowchart illustrating the time series table creation process according to an embodiment of this application. It shows the complete process of collecting the basic information required to create the time series data table, creating the physical table structure, and recording the metadata of the time series data table during the time series data table creation process, including the following steps:

[0146] Begin, prepare to create parameters;

[0147] Verify the table structure;

[0148] How to determine if the table exists?

[0149] Yes, the end;

[0150] No, define the partition key;

[0151] Set the block time interval;

[0152] Create SQL to generate a time series table;

[0153] Perform the creation operation;

[0154] Determine: Was the execution successful?

[0155] Yes, create metadata record, end;

[0156] No, end.

[0157] Reference Figure 4 , Figure 4 This is a flowchart of the cross-modal query execution process of this application. The flowchart illustrates the entire process of statement splitting, parallel execution, result association and aggregation of time-series tables and relational tables in cross-modal queries, reflecting the distributed collaborative computing logic. The specific process is as follows:

[0158] Parse the SELECT statement;

[0159] Metadata manager, obtain table structure, partition and other information;

[0160] Partition location: Determine the partition where the data is located;

[0161] Block scanner, ready to read data blocks;

[0162] Parallel processing of data blocks involves executing the following sub-processes based on data type and task:

[0163] Compressed blocks are decompressed, uncompressed blocks are scanned, indexes are filtered, and statistical information is aggregated.

[0164] Results merged;

[0165] Finally, the query results are returned.

[0166] The following examples illustrate this application:

[0167] In the Industrial Internet of Things (IIoT) scenario, a factory needs to collect operational data from 100,000 devices (each device generates 10 metrics per second, including indicators such as temperature and pressure, as well as tag information such as device model and workstation), and also needs to analyze the data by linking it with relational data such as device files.

[0168] A multi-mode time series data system is constructed using the scheme proposed in this application:

[0169] 1. Storage configuration: Tag data (equipment model, workstation, etc.) adopts row storage mode, and Metrics data (temperature, pressure, etc.) adopts column storage mode and zstd compression is enabled.

[0170] 2. Sharding strategy: Sharding is based on the device number (Tag column) hash and distributed to 10 DN nodes. Each node is divided into Chunks at 1-day time intervals.

[0171] 3. Business Operations:

[0172] 4. Data insertion: Processes 1 million metrics per second, with insertion latency controlled within 50ms through batch submission and asynchronous writing.

[0173] 5. Query and Analysis: Query equipment operation data by time range (e.g., the last hour) + workstation (Tag) and associate it with equipment maintenance records in the relationship table. The query response time is ≤300ms.

[0174] 6. Data Update: Modify equipment workstation information (Tag update), operation response time ≤10ms; delete historical metrics data that expired 3 months ago, batch cleanup via Chunk, efficiency improved by 4 times.

[0175] The implementation results show that, compared with traditional solutions, the system reduces storage overhead by 45%, improves query performance by 60%, and improves cross-modal analysis efficiency by 75%, fully meeting the needs of high-concurrency, low-latency industrial-grade time-series data processing.

[0176] Reference Figure 5 This application also provides a multi-mode time series data processing system, including:

[0177] The table creation module 510 is used to receive a table creation request, create a row storage table and a column storage table according to the table creation request, and establish an association mapping relationship between the row storage table and the column storage table.

[0178] The writing module 520 is used to receive the time series data to be inserted and extract the tag data and indicator data, determine the target data node based on the hash calculation of the tag data, write the tag data into the row storage table, and write the indicator data into the target time series data block of the column storage table.

[0179] The query module 530 is used to receive query requests, extract filtering conditions and cross-model query requirements based on the query requests, partition, prune and scan the stored time-series data based on the filtering conditions, and generate query results by splitting query tasks for parallel execution and merging results based on the cross-model query requirements.

[0180] The operation module 540 is used to receive an operation request, determine the affected time-series data block based on the operation request, update the tag data and indicator data according to the type of operation request, or delete invalid data.

[0181] Optionally, the row storage table stores the tag data in a row-oriented storage format to support transaction operations such as adding, deleting, modifying, and querying the tag data;

[0182] The column storage table stores indicator data in a columnar storage format and uses an adaptive compression algorithm to compress the data, in order to support sequential writing and batch querying of the indicator data.

[0183] The row storage table and the column storage table establish an association mapping relationship through entity identifiers, and an index is built based on the entity identifiers to realize the location from tag data to corresponding indicator data.

[0184] Optionally, the process of writing the time-series data specifically includes:

[0185] Parse the time series data to be inserted, and extract the label data, indicator data and corresponding timestamps;

[0186] Based on the tag data, a hash calculation is performed to determine the target data node for storing the time-series data;

[0187] In the target data node, the target time-series data block corresponding to the timestamp is determined by querying the cache;

[0188] If the target time-series data block is found in the cache, the tag data is written to the row storage table, and the indicator data is written to the target time-series data block in column storage format; if the target time-series data block is not found in the cache, a new time-series data block is created based on the timestamp, the new time-series data block is used as the target time-series data block, and the indicator data is written to the new time-series data block.

[0189] Optionally, the step of creating a new time-series data block based on the timestamp and writing the indicator data into the new time-series data block includes:

[0190] If the target time-series data block is not found in the cache, a time interval slice is calculated based on the timestamp, and the dimension metadata associated with the existing time interval slice is reused to generate a new time-series data block.

[0191] Optionally, the response process for the query request specifically includes:

[0192] In response to the query request, extract time filtering conditions, tag filtering conditions, and cross-data model association requirements;

[0193] Based on the time filtering conditions and tag filtering conditions, the time-series data block is partitioned and pruned to obtain a target data block set. The compressed data in the target data block set is then decompressed and scanned.

[0194] The query request is determined to be a cross-modal query. The query request is split into a first query task corresponding to the time series table and a second query task corresponding to the relation table, and executed in parallel using different data nodes.

[0195] It receives and merges query results from different data nodes, performs data association, aggregation, and sorting operations, and generates and returns the final query results.

[0196] Optionally, the response process for the operation request specifically includes:

[0197] In response to the operation request, the target table, filtering conditions, and updated content are extracted based on the operation request;

[0198] The affected time-series data blocks are located based on the filtering conditions, and the compressed time-series data blocks are decompressed.

[0199] Filter target data in the decompressed time-series data block according to the time filtering conditions and tag filtering conditions;

[0200] When the operation request is an update request, the corresponding tag data is updated in the row storage table, and the indicator data is updated in the corresponding column storage time series data block;

[0201] When the operation request is a deletion request, the target data is logically deleted in the corresponding column storage time-series data block;

[0202] The time-series data blocks after the operation are recompressed and stored, and the relevant metadata and index information are updated.

[0203] Commit data operation transactions to ensure the atomicity of operations and data consistency.

[0204] Optionally, the implementation process of the cross-modal query includes:

[0205] A syntax parsing extension mechanism is used to add time-series table-specific syntax support to the parser of data nodes, and to identify the association conditions between time-series data tables and relational data tables in the query request;

[0206] A distributed execution optimization mechanism is used to generate a distributed query execution plan in the optimizer of the data node based on the association conditions, route the first query task for the time-series data table to the corresponding data node for execution, and execute the second query task for the relational data table in parallel;

[0207] The result collaborative processing mechanism is used to receive time-series data sub-results and relational data sub-results from each data node at the computing node, and to perform correlation calculations on the time-series data sub-results and relational data sub-results through connection operators to generate the fused final query result.

[0208] Reference Figure 6 Based on the methods in the above embodiments, this application provides an electronic device that may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the methods in the above embodiments.

[0209] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0210] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0211] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0212] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0213] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0214] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0215] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0216] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for processing multi-modal time series data, characterized in that, include: Receive a table creation request, create a row storage table and a column storage table according to the table creation request, and establish an association mapping relationship between the row storage table and the column storage table; Receive the time series data to be inserted and extract the tag data and indicator data. Determine the target data node based on the hash calculation of the tag data. Write the tag data into the row storage table and write the indicator data into the target time series data block of the column storage table. Receive a query request, extract filtering conditions and cross-model query requirements based on the query request, partition, prune and scan the stored time series data based on the filtering conditions, and generate query results by splitting query tasks for parallel execution and merging results based on the cross-model query requirements; Receive an operation request, determine the affected time-series data block based on the operation request, and update the tag data and indicator data according to the type of operation request, or delete invalid data. The process of writing the time-series data specifically includes: Parse the time series data to be inserted, and extract the label data, indicator data and corresponding timestamps; Based on the tag data, a hash calculation is performed to determine the target data node for storing the time-series data; In the target data node, the target time-series data block corresponding to the timestamp is determined by querying the cache; If the target time-series data block is hit from the cache, the tag data is written to the row storage table, and the indicator data is written to the target time-series data block in column storage format; If the target time-series data block is not found in the cache, a time interval slice is calculated based on the timestamp, and the dimension metadata associated with the existing same time interval slice is reused to generate a new time-series data block, so as to avoid repeatedly creating and storing the same time period description information for each new data block.

2. The multi-modal time-series data processing method according to claim 1, characterized in that, The row storage table stores the tag data in a row-oriented storage format, which supports transaction operations such as adding, deleting, modifying, and querying the tag data. The column storage table stores indicator data in a columnar storage format and uses an adaptive compression algorithm to compress the data, in order to support sequential writing and batch querying of the indicator data. The row storage table and the column storage table establish an association mapping relationship through entity identifiers, and an index is built based on the entity identifiers to realize the location from tag data to corresponding indicator data.

3. The multi-modal time-series data processing method according to claim 1, characterized in that, The response process for the query request specifically includes: In response to the query request, extract time filtering conditions, tag filtering conditions, and cross-data model association requirements; Based on the time filtering conditions and tag filtering conditions, the time-series data block is partitioned and pruned to obtain a target data block set. The compressed data in the target data block set is then decompressed and scanned. The query request is determined to be a cross-modal query. The query request is split into a first query task corresponding to the time series table and a second query task corresponding to the relation table, and executed in parallel using different data nodes. It receives and merges query results from different data nodes, performs data association, aggregation, and sorting operations, and generates and returns the final query results.

4. The multi-modal time-series data processing method according to claim 3, characterized in that, The response process for the operation request specifically includes: In response to the operation request, the target table, filtering conditions, and updated content are extracted based on the operation request; The affected time-series data blocks are located based on the filtering conditions, and the compressed time-series data blocks are decompressed. Filter target data in the decompressed time-series data block according to the time filtering conditions and tag filtering conditions; When the operation request is an update request, the corresponding tag data is updated in the row storage table, and the indicator data is updated in the corresponding column storage time series data block; When the operation request is a deletion request, the target data is logically deleted in the corresponding column storage time-series data block; The time-series data blocks after the operation are recompressed and stored, and the relevant metadata and index information are updated. Commit data operation transactions to ensure the atomicity of operations and data consistency.

5. The multi-modal time series data processing method according to claim 3, characterized in that, The implementation process of the cross-modal query includes: A syntax parsing extension mechanism is used to add time-series table-specific syntax support to the parser of data nodes, and to identify the association conditions between time-series data tables and relational data tables in the query request; A distributed execution optimization mechanism is used to generate a distributed query execution plan in the optimizer of the data node based on the association conditions, route the first query task for the time-series data table to the corresponding data node for execution, and execute the second query task for the relational data table in parallel; The result collaborative processing mechanism is used to receive time-series data sub-results and relational data sub-results from each data node at the computing node, and to perform correlation calculations on the time-series data sub-results and relational data sub-results through connection operators to generate the fused final query result.

6. A multi-modal time-series data processing system, characterized in that, include: The table creation module is used to receive a table creation request, create a row storage table and a column storage table according to the table creation request, and establish an association mapping relationship between the row storage table and the column storage table. The writing module is used to receive the time series data to be inserted and extract the tag data and indicator data, determine the target data node based on the hash calculation of the tag data, write the tag data into the row storage table, and write the indicator data into the target time series data block of the column storage table. The query module is used to receive query requests, extract filtering conditions and cross-model query requirements based on the query requests, partition, prune and scan the stored time-series data based on the filtering conditions, and generate query results by splitting query tasks for parallel execution and merging results based on the cross-model query requirements. An operation module is used to receive operation requests, determine the affected time-series data blocks based on the operation requests, update the tag data and indicator data according to the type of operation request, or delete invalid data. The process of writing the time-series data specifically includes: Parse the time series data to be inserted, and extract the label data, indicator data and corresponding timestamps; Based on the tag data, a hash calculation is performed to determine the target data node for storing the time-series data; In the target data node, the target time-series data block corresponding to the timestamp is determined by querying the cache; If the target time-series data block is hit from the cache, the tag data is written to the row storage table, and the indicator data is written to the target time-series data block in column storage format; If the target time-series data block is not found in the cache, a time interval slice is calculated based on the timestamp, and the dimension metadata associated with the existing same time interval slice is reused to generate a new time-series data block, so as to avoid repeatedly creating and storing the same time period description information for each new data block.

7. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-5.

Images